Appeared that ASAN build could return nil values from child
process executed via popen. Which sporadically produces
error messages:

  not ok 1 ..console_debugger_session.test_interactive_debugger_session
  #   ...ntool/test/app-luatest/console_debugger_session_test.lua:11:
  # attempt to index local 's' (a nil value)
  #   stack traceback:

Due to used timeouts we are ok to receive nil at one of a loop
cycle - we simply repeat read to popen object. So handle this
case gracefully.

NO_DOC=bugfix
NO_CHANGELOG=internal

SQL select1:1-8.4 test didn't have any rationale in modern tarantool SQL
and had been turned off since 2017.

Closes: #5737
NO_DOC=test
NO_CHANGELOG=test

All the heavy lifting was done in the previous commit that dropped
the syslog formatter. This commit just removes the checks that forbid
using syslog with json.

Closes #7860

@TarantoolBot document
Title: JSON log format can now be used with syslog

The configuration reference says that syslog is incompatible with
the JSON log format:

https://www.tarantool.io/en/doc/latest/reference/configuration/#confval-log_format

This isn't true anymore. If JSON is used with syslog, then a JSON
message body is appended to the syslog header.

See https://github.com/tarantool/tarantool/issues/7860.

Some internal modules have been recently copied to luatest repo [1,2,3]
and now they can be safely removed, and the corresponding functionality
from luatest can be used instead.

Affected modules:

- test/luatest_helpers/misc.lua
- test/luatest_helpers/fiber.lua
- test/luatest_helpers/proxy/*

[1] https://github.com/tarantool/luatest/pull/247
[2] https://github.com/tarantool/luatest/pull/248
[3] https://github.com/tarantool/luatest/pull/255

Closes tarantool/luatest#238
Closes tarantool/luatest#251

NO_DOC=testing stuff
NO_TEST=testing stuff
NO_CHANGELOG=testing stuff

tarantool.debug.getsources() used to provide
lua sources only for libserver lua modules,
and was not providing sources for src/box/lua/*
modules.

Now this code works:
```lua
local tnt = require 'tarantool'
local code1 = tnt.debug.getsources('box/session')
local code1 = tnt.debug.getsources('@builtin/box/session..lua')
```

Closes #7839

NO_DOC=bugfix
NO_CHANGELOG=later

`dlsym` is known to be buggy in FreeBSD. See #7640.

NO_DOC=internal
NO_CHANGELOG=internal

See [1] for some details on why the code for log configuration needs
some care. In short log config validity checks are spread thru many
places, on some code paths we use one checks and on other code paths
other checks. And we make repetitive validity checks many times in
runtime on single configure call.

We can also reuse code for setting default values, checking options
type and resetting values to default.

- As a side effect of refactoring one can now reset values to default thru
  `log.cfg()` so now `log.cfg()` is on par with `box.cfg` in this respect.

- This patch also drops conversion `log_level` from string to number.

  Before (shorten):

    tarantool> box.cfg{log_level='warn'}
    tarantool> box.cfg.log_level
    - info
    tarantool> log.cfg.level
    - 5

  Also:
    tarantool> log.cfg{level='info'}
    tarantool> log.cfg.level
    - 5
    tarantool> box.cfg{}
    tarantool> box.cfg.log_level
    - 5

  After patch if `log_level`/`level` is given as string than it is saved
  and returned as string too. I guess it should not affect users but looks
  more handy.

- Also fixed issue with inconsistent setting `log_nonblock` thru
  `box.cfg()` and `log.cfg()`. In former case `nil` means setting default
  depending on logger type. In the latter case `nil` meant setting
  `nonblock` to `false`.

- Also patch fixes #7447.

Closes #7447.

[1] PR for this refactoring
https://github.com/tarantool/tarantool/pull/7454

NO_DOC=refactoring/tiny API improvemnent

It is same as luaL_dosting but also sets fiber diag.

Part of #7447

NO_DOC=internal
NO_CHANGELOG=internal

If a complex (tuple) foreign key is added along with a new field name
that participates in that foreign key, the foreign key does not work
correctly. This happens because the new name of a local field is added
to new_space->def->dict only in ModifySpace::alter, while before that,
new_space->def->dict points to the old dictionary with old names (see
ModifySpace::alter_def). So when local_field_no is initialized earlier
in alter_space_do -> space_create -> tuple_constraint_fkey_init, it's
unable to find the new field number in the old dictionary.

Fix this by moving `new_def->dict = alter->old_space->def->dict;` from
ModifySpace::alter_def() to ModifySpace::alter(). Note that, as before,
we refer to the old dictionary from the new space (just put new names
into it), because that dict is referenced by existing tuple formats.

Closes #7652

NO_DOC=bugfix

The test is expected the replication is broken by the time of the
server switch. Apparently, switch won the race, so the expectation
is changed for waiting.

Closes https://github.com/tarantool/tarantool-qa/issues/271

NO_CHANGELOG=testing
NO_DOC=testing
NO_TEST=testing

It was disabled in the commit ccfef5fd ("memtx: ignore
slab_alloc_granularity < machine word size") to postpone addressing
unaligned objects issues.

Add PACKED attribute to inform compiler of unaligned objects. We also
need to introduce `struct stailq_entry_ptr` to be able to inform compiler
that `last` pointer unaligned too.

Closes #7422

NO_DOC=internal
NO_CHANGELOG=internal

`say_logrotate` does not rotate logs synchronously. It posts tasks to
coio which executes them in it's pool thread. On application exit we
destroy logs calling `log_destroy`. This function waits for rotate task
to finish because if it will free resources immediately then unfinished
rotate task can have use-after-free issues. Waiting crashes because at
this moment event loop is not running which is required for
`fiber_cond_wait` to work.

Note that if there is no crash then we will hang forever waiting in
`log_destroy` because event loop is not running and
`log_rotate_async_cb` will never be executed.

Let's use mutexes and conditions instead. It solves both problems with
crash and hang. Thanks Vladimir Davydov for idea.

Fixes #4450

NO_DOC=bugfix

This commit fixes BEGIN, COMMIT, and ROLLBACK counters in the box.stat()
output. Before this commit, they always showed 0. Now, they report
the number of started, committed, and rolled back transactions,
respectively.

Closes #7583

NO_DOC=bug fix

Currently, if a snapshot contains some correct entries, but doesn't
include system spaces, Tarantool crashes with segmentation fault, or
for Debug build: void diag_raise(): Assertion `e != NULL' failed.
This happens because memtx_engine_recover_snapshot returns -1, while
diag is not set. Let's panic instead of a crash.

Closes #7800

NO_DOC=bugfix

memtx_read_view_opts contains the only flag include_temporary_tuples,
which is set to read_view_opts.enable_temporary_spaces. Let's drop this
intermediary structure and pass read_view_opts to the allocator instead.

NO_DOC=refactoring
NO_CHANGELOG=refactoring

0xc1 isn't a valid MsgPack header, but it was allowed by mp_check.
As a result, msgpack.decode crashed while trying to decode it.
This commit updates the msgpuck library to fix this issue.

Closes #7818

NO_DOC=bug fix

So one can easily check current box status.

NO_DOC=minor change

Closes #7255

We're planning to introduce a basic C API for user read views (EE-only).
Like all other box C API functions, the new API functions will use the
existing box error C API for reporting errors. The problem is that
a read view created using C API should be usable from user threads
(started with the pthread lib) while the box error C API doesn't work
in user threads, because those threads don't have the cord pointer
initialized (a diagnostic area is stored in a cord object).

To address this issue, let's create a new cord object automatically on
first use of cord() if it wasn't created explicitly. Automatically
created object is destroyed at thread exit (to achieve that, we use
the C++ RAII concept).

Closes #7814

NO_DOC=The C API documentation doesn't say anything about threads.
       Let's keep it this way for now. We're planning to introduce
       a new C API to work with threads in C modules. We'll update
       the doc when it's ready.

getenv() return values cannot be trusted, because an attacker might set
them. For instance, we shouldn't expect, that getenv() returns a value
of some sane size.

Another problem is that getenv() returns a pointer to one of
`char **environ` members, which might change upon next setenv().

Introduce a wrapper, getenv_safe(), which returns the value only when
it fits in a buffer of a specified size, and copies the value onto the
buffer. Use this wrapper everywhere in our code.

Below's a slightly decorated output of `grep -rwn getenv ./src --include
*.c --include *.h --include *.cc --include *.cpp --include *.hpp
--exclude *.lua.c` as of 2022-10-14.
`-` marks invalid occurences (comments, for example),
`*` marks the places that are already guarded before this patch,
`X` mars the places guarded in this patch, and
`^` marks places fixed in the next commit:

NO_WRAP
```
* ./src/lib/core/coio_file.c:509:	const char *tmpdir = getenv("TMPDIR");
X ./src/lib/core/errinj.c:75: const char *env_value = getenv(inj->name);
- ./src/proc_title.c:202: * that might try to hang onto a getenv() result.)
- ./src/proc_title.c:241:	* is mandatory to flush internal libc caches on getenv/setenv
X ./src/systemd.c:54: sd_unix_path = getenv("NOTIFY_SOCKET");
* ./src/box/module_cache.c:300: const char *tmpdir = getenv("TMPDIR");
X ./src/box/sql/os_unix.c:1441: azDirs[0] = getenv("SQL_TMPDIR");
X ./src/box/sql/os_unix.c:1446: azDirs[1] = getenv("TMPDIR");
* ./src/box/lua/console.c:394: const char *envvar = getenv("TT_CONSOLE_HIDE_SHOW_PROMPT");
^ ./src/box/lua/console.lua:771: local home_dir = os.getenv('HOME')
^ ./src/box/lua/load_cfg.lua:1007: local raw_value = os.getenv(env_var_name)
X ./src/lua/init.c:575: const char *path = getenv(envname);
X ./src/lua/init.c:592: const char *home = getenv("HOME");
* ./src/find_path.c:77: snprintf(buf, sizeof(buf) - 1, "%s", getenv("_"));
```
NO_WRAP

Part-of #7797

NO_DOC=security

This patch fixes the issue described in issue #5310 when the tuple
format has more fields than the space format. This solution is more
general than the solution in 89057a21.

Follow-up #5310
Closes #4666

NO_DOC=bugfix

If we raise different errors in case of entering an invalid password and
entering the login of a non-existent user during authorization, it will
open the door for an unauthorized person to enumerate users.
So let's unify raised errors in the cases described above.

Closes #tarantool/security#16

NO_DOC=security fix

NO_TEST=see it elsewhere

Part of #7593

@TarantoolBot document
Title: Console debugger for Lua

Console debugger luadebug.lua
==============================

Module `luadebug.lua` is available as console debugger of Lua scripts.
It's activated via:

```
local debugger = require 'luadebug'
debugger()
```

Originally we have used 3rd-party code from slembcke/debugger.lua but
significantly refactored since then.

Currently available console shell commands are:
```
    c|cont|continue
    - continue execution
    d|down
    - move down the stack by one frame
    e|eval $expression
    - execute the statement
    f|finish|step_out
    - step forward until exiting the current function
    h|help|?
    - print this help message
    l|locals
    - print the function arguments, locals and upvalues
    n|next|step_over
    - step forward by one line (skipping over functions)
    p|print $expression
    - execute the expression and print the result
    q|quit
    - exit debugger
    s|st|step|step_into
    - step forward by one line (into functions)
    t|trace|bt
    - print the stack trace
    u|up
    - move up the stack by one frame
    w|where $linecount
    - print source code around the current line
```

Console debugger `luadebug.lua` allows to see sources of builtin
Tarantool module (e.g. `@builtin/datetime.lua`), and it uses new
function introduced for that purpose `tarantool.debug.getsources()`,
one could use this function in any external GUI debugger (i.e. vscode
or JetBrains) if need to show sources of builtin modules while they
have been debugged.

> Please see third_party/lua/README-luadebug.md for a fuller description
> of an original luadebug.lua implementation.

Created luatest test for interactive debugger luadebug.lua.
We use separate debug-target.lua for execution under
control of debugger session.

NO_DOC=test
NO_CHANGELOG=test

Extend Tarantool kernel internal API with the call
`tarantool.debug.getsources()` to allow to retrieve sources
of a Tarantool `builtin/*` modules to show them in the
debugger shell.

Created simple luatest script for checking consistency
of a values returned from `require 'tarantool'.debug.getsources()`
and an ctual script file content we expected to receive.

NO_DOC=see future commit
NO_CHANGELOG=see future commit

The _vfunc system space is the sysview for the _func system space.
However, the _vfunc format is different from the _func format. This
patch makes the _vfunc format the same as the _func format.

Closes #7822

NO_DOC=bugfix

We used to ignore timezone difference (in `tzoffset`) for
datetime subtraction operation:

```
tarantool> datetime.new{tz='MSK'} - datetime.new{tz='UTC'}
---
- +0 seconds
...

tarantool> datetime.new{tz='MSK'}.timestamp -
           datetime.new{tz='UTC'}.timestamp
---
- -10800
...
```

Now we accumulate tzoffset difference in the minute component
of a resultant interval:

```
tarantool> datetime.new{tz='MSK'} - datetime.new{tz='UTC'}
---
- -180 minutes
...
```

Closes #7698

NO_DOC=bugfix

We did not take into consideration the fact that
as result of date/time arithmetic we could get
in a different timezone, if DST boundary has been
crossed during operation.

```
tarantool> datetime.new{year=2008, month=1, day=1,
			tz='Europe/Moscow'} +
	   datetime.interval.new{month=6}
---
- 2008-07-01T01:00:00 Europe/Moscow
...
```

Now we resolve tzoffset at the end of operation if
tzindex is not 0.

Fixes #7700

NO_DOC=bugfix

Currently, in case of recovery from an old snapshot, Tarantool allows to
perform DDL operations on an instance with non-upgraded schema.
It leads to various unpredictable errors (because the DDL code assumes
that the schema is already upgraded). This patch forbids the following
operations unless the user has the most recent schema version:
- box.schema.space.create
- box.schema.space.drop
- box.schema.space.alter
- box.schema.index.create
- box.schema.index.drop
- box.schema.index.alter
- box.schema.sequence.create
- box.schema.sequence.drop
- box.schema.sequence.alter
- box.schema.func.create
- box.schema.func.drop

Closes #7149

NO_DOC=bugfix

By default a user might not have privileges to access the _schema space,
that will cause an error during schema_needs_upgrade(), which calls
get_version(). Fix this by using C variable dd_version_id, which is
updated in the _schema.version replace trigger.

There's a special case for upgrade() during bootstrap() - triggers are
disabled during bootstrap, that's why dd_version_id is not being updated.
Handle this by passing _initial_version=1.7.5 to the upgrade function.

Part of #7149

NO_DOC=internal
NO_CHANGELOG=internal

This patch fixed the assertion when JOIN uses index of unsupported type.

Closes #5678

NO_DOC=bugfix

This commit adds support of transaction isolation levels introduced
earlier for memtx mvcc by commit ec750af6 ("txm: introduce
transaction isolation levels"). The isolation levels work exactly in
the same way as in memtx:

 - Unless a transaction explicitly specifies the 'read-committed'
   isolation level, it'll skip prepared statements, even if they are
   visible from its read view. The background for this was implemented
   in the previous patches, which added the is_prepared_ok flag to
   cache and mem iterators.

 - If a transaction skips a prepared statement, which would otherwise be
   visible from its read view, it's sent to the most recent read view
   preceding the prepared statement LSN. Note, older prepared statements
   are still visible from this read view and can actually be selected if
   committed later.

 - A transaction using the 'best-effort' isolation level (default) is
   switched to 'read-committed' when it executes the first write
   statement.

The implementation is tested by the existing memtx mvcc tests that were
made multi-engine in the scope of this commit. However, we add one more
test case - the one that checks that a 'best-effort' read view is
properly updated in case there is more than one prepared transaction.
Also, there are few tests that relied upon the old implementation and
assumed that select from Vinyl may return unconfirmed tuples. We update
those tests here as well.

Closes #5522

NO_DOC=already documented

To implement read-confirmed and best-effort isolation levels, we need
to skip unconfirmed (aka prepared) statements in the cache iterator. To
achieve that, we add a new flag is_prepared_ok. Unless the flag is set,
the iterator will skip prepared statements even if they are visible from
the iterator read view. Note, in contrast to the mem iterator, we don't
need to keep track of the min skipped statement LSN, because the cache
is just a view of the underlying levels so we'll find it out when we
descend to the mem level.

Needed for #5522

NO_DOC=internal
NO_CHANGELOG=internal

To implement read-confirmed and best-effort isolation levels, we need
to skip unconfirmed (aka prepared) statements in the mem iterator. To
achieve that, we add a new flag is_prepared_ok. Unless the flag is set,
the iterator will skip prepared statements even if they are visible from
the iterator read view. Upon skipping a statement, the iterator updates
min_skipped_plsn if the LSN of the skipped statement is less. We'll use
this LSN to update the transaction read view accordingly.

Needed for #5522

NO_DOC=internal
NO_CHANGELOG=internal

unit/vy_mem:
 - Remove the code creating unused lsregion.
 - Make test key_def and tuple_format global variables.
 - Replace assert() with fail().

unit/vy_cache:
 - Add missing test plan.

both:
 - Move history_node_pool to test/unit/vy_iterator_helpers.c.

Needed for #5522

NO_DOC=test
NO_TEST=test
NO_CHANGELOG=test

If index.get is called outside a transaction, we use the global read
view for it and set tx to NULL. This works fine for now, but may result
in dirty reads in a single statement, because prepared but not yet
committed to WAL statements are visible in the global read view. We are
planning to fix it in the tx manager. Let's make index.get create a
dummy transaction so once we fix it, index.get will always return
committed statements.

Note, index.pairs already creates a dummy transaction if called
outside a transaction (see vinyl_index_create_iterator) so this patch
makes behavior consistent across both read paths.

Needed for #5522

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

There was a bug that an instance could ack a transaction from an
old Raft term thus allowing the old leader to CONFIRM it, even if
that first instance knew there is a newer Raft term going on.

As a result, the old leader could write CONFIRM even if there is
already a new leader elected and the synchro quorum was > half.
That led to split-brain, when bad txn reached the new leader, and
PROMOTE reached the old leader.

Split-brain here is totally unnecessary. If the quorum is correct,
synchro timeout is infinite, and there is no async transactions,
then split-brain shouldn't ever happen.

The fix is as simple as attach the current Raft term number to
applier heartbeats.

In the testcase above if terms are attached, the old leader gets
ACK + new term. That causes the old leader freeze even if the
pending txn got quorum. The old leader can't CONFIRM nor ROLLBACK
its pending txns until a new leader is elected.

Freeze is guaranteed, because if a new leader was elected, then it
had got votes from > half cluster. It means > half nodes have the
new term. That in turn means the old leader during collecting ACKs
for its "new" txn will get the new term number from at least one
replica.

When the new leader finished writing PROMOTE, it either confirms
or rolls back the txn of the old leader (depending on whether it
has reached the new leader before promotion). Neither result
causes split brain. The rollback only causes a non-critical error
on the old leader raised by the bad txn's commit attempt.

There were some alternatives considered. One of the most promising
ones was to make instances reject txns if they see these txns
coming from an instance having an old Raft term. It would help in
the test provided above. But wouldn't do in a more complicated
test, when there is a third node which gets the bad transaction,
then gets local term bumped, and then replicates to any other
instance. Others would accept that bad txn, because the sender has
a newer Raft term, even though the txn author is still in the old
term. Tracking terms of txn author is not possible in too many
cases so as to rely on that.

Closes #7253

@TarantoolBot document
Title: New iproto field in applier -> relay ACKs
The applier->relay channel (from replica back to master) is used
only for sending ACKs. Replication data goes the other way
(relay->applier).

These ACKs had 2 fields: `IPROTO_VCLOCK (0x26)` and
`IPROTO_VCLOCK_SYNC (0x5a)`.

Now they have a new field: `IPROTO_TERM (0x53)`. It is a unsigned
number containing `box.info.election.term` of the sender node
(applier, replica).

The function play_wal_until_synchro_queue_is_busy() was used in a
few tests copy-pasted since it was considered to be too specific
for a few rare tests. But apparently it is going to be used again
in a new test in a future commit.

The patch makes this function a method of server object to reuse
it properly.

Needed for #7253

NO_DOC=refactoring
NO_CHANGELOG=refactoring

Currently if a non-string type is passed to luaT_key_def_set_part,
lua_tolstring returns null-pointer type_name, which is passed to
a string formatting function in diag_set.

Closes #5222

NO_DOC=bugfix

Don't accept an empty string or leading part of "str" or "num" as a
valid field type.

Closes #5940

NO_DOC=Partial field types weren't documented

Co-authored-by: Alexander Turenko <alexander.turenko@tarantool.org>

Since the function is actually an eval, by default there should
be no execute access right in public role.

Closes tarantool/security#14

NO_DOC=bugfix