Extend Tarantool kernel internal API with the call
`tarantool.debug.getsources()` to allow to retrieve sources
of a Tarantool `builtin/*` modules to show them in the
debugger shell.

Created simple luatest script for checking consistency
of a values returned from `require 'tarantool'.debug.getsources()`
and an ctual script file content we expected to receive.

NO_DOC=see future commit
NO_CHANGELOG=see future commit

Use 3rd-party module https://github.com/slembcke/debugger.lua
as a basis for our debugger CLI shell, rename it to
luadebug.lua to avoid collision with vscode and JetBrains
debugger helper scripts.

NO_DOC=yet
NO_CHANGELOG=yet
NO_TEST=yet

The _vfunc system space is the sysview for the _func system space.
However, the _vfunc format is different from the _func format. This
patch makes the _vfunc format the same as the _func format.

Closes #7822

NO_DOC=bugfix

Allocation of URI parameters and their values dynamic arrays is done
inefficiently: they are reallocated each time a new parameter or parameter
value is added — grow them exponentially instead.

`struct uri_param` and `struct uri` are exposed in Lua via FFI
(see src/lua/uri.lua): add warnings about the necessity of reflecting
changes to them in `ffi.cdecl`.

Closes #7155

NO_DOC=optimization
NO_TEST=optimization

We used to ignore timezone difference (in `tzoffset`) for
datetime subtraction operation:

```
tarantool> datetime.new{tz='MSK'} - datetime.new{tz='UTC'}
---
- +0 seconds
...

tarantool> datetime.new{tz='MSK'}.timestamp -
           datetime.new{tz='UTC'}.timestamp
---
- -10800
...
```

Now we accumulate tzoffset difference in the minute component
of a resultant interval:

```
tarantool> datetime.new{tz='MSK'} - datetime.new{tz='UTC'}
---
- -180 minutes
...
```

Closes #7698

NO_DOC=bugfix

We did not take into consideration the fact that
as result of date/time arithmetic we could get
in a different timezone, if DST boundary has been
crossed during operation.

```
tarantool> datetime.new{year=2008, month=1, day=1,
			tz='Europe/Moscow'} +
	   datetime.interval.new{month=6}
---
- 2008-07-01T01:00:00 Europe/Moscow
...
```

Now we resolve tzoffset at the end of operation if
tzindex is not 0.

Fixes #7700

NO_DOC=bugfix

Currently, in case of recovery from an old snapshot, Tarantool allows to
perform DDL operations on an instance with non-upgraded schema.
It leads to various unpredictable errors (because the DDL code assumes
that the schema is already upgraded). This patch forbids the following
operations unless the user has the most recent schema version:
- box.schema.space.create
- box.schema.space.drop
- box.schema.space.alter
- box.schema.index.create
- box.schema.index.drop
- box.schema.index.alter
- box.schema.sequence.create
- box.schema.sequence.drop
- box.schema.sequence.alter
- box.schema.func.create
- box.schema.func.drop

Closes #7149

NO_DOC=bugfix

By default a user might not have privileges to access the _schema space,
that will cause an error during schema_needs_upgrade(), which calls
get_version(). Fix this by using C variable dd_version_id, which is
updated in the _schema.version replace trigger.

There's a special case for upgrade() during bootstrap() - triggers are
disabled during bootstrap, that's why dd_version_id is not being updated.
Handle this by passing _initial_version=1.7.5 to the upgrade function.

Part of #7149

NO_DOC=internal
NO_CHANGELOG=internal

This patch fixed the assertion when JOIN uses index of unsupported type.

Closes #5678

NO_DOC=bugfix

This commit adds support of transaction isolation levels introduced
earlier for memtx mvcc by commit ec750af6 ("txm: introduce
transaction isolation levels"). The isolation levels work exactly in
the same way as in memtx:

 - Unless a transaction explicitly specifies the 'read-committed'
   isolation level, it'll skip prepared statements, even if they are
   visible from its read view. The background for this was implemented
   in the previous patches, which added the is_prepared_ok flag to
   cache and mem iterators.

 - If a transaction skips a prepared statement, which would otherwise be
   visible from its read view, it's sent to the most recent read view
   preceding the prepared statement LSN. Note, older prepared statements
   are still visible from this read view and can actually be selected if
   committed later.

 - A transaction using the 'best-effort' isolation level (default) is
   switched to 'read-committed' when it executes the first write
   statement.

The implementation is tested by the existing memtx mvcc tests that were
made multi-engine in the scope of this commit. However, we add one more
test case - the one that checks that a 'best-effort' read view is
properly updated in case there is more than one prepared transaction.
Also, there are few tests that relied upon the old implementation and
assumed that select from Vinyl may return unconfirmed tuples. We update
those tests here as well.

Closes #5522

NO_DOC=already documented

To implement read-confirmed and best-effort isolation levels, we need
to skip unconfirmed (aka prepared) statements in the cache iterator. To
achieve that, we add a new flag is_prepared_ok. Unless the flag is set,
the iterator will skip prepared statements even if they are visible from
the iterator read view. Note, in contrast to the mem iterator, we don't
need to keep track of the min skipped statement LSN, because the cache
is just a view of the underlying levels so we'll find it out when we
descend to the mem level.

Needed for #5522

NO_DOC=internal
NO_CHANGELOG=internal

To implement read-confirmed and best-effort isolation levels, we need
to skip unconfirmed (aka prepared) statements in the mem iterator. To
achieve that, we add a new flag is_prepared_ok. Unless the flag is set,
the iterator will skip prepared statements even if they are visible from
the iterator read view. Upon skipping a statement, the iterator updates
min_skipped_plsn if the LSN of the skipped statement is less. We'll use
this LSN to update the transaction read view accordingly.

Needed for #5522

NO_DOC=internal
NO_CHANGELOG=internal

This commit replaces vy_tx_manager::last_prepared_tx with a list of all
prepared transactions sorted by PSN. We need this list to find the
newest prepared transaction that is older than a given PSN, which is
required to switch to the most recent read view in the 'best-effort'
isolation level when an unconfirmed (prepared) statement is skipped.

Needed for #5522

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

Unconfirmed statements have lsn >= MAX_LEN. Let's add a helper function
that would check this.

Needed for #5522

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

If index.get is called outside a transaction, we use the global read
view for it and set tx to NULL. This works fine for now, but may result
in dirty reads in a single statement, because prepared but not yet
committed to WAL statements are visible in the global read view. We are
planning to fix it in the tx manager. Let's make index.get create a
dummy transaction so once we fix it, index.get will always return
committed statements.

Note, index.pairs already creates a dummy transaction if called
outside a transaction (see vinyl_index_create_iterator) so this patch
makes behavior consistent across both read paths.

Needed for #5522

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

With libunwind/libunwind@f67ef28 we can now use `unw_backtrace` with
PAC enabled on AARCH64 and remove the workaround with glibc's `backtrace`
for this case.

Closes #7285

NO_CHANGELOG=refactoring
NO_DOC=refactoring
NO_TEST=refactoring

There was a bug that an instance could ack a transaction from an
old Raft term thus allowing the old leader to CONFIRM it, even if
that first instance knew there is a newer Raft term going on.

As a result, the old leader could write CONFIRM even if there is
already a new leader elected and the synchro quorum was > half.
That led to split-brain, when bad txn reached the new leader, and
PROMOTE reached the old leader.

Split-brain here is totally unnecessary. If the quorum is correct,
synchro timeout is infinite, and there is no async transactions,
then split-brain shouldn't ever happen.

The fix is as simple as attach the current Raft term number to
applier heartbeats.

In the testcase above if terms are attached, the old leader gets
ACK + new term. That causes the old leader freeze even if the
pending txn got quorum. The old leader can't CONFIRM nor ROLLBACK
its pending txns until a new leader is elected.

Freeze is guaranteed, because if a new leader was elected, then it
had got votes from > half cluster. It means > half nodes have the
new term. That in turn means the old leader during collecting ACKs
for its "new" txn will get the new term number from at least one
replica.

When the new leader finished writing PROMOTE, it either confirms
or rolls back the txn of the old leader (depending on whether it
has reached the new leader before promotion). Neither result
causes split brain. The rollback only causes a non-critical error
on the old leader raised by the bad txn's commit attempt.

There were some alternatives considered. One of the most promising
ones was to make instances reject txns if they see these txns
coming from an instance having an old Raft term. It would help in
the test provided above. But wouldn't do in a more complicated
test, when there is a third node which gets the bad transaction,
then gets local term bumped, and then replicates to any other
instance. Others would accept that bad txn, because the sender has
a newer Raft term, even though the txn author is still in the old
term. Tracking terms of txn author is not possible in too many
cases so as to rely on that.

Closes #7253

@TarantoolBot document
Title: New iproto field in applier -> relay ACKs
The applier->relay channel (from replica back to master) is used
only for sending ACKs. Replication data goes the other way
(relay->applier).

These ACKs had 2 fields: `IPROTO_VCLOCK (0x26)` and
`IPROTO_VCLOCK_SYNC (0x5a)`.

Now they have a new field: `IPROTO_TERM (0x53)`. It is a unsigned
number containing `box.info.election.term` of the sender node
(applier, replica).

New term handling from a remote instance is moved into a new
function raft_process_term(). It is going to be used in a next
commit to handle incoming bare terms, not full raft_msgs.

Part of #7253

NO_DOC=refactoring
NO_CHANGELOG=refactoring
NO_TEST=refactoring

The patch adds structs applier_heartbeat and relay_heartbeat.

It is similar to the previous commit, which does the same for all
the other replication requests, but these work not via struct
replication_request.

The heartbeats have their own codecs. The goal is to avoid the big
expensive replication_request usage with a lot of optional fields.
It makes sense, because the HBs are sent somewhat frequently.

They also store members by values. But copying before encoding and
decoding is avoided by storing the HB structs by values right
inside applier and relay.

The motivation is to simplify addition of a new field to
applier_heartbeat in a next commit.

Needed for #7253

NO_DOC=refactoring
NO_CHANGELOG=refactoring
NO_TEST=refactoring

There were 3 comments which mentioned relay stream restart. It
could happen when replicas ignored data from non-leaders. A newly
elected leader had to re-send data, which others ignored, while
the source wasn't a leader.

Was removed in the commit 6336ea26
("refactoring: remove relay_restart_recovery as unneeded").

This is fixed now, because otherwise some of these comments would
need to be updated by a next commit in this series.

In scope of #7253

NO_DOC=refactoring
NO_CHANGELOG=refactoring
NO_TEST=refactoring

The patch adds `struct <name>_request` types for replication
requests: join, subscribe, register. Only not for ack/heartbeat
requests. They are covered separately in a next commit.

The motivation is to make them independent. Previously a lot of
codecs were calling xrow_encode/decode_subscribe() under the hood
with tons of optional params.

Now inside they all fallback to internal
`struct replication_request`. The code duplication is reduced,
because all encoding/decoding is handled by replication_request.
Also alteration of any of the requests only requires to update
that request itself and replication_request codec.

Additionally, this is consistent with other requests having their
own structs like synchro_request, raft_request, call_request, etc.

Alongside the commit fixes xrow_decode_register(), which used to
try to decode version_id, but the result wasn't used and the
version_id wasn't encoded in xrow_encode_register() anyway.

This work is done now, because soon replication acks will get a
new field and it wouldn't be good to update all this codecs mess
again just for that field.

The downside is that the requests store their members by values.
That requires to copy them before encoding. Not good for big
fields like vclocks. OTOH the requests affected by this commit are
sent very rarely, perf is not important at all.

Needed for #7253

NO_DOC=refactoring
NO_CHANGELOG=refactoring
NO_TEST=refactoring

Currently if a non-string type is passed to luaT_key_def_set_part,
lua_tolstring returns null-pointer type_name, which is passed to
a string formatting function in diag_set.

Closes #5222

NO_DOC=bugfix

Don't accept an empty string or leading part of "str" or "num" as a
valid field type.

Closes #5940

NO_DOC=Partial field types weren't documented

Co-authored-by: Alexander Turenko <alexander.turenko@tarantool.org>

Since the function is actually an eval, by default there should
be no execute access right in public role.

Closes tarantool/security#14

NO_DOC=bugfix

Prior to this patch, it was possible to call box.execute() before box
was initialized, i.e. before calling box.cfg(). This, however, caused
box.cfg() to be called automatically, which could be problematic as some
parameters could not be changed after box.cfg() was called. After this
patch, box.execute() will only be available when the box has been
initialized.

Closes #4726

@TarantoolBot document
Title: box.execute() now available only after initialization of box

Previously, it was possible to call box.execute() before the box was
configured, in which case the box was configured automatically, which
could lead to problems with box parameters. Now box.execute() can only
be called after the box has been properly configured.

It is also forbidden to set language to SQL in a console with an
unconfigured box.

Fix a simple typo that caused the problem.

Closes #7645

NO_DOC=bugfix

This patch introduces new rules to determine type of NULLIF() built-in
function.

Closes #6990

@TarantoolBot document
Title: New rules to determine type of result of NULLIF

The type of the result of NULLIF() function now matches the type of the
first argument.

This patch introduces new rules to determine type of CASE operation.

Part of #6990

@TarantoolBot document
Title: New rules to determine type of result of CASE

New rules are applied to determine the type of the CASE operation. If
all values are NULL with no type, or if a bind variable exists among
the possible results, then the type of CASE is ANY. Otherwise, all NULL
values with no type are ignored, and the type of CASE is determined
using the following rules:
1) if all values of the same type, then type of CASE is this type;
2) otherwise, if any of the possible results is of one of the
incomparable types, then the type of CASE is ANY;
3) otherwise, if any of the possible results is of one of the
non-numeric types, then the type of CASE is SCALAR;
4) otherwise, if any of the possible results is of type NUMBER, then the
type of CASE is NUMBER;
5) otherwise, if any of the possible results is of type DECIMAL, then
the type of CASE is DECIMAL;
6) otherwise, if any of the possible results is of type DOUBLE, then the
type of CASE is DOUBLE;
7) otherwise the type of CASE is INTEGER.

The commit c1c77782 ("replication: fix bootstrap failing with
ER_READONLY") made applier retry connection infinitely upon receiving a
ER_READONLY error on join. At the time of writing that commit, this was
the only way to make join retriable. Because there were no retries in
scope of bootstrap_from_master. The join either succeeded or failed.

Later on, bootstrap_from_master was made retriable in commit
f2ad1dee ("replication: retry join automatically"). Now when
bootstrap_from_master fails, replica reconnects to all the remote nodes,
thus updating their ballots, chooses a new (probably different from the
previous approach) bootstrap leader, and retries booting from it.

The second approach is more preferable, and here's why. Imagine
bootstrapping a cluster of 3 nodes, A, B and C in a full-mesh topology.
B and C connect to all the remote peers almost instantly, and both
independently decide that B will be the bootstrap leader (it means it
has the smallest uuid among A, B, C).

At the same time, A can't connect to C. B bootstraps the cluster, and
joins C. After C is joined, A finally connects to C. Now A can choose a
bootstrap leader. It has an old B's ballot (smallest uuid, but not yet
booted) and C's ballot (already booted). This is because C's ballot is
received after cluster bootstrap, and B's ballot was received earlier
than that. So A believes C is a better bootstrap leader, and tries to
boot from it.

A will fail joining to C, because at the same time C tries to sync with
everyone, including A, and thus stays read-only. Since A retries joining
to the same instance over and over again, this situation makes the A and
C stuck forever.

Let's retry ER_READONLY on another level: instead of trying to join to
the same bootstrap leader over and over, try to choose a new bootstrap
leader and boot from it.

In the situation described above, this means that A would try to join to
C once, fail due to ER_READONLY, re-fetch new ballots from everyone and
choose B as a join master (now it has smallest uuid and is booted).

The issue was discovered due to linearizable_test.lua hanging
occasionally with the following output:
NO_WRAP
 No output during 40 seconds. Will abort after 320 seconds without output. List of workers not reporting the status:
- 059_replication-luatest [replication-luatest/linearizable_test.lua, None] at /tmp/t/059_replication-luatest/linearizable.result:0
[059] replication-luatest/linearizable_test.lua                       [ fail ]
[059] Test failed! Output from reject file /tmp/t/rejects/replication-luatest/linearizable.reject:
[059] TAP version 13
[059] 1..6
[059] # Started on Thu Sep 29 10:30:45 2022
[059] # Starting group: linearizable-read
[059] not ok 1	linearizable-read.test_wait_others
[059] #   ....11.0~entrypoint.531.dev/test/luatest_helpers/server.lua:104: Waiting for "readiness" on server server_1-q7berSRY4Q_E (PID 53608) timed out
[059] #   stack traceback:
[059] #   	....11.0~entrypoint.531.dev/test/luatest_helpers/server.lua:104: in function 'wait_for_readiness'
[059] #   	...11.0~entrypoint.531.dev/test/luatest_helpers/cluster.lua:92: in function 'start'
[059] #   	...t.531.dev/test/replication-luatest/linearizable_test.lua:50: in function <...t.531.dev/test/replication-luatest/linearizable_test.lua:20>
[059] #   	...
[059] #   	[C]: in function 'xpcall'
NO_WRAP

Part-of #7737

NO_DOC=bugfix

This patch fixed the assertion when using INDEXED BY with an index that
is at least the third in space.

Closes #5976

NO_DOC=bugfix

If the length of the tuple is greater than the number of fields in the
format, it is possible that the cursor in the VDBE will be overridden
with zeros.

Closes #5310

NO_DOC=bugfix

dt.new() will raise a clear error on wrong timestamp type.

Closes #7273

NO_DOC=bugfix
NO_CHANGELOG=bugfix

For reasons described in #7231 HASH index 'GT' iterator type is deprecated:
print a warning exactly once about the deprecation.

Closes #7231

@TarantoolBot document
Title: memtx HASH index 'GT' iterator deprecation

memtx HASH index 'GT' iterator is deprecated since Tarantool 2.11
(tarantool/tarantool#7231) and will removed in a future release of
Tarantool: the user will get a warning when using it.

It's incorrect to create a tuple format when a read view is created,
because the tuple format registry isn't MT-safe so accessing a tuple
format created in the main thread from another thread by id (for
tuple_format(tuple)) may not work as expected. To address this issue,
we'll introduce a thread-local tuple format-registry. As a preparation
for this, let's create the tuple format when a read view is activated
by the thread that's going to use it (see the EE repository) while on
read view creation we now create only a field definition array necessary
to construct the format.

Needed for https://github.com/tarantool/tarantool-ee/issues/247

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

A read view must be activated in the thread that's going to use it.
This is required to support space upgrade and tuple format, which are
only used in EE. "Raw" index read view methods (returning raw message
pack data) don't require any activation by design so let's move read
view activate/deactivate logic completely to the EE repository.

Needed for https://github.com/tarantool/tarantool-ee/issues/247

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Commit 7f2c7609 ("box: add option to disable Lua FFI for read
operations") added an internal configuration option that switches all
index read operations from FFI to Lua C, which was necessary to
implement space upgrade in the EE repository. Turns out there was a
minor bug in that commit - when FFI is disabled, method usage errors
(raised when object.method is called instead of object:method) stop
working. Fix this bug and add an extensive test that checks that Lua C
and FFI implementations of index read operations are equivalent.

Fixes https://github.com/tarantool/tarantool-ee/issues/254

NO_DOC=bug fix
NO_CHANGELOG=bug manifests itself in only in EE

Concurrent transactions can try to insert tuples that intersect only by
parts of secondary index: in this case when one of them gets prepared, the
others get conflicted, but the committed story does not get retained
(because the conflicting statements are not added to the committed story's
delete statement list as opposed to primary index) and is lost after
garbage collection: retain stories if there is a newer uncommitted story
in the secondary indexes' history chain.

Closes #7712

NO_DOC=bugfix

In most places (schema.lua for example) we use box.error() to throw
error after error return from C function. Function is supposed to set
diag and box.error() will throw error based on the diag. For
example (crypto.lua):

    local ctx = ffi.C.crypto_stream_new(algo, mode, direction)
    if ctx == nil then
        box.error()
    end

This code is nice in case C function does not forget to set a diag.

But we have a different usage idiom of box.error().

    local result = pcall(module.cfg)
    if not result and oldvals ~= nil then
        rollback_module(module, oldcfg, keys, oldvals)
        return box.error()
    end

How it can go wrong? module.cfg is not necessary a C function. If it
is a Lua function it can throw error without diag. And if diag is not
set then box.error() is NOOP. So if there is a bug in code then Lua
generates error and it will be hided in this idiom. Or code can
intentionally throw Lua error() and again it will be hided. Or diag can
be set in some other place before and error will not be hided but
instead we will receive incorrect diagnosis.

One can argue that all these cases are some kind of bugs. But in this
case it will be useful to have correct diagnosis given it does not
involve much effort.

In short let's abandon pcall/box.error usage and use pcall/error(err)
instead.

Follow-up https://github.com/tarantool/tarantool-ee/issues/200

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

Pagination for vinyl works exactly like in memtx: the feature is tested
by the existing test, which was providently created as multi-engine.

The idea is to set vy_read_iterator::last, which is used for restoring
the iterator after changes in the LSM-tree structure, to the iterator
position provided by the user at the time when the iterator is opened.
Note, we can't use vy_read_iterator::last for reporting the current
iterator position back to the user, because the iterator is closed on
the last iteration so as not to postpone freeing resources until
the Lua garbage collector kicks in, so we have to add a new member to
the vinyl_iterator struct for storing the current position and update
it after each call to vy_read_iterator_next().

Closes #7634

NO_DOC=later
NO_CHANGELOG=later

Add parts required to call actual flightrec reconfiguration.

Part of https://github.com/tarantool/tarantool-ee/issues/200

NO_DOC=in EE version
NO_CHANGELOG=in EE version