Export IPROTO constants and features to Lua: for now, simply copy-paste all
constants from `src/box/iproto_{constants, features}.h` — it would be nice
to generate them in the future (#7103).

Closes #7894

@TarantoolBot document
Title: export IPROTO constants and features to Lua
or the API description and usage exmaples, see:
* [design document](https://www.notion.so/tarantool/box-iproto-override-44935a6ac7e04fb5a2c81ca713ed1bce#dcaf854d2a9f4743ae25661c16528523);
* tarantool/tarantool#7894.

Closes #7639

@TarantoolBot document
Title: Pagination

Introduce pagination to memtx and vinyl trees. It allows to start iteration
after last previously selected tuple. Since tuples can be huge in size,
there is a new entity called `position` - opaque object that represents
position of tuple in index. It is encoded to base64 format and can be
transferred and stored without any issues with non-printable characters.

Lua API of pagination:
- `select`
Select is provided with two new options: `fetch_pos` and `after`.
Option `after` can take tuple (or table, representing it) or
`position`. If it is passed, iteration will begin after tuple which
is described by the option. If there is no tuple described by `after` in
index, iteration starts with a tuple that would have been preceded by a
described one. Empty string ("") or `box.NULL` can be passed as a start
position. Here is an example:
```lua
last_tuple = box.NULL
while true do
    tuples = s:select(key, {limit=1000, after=last_tuple})
    if #tuples == 0 then
        break
    end
    last_tuple = tuples[#tuples]
    process_data(tuples)
end
```

The second option is `fetch_pos` - one can fetch a `position` of last
selected tuple, it will be returned as the second value. If no tuples
were fetched, `position` will be `nil`.
The snippet can be simplified by using `fetch_pos` option:
```lua
pos = ""
while true do
    tuples, pos = s:select(key, {limit=1000, after=pos, fetch_pos=true})
    if pos == nil then
        break
    end
    process_data(tuples)
end
```
However, the benefits of using position are not only lower memory
consumption and easier syntax - position is the only way to paginate
over multikey and functional indexes - an error will be thrown when you
use option `after` with tuple because tuple has not enough information
to describe its position in such indexes.

-`tuple_pos`
Index has a new method `index:tuple_pos(tuple)`, it returns `position`
of passed tuple (or a table, representing a tuple) in this index, even
if there is no such tuple. Passed tuple must match format of the space.
Does not work with multikey and functional indexes - an error will be
thrown.
```lua
pos = ""
while true do
    tuples = s:select(key, {limit=1000, after=pos})
    if #tuples == 0 then
        break
    end
    last_tuple = tuples[#tuples]
    pos = s.index.pk:tuple_pos(last_tuple)
    process_data(tuples)
end
```

-`pairs`
Pairs is provided only with `after` option with the same semantics.

```lua
for _, tuple in s:pairs(10, {after={10, 5}} do
    process_tuple(tuple)
end
```

-IPROTO
IPROTO is provided with new keys:

0x2e - IPROTO_AFTER_POSITION - start iteration after passed
`position`. It has type MP_STR.

0x2f - IPROTO_AFTER_TUPLE - start iteration after passed tuple.
It has type MP_ARRAY.

0x1f - IPROTO_FETCH_POSITION - send position of last fetched tuple
in response. It has type MP_BOOL.

0x35 - IPROTO_POSITION - `position`, sent in response if
IPROTO_FETCH_POSITION is true. It has type MP_STR.

To start iteration from the beginning, one can send empty MP_STR as
IPROTO_AFTER_POSITION or send no `position` at all.

IPROTO_VERSION is bumped, new feature `pagination` is added.

-`net.box`
Net box select is provided with the same options `after` and
`fetch_pos`. It has the same behavior as in index select. The only
difference is format of returned values:
If option `buffer` is passed, the whole response is written to buffer,
only number of written bytes is returned.
If option `skip_header` is passed (if `skip_header` is passed, then
`buffer` is necessarily passed too), only IPROTO_DATA (without header)
is written in buffer, select returns number of bytes written as the
first value and `position` of last selected tuple as the second one.
If request is async and without buffer, then table with tuples is
returned if `fetch_pos` is false or nil (old behavior) and table with
table of tuples and `position` of last selected tuples is returned if
`fetch_pos` is true.
If no options, described above, are passed, then table with tuples is
returned as the first value and new `position` as the second one if
`fetch_pos` option is true.

Synchronous API:
```lua
pos = ""
while true do
    tuples, pos = conn.space.s:select(key, {limit=1000, after=pos,
        fetch_pos=true})
    if pos == nil then
        break
    end
    process_data(tuples)
end
```

Asynchronous API:
```lua
pos = ""
while true do
    ret = conn.space.s:select(key, {limit=1000, after=pos,
        fetch_pos=true, is_async=true})
    tuples = ret[1]
    pos = ret[2]
    if pos == nil then
        break
    end
    process_data(tuples)
end
```

NO_TEST=no changes

The patch introduces pagination to pairs: one can create iterator which
will start after specific tuple or position, described by option after.
Tuple (both cdata and array) and position (obtained by index:tuple_pos
or fetch_pos option of select) can be passed as option after.

NO_CHANGELOG=next commit
NO_DOC=next commit

Discovered by Coverity:

https://scan7.scan.coverity.com/reports.htm#v43693/p13437/fileInstanceId=130384650&defectInstanceId=18636974&mergedDefectId=1527607

Follow-up #5665

NO_TEST=nit
NO_CHANGELOG=nit
NO_DOC=nit

The test fails with:

  not ok 2 ...ty_or_nil_select.test_log_entry_presence_for_memtx_user_space
  gh_6539_log_user_space_empty_or_nil_select_test.lua:122: log must contain
  a critical entry about `box.space.test_memtx:select({0}, {limit = 1001,
  iterator = "ALL"})` call on a memtx user space
  expected: a value evaluating to true, actual: nil

Looks like it happens, because g.server:grep_log() is executed before
g.server:eval(call) completes writing to the log.
In other tests grep_log() is wrapped into t.helpers.retrying() in order
to avoid such issues, so do the same here.

May close tarantool/tarantool-qa#264

NO_DOC=test fix
NO_CHANGELOG=test fix

This commit adds the part number to the ER_WRONG_INDEX_PARTS error
message format to make it easier for the user to figure out what
went wrong.

Closes #7933

NO_DOC=verbosity

Currently, we include the bad field number into the error message
manually, using tt_sprintf, which looks cumbersome. For some fields,
we don't include it at all (e.g. invalid constraint definition).
Let's include it into the error message format instead.

Part of #7933

NO_DOC=verbosity
NO_CHANGELOG=later

Passing the error code to opts_decode complicates adding the space field
or index part number to the error message. Let's use the IllegalParams
error in opts_decode instead and set the proper ClientError at each call
site.

Needed for #7933

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Currently, we raise the generic ER_IDENTIFIER error while we're supposed
to use the error code passed to opts_decode for verbosity. Fix this.

Follow-up commit 1d00b544 ("box: check constraint name against
identifier rules").

Part of #7933

NO_DOC=verbosity
NO_CHANGELOG=later

We're planning to simplify the opts_decode function protocol: instead of
passing an error code to it and let it set ClientError, we'll rework it
so that it may only set IllegalParams error, which is supposed to be
converted to ClientError with a proper error code at the upper level.
This will let us append extra information, such as field number, to the
error.

The problem is, space_upgrade_def_decode sets ER_UNSUPPORTED error,
which is against the new protocol. Let's move the error setting to
space_upgrade_check_alter, which looks like a better place for it,
anyway. To achieve that, we make space_upgrade_def_new skip the input
and return BAD_PTR, which is used as a marker indicating that the space
upgrade options were set.

Needed for #7933

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

For some options (OPT_STRPTR, OPT_CUSTOM), opts_decode may need to
allocate the result from the fiber region. The allocation shouldn't
normally fail (if it fails, it means the system is misconfigured and it
doesn't make much sense to carry on; essentially, the fiber region is
used like a stack). Still, we try to handle memory allocation failures
gracefully there by returning OutOfMemory error. This complicates the
opts_decode function protocol: we can't use IllegalParams for
propagating errors and convert the error to ClientError with a proper
error code at the call site, because the error would look ridiculous on
OOM (for example, "Wrong space options: Failed to allocate XX bytes for
..."). Instead, we have to pass the error code for ClientError to
opts_decode, which, in turn, makes it impossible to set the proper space
field or index part number for the sake of verbosity.

To address this issue, let's use xregion_alloc instead of region_alloc
and remove all OutOfMemory errors from opts_decode itself as well as all
callbacks that may allocate memory from the region.

Needed for #7933

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

When pagination is used, we must not assume that the first result
returned by the read iterator is the first tuple matching the iterator
criteria, because there may actually be tuples between the search and
after keys. Set vy_read_iterator::is_cache_boundary accordingly when
the iterator is opened.

Fixes commit 3f026339 ("vinyl: implement iterator pagination").

Closes #7943

NO_DOC=bug fix
NO_CHANGELOG=bug fix for unreleased feature

The read iterator skips DELETE statements. If a skipped DELETE statement
was read from the transaction write set, the read iterator breaks the
current cache chain by clearing vy_read_iterator::last_cached, because
the deleted tuple may actually be present at a lower level so adding
a gap there until the transaction is committed would be incorrect, see
commit dd926790 ("vinyl: fix a bug in adding of a cache chain").

The problem is vy_cache_add() assumes that if last_cached is NULL
the tuple added to the cache must be the first tuple that meets the
iteration criteria and sets the cache node boundary level accordingly.
This may lead to the previous tuple being erroneously skipped later
in case the transaction that attempted to delete it is rolled back.

Fix this issue by passing the flag indicating if the current tuple is
the first result of the iteration explicitly to vy_cache_add().

Closes #7947

NO_DOC=bug fix

As it breaks sane usage of region as a data stack:

	size_t region_svp = region_used(&fiber()->gc);
	/* some allocation on fiber gc and usage of allocated memory. */
	region_truncate(&fiber()->gc, region_svp);

If in the above snippet one calls a function that in turn calls
`fiber_gc` then the snippet code may have use-after-free and later UB
on truncation.

For this reason let's get read of fiber_gc. However we need to make sure
we won't introduce leaks this way. So before actually removing
fiber_gc we make it perform leak check instead and only after fixing
all the leaks the fiber_gc was removed.

In order to find the leak easily the backtrace of the first fiber gc
allocation that is not truncated is saved and then reported.

In order to catch leaks that are not triggered by the current test suit
and to prevent introducing leaks in future patches the leak check is
added on fiber exit/recycle and for long living system fibers on every loop
iteration.

Leak check in release build is on but without leak backtrace info by
default for performance reasons. Backtrace can be provided by using
`fiber.leak_backtrace_enable()` knob before starting leaking fiber.

Regularly leaks are only reported in log but it will not help to
catch errors when running test suits so build option ABORT_ON_LEAK
is added. When it is on we abort on leak. This option is turned off
for all builds that used in CI.

Closes #5665

NO_CHANGELOG=internal
NO_DOC=internal

Currently in space_execute_dml we have some txn related objects
allocated on fiber region. Use txn region as in other places.

NO_DOC=internal
NO_CHANGELOG=internal
NO_TEST=internal

Currently `backtrace_snprint ` indirectly uses `tt_static_buf()` by
itself. As a result its callers cannot use `tt_static_buf()`. With
large enough backtrace stack size buffer passed to `backtrace_snprint`
will be overwritten inside `backtrace_frame_resolve` call.

Part of #5665

NO_DOC=internal
NO_CHANGELOG=internal
NO_TEST=internal

* Ensure correct stack top for OOM error message.
* x86/x64: Check for jcc when using xor r,r in emit_loadi().
* Save trace recorder state around VM event call.
* Fix io.close() error message.
* Fix io.close().
* Cleanup math function compilation and fix inconsistencies.

Closes #3840
Closes #6782
Part of #7230

NO_DOC=LuaJIT submodule bump
NO_TEST=LuaJIT submodule bump

If create_index('...', {parts = {...}}) is called without the `collation`
option, then use collation from the space format.

Closes #5104

@TarantoolBot document
Title: Index options are inherited from the space format
Root document: https://www.tarantool.io/en/doc/latest/reference/reference_lua/box_space/create_index/

If `is_nullable` and `collation` options are not set, then they are
inherited from the space format, if applicable for that field type.

Local spaces can be written to on any replica, even on a read-only one.
This makes sense, because local space data isn't replicated, so it can't
lead to a conflict or violate consistency anyhow.

However, when used together with synchronous replication, local spaces
can't be written to by anyone but the synchro queue owner:
```
tarantool> box.info.synchro.queue.len
---
- 1
...
tarantool> box.space.loc:replace{2}
---
- error: Found uncommitted sync transactions from other instance with id 1
...
```
Fix this and allow to put transactions touching local spaces to the
synchro queue even if it is claimed by someone else.

Note, we can't let local transactions bypass the synchro queue completely.
This would lead to consistency loss in a case when synchro queue
contains a transaction [sync_row, local_row], and another transcation
[local_row], probably based on the sync transaction, bypasses the limbo.

Closes #7592

NO_DOC=bugfix

txn_commit() and txn_commit_try_async() share the common logic of
determining if a transaction should go to the synchronous transaction
queue and then putting the transaction there.

Factor the common places out to a helper.

Part-of #7592

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

If you shut down the replica and delete all xlog files on it, and it
reconnect to the master, may occur the restore error. Because there are
intermediate xlog files between the snap on the replica and the last
xlog on the master. When restoring xlog files on the replica, it can be
occurred that prev_signature < signature and this will lead to the crash
of the master.

Closes #5158

NO_DOC=bugfix

It appears that we have to wait for commit completion on master before
start waiting for it on replica.

Closes tarantool/tarantool-qa#284

NO_DOC=test fix
NO_CHANGELOG=test fix

Added a description for the --format flag
of the tarantoolctl cat command.

Fixes #7099

NO_DOC=bugfix
NO_TEST=help message fix

Two more places where replica didn't wait until data is replicated.

Follow-up tarantool/tarantool-qa#274

NO_CHANGELOG=test fix
NO_DOC=test fix

Existence of txn_commit_ro_stmt assumes there is txn_rollback_ro_stmt,
but it does not exist. Instead, currently we use txn_rollback_stmt, which
has nothing to do with 'ro' statements, and will just rollback the
currently running normal statement.

As an example, consider box_index_iterator. It calls txn_begin_ro_stmt,
which does not produce any artefacts in an existing `struct txn`. But in
case of the iterator creation fail it calls txn_rollback_stmt, which will
rollback the currently being executed 'rw' statement.

Drop txn_commit_ro_stmt and introduce txn_end_ro_stmt, which is called
regardless of 'ro' statement result.

NO_DOC=bugfix

Closes #5501

`box_schema_version` symbol is currently exposed via `exports`, but it is
not part of the public API: export it to public API and Lua.

Retain `box.internal.schema_version` for backward compatibility and add
deprecation warning issued only once.

Closes #7904

@TarantoolBot document
Title: export `box_schema_version` to public API and Lua
`box_schema_version`, which returns the server's current monotonic schema
version, is now part of Tarantool public API, and is also accessible from
Lua:
```lua
box.info.schema_version
-- 81
```

Schema version is a number that is designed for simple detection of changes
in the database schema. It is guaranteed that if space/index/tuple API of
database is changed (any space/index is added/deleted or any name of
space/index/field is changed) - this value will grow. Another (weak)
guarantee is that the value usually remains the same if there were no
changes in system spaces (with id < 512), so usually it is changed rather
seldom. The value may grow when any other change in system spaces takes
place, but the exact behavior is unspecified and an application should not
rely on it. Using of this value inside of a transaction that makes such
changes is unspecified either.

IPROTO_SCHEMA_VERSION section of IPROTO response is set to this number.

The requirement of the schema version is that it monotonically grows: in
order to protect from wrapping, change its type and the type of everything
depending on it to `uint64_t`.

Needed for #7904

NO_CHANGELOG=enhancement
NO_DOC=enhancement
NO_TEST=enhancement

The patch introduces after and fetch_pos options to net.box select.
New options are equivalents of the same options of index:select.
Behavior of `is_async`, `buffer` and `skip_header` options will be
described in documentation. If server does not support pagination,
select with any pagination option will throw an error.

Closes #7637

NO_CHANGELOG=see later commits
NO_DOC=see later commits

The patch adds new iproto keys needed for pagination and extends
tx_process_select, used by IPROTO. IPROTO and NETBOX_IPROTO versions
are updated, iproto feature pagination is introduced, new keys are:

0x2e - IPROTO_AFTER_POSITION - start iteration after passed
iterator position. It has type MP_STR.

0x2f - IPROTO_AFTER_TUPLE - start iteration after passed tuple.
It has type MP_ARRAY.

0x1f - IPROTO_FETCH_POSITION - send position of last fetched tuple
in response. It has type MP_BOOL.

0x35 - IPROTO_POSITION - iterator position, sent in response if
IPROTO_FETCH_POSITION is true. It has type MP_STR.

Part of #7637

NO_CHANGELOG=see later commits
NO_DOC=see later commits

Currently AArch64 version of coro_transfer stores x19-x30 and d8-d15
registers to the stack, but only after that it updates the stack pointer.
If a SIGALRM signal is delivered during the execution of coro_transfer,
the signal handler will use the stack starting from current sp, thus
corrupting the saved registers.

Fix this by updating the stack pointer at the beginning of coro_transfer.
x2 register is still required, because `str sp, [x0, #0]` is invalid in
the A64 instruction set.

Closes #7484
Closes #7523

NO_DOC=bugfix
NO_TEST=Hard to create a stable reproducer,
        mostly covered by existing tests.

NO_DOC=fix protocol in link
NO_TEST=fix protocol in link
NO_CHANGELOG=fix protocol in link

The test fails with:

  master | 2022-11-11 09:03:32.093 [4128822] main/103/default.lua F>
  can't initialize storage: unlink, called on fd 30, aka unix/:(socket),
  peer of unix/:(socket): Address already in use

Looks like it happens, because both test cases share the socket path.
The fix is the same as in commit 3f86cd04 ("test: fix flaky
'test_ignore_with_force_recovery'") - use different socket paths.

Follow-up commit b2dab5f4 ("memtx: log bad row on snapshot recovery
error").

NO_DOC=test fix
NO_CHANGELOG=test fix

* ci: add workflow for legacy CMake GNU Make build
* build: fix build with the original Makefile
* ci: drop integration for macOS with disabled GC64
* ci: add 'self-hosted' tag to runs-on
* ci: use Ninja generator in CI workflows
* ci: use CMAKE_EXTRA_PARAMS in LuaJIT integration
* test: replace result variable in MakeLuaPath.cmake
* ci: merge Linux and macOS workflows
* ci: merge x86_64 and ARM64 workflows
* ci: remove arch prefix for macOS M1 workflow
* ci: remove excess parallel level setup
* ci: use out of source build in GitHub Actions
* test: fix tarantool suite for out of source build
* test: introduce MakeLuaPath.cmake helper
* test: introduce utils.profilename helper

NO_DOC=LuaJIT submodule bump
NO_TEST=LuaJIT submodule bum
NO_CHANGELOG=ci

When replica heartbeat is read while another heartbeat is en route, it's
delivery is delayed for a full replication_timeout.

This can be seen in replication/qsync_basic.test.lua occasional hangs on
these lines:
```
--
-- gh-5100: replica should send ACKs for sync transactions after
-- WAL write immediately, not waiting for replication timeout or
-- a CONFIRM.
--
-- on replica:
box.cfg{replication_timeout = 1000, replication_synchro_timeout = 1000}
test_run:switch('default')
box.cfg{replication_timeout = 1000, replication_synchro_timeout = 1000}
<factored-out>
-- Now commit something sync. It should return immediately even
-- though the replication timeout is huge.
box.space.sync:replace{4}

```

Change that so a pending heartbeat is sent immediately upon status
message return from tx thread.

Closes #7869

NO_TEST=hard to test
NO_DOC=bugfix
NO_CHANGELOG=not user-visible

Fun fact: our self-hosted macOS runner has the same name as the one
provided by GitHub. Hence sometimes when no self-hosted runners are
available, the public GitHub one is chosen.

This patch enforces LuaJIT integration workflow to use only self-hosted
runner by explicitly specifying this in runs-on section.

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

Reviewed-by: Yaroslav Lobankov <y.lobankov@tarantool.org>
Signed-off-by: Igor Munkin <imun@tarantool.org>

We log bad row on xlog recovery error. We should log bad row on snapshot
recovery error ase well to ease debugging.

Closes #7917

NO_DOC=bug fix

fh:pread() will raise a clear error on wrong arguments.

Closes #4963

NO_DOC=bugfix
NO_CHANGELOG=bugfix

If we find a newer story in the secondary index, the statement of which
deletes a story added in the same transaction, there can be two cases: if
the secondary index is not unique, then it is conflicted by primary index
(since it's `cmp_def` is extended with the primary index's
`key_def`, every history chain in the secondary index has a corresponding
chain in the primary index). Otherwise, the prepared story must also
conflict the newer one.

Summarizing the above: remove the unnecessary check.

Closes #7761

NO_DOC=bugfix

Gap trackers are stored in the story at the top of the history chain (see
`memtx_tx_story_{un}link_top_light`): following this logic, during rollback
we need to rebind them to the older story. We already maintain this in
56cf737c, since a story is not retained iff it has a newer story (which
means it does not store trackers) or if it has an older story (in which
case rebinding is done in `memtx_tx_story_{un}link_top_light`): add an
assertion that a story deleted during rollback does not have any gap
trackers and a test for this case.

Follow-up 56cf737c

NO_CHANGELOG=internal
NO_DOC=internal

`memtx_tx_story_unlink_both` is called in two separate contexts: on space
delete and on rollback. In the former case we need to simply unlink the
story, while in the latter case we need to rebind read and gap trackers,
and, perhaps do some other logic in the future. Calling
`memtx_tx_story_unlink_both` in the former context can trigger assertion:
split the function and its helpers into two separate functions for each
case, grouping the common logic into third `*_common` functions.

Closes #7757

NO_DOC=bugfix