The read iterator skips DELETE statements. If a skipped DELETE statement
was read from the transaction write set, the read iterator breaks the
current cache chain by clearing vy_read_iterator::last_cached, because
the deleted tuple may actually be present at a lower level so adding
a gap there until the transaction is committed would be incorrect, see
commit dd926790 ("vinyl: fix a bug in adding of a cache chain").

The problem is vy_cache_add() assumes that if last_cached is NULL
the tuple added to the cache must be the first tuple that meets the
iteration criteria and sets the cache node boundary level accordingly.
This may lead to the previous tuple being erroneously skipped later
in case the transaction that attempted to delete it is rolled back.

Fix this issue by passing the flag indicating if the current tuple is
the first result of the iteration explicitly to vy_cache_add().

Closes #7947

NO_DOC=bug fix

As it breaks sane usage of region as a data stack:

	size_t region_svp = region_used(&fiber()->gc);
	/* some allocation on fiber gc and usage of allocated memory. */
	region_truncate(&fiber()->gc, region_svp);

If in the above snippet one calls a function that in turn calls
`fiber_gc` then the snippet code may have use-after-free and later UB
on truncation.

For this reason let's get read of fiber_gc. However we need to make sure
we won't introduce leaks this way. So before actually removing
fiber_gc we make it perform leak check instead and only after fixing
all the leaks the fiber_gc was removed.

In order to find the leak easily the backtrace of the first fiber gc
allocation that is not truncated is saved and then reported.

In order to catch leaks that are not triggered by the current test suit
and to prevent introducing leaks in future patches the leak check is
added on fiber exit/recycle and for long living system fibers on every loop
iteration.

Leak check in release build is on but without leak backtrace info by
default for performance reasons. Backtrace can be provided by using
`fiber.leak_backtrace_enable()` knob before starting leaking fiber.

Regularly leaks are only reported in log but it will not help to
catch errors when running test suits so build option ABORT_ON_LEAK
is added. When it is on we abort on leak. This option is turned off
for all builds that used in CI.

Closes #5665

NO_CHANGELOG=internal
NO_DOC=internal

Currently in space_execute_dml we have some txn related objects
allocated on fiber region. Use txn region as in other places.

NO_DOC=internal
NO_CHANGELOG=internal
NO_TEST=internal

Currently `backtrace_snprint ` indirectly uses `tt_static_buf()` by
itself. As a result its callers cannot use `tt_static_buf()`. With
large enough backtrace stack size buffer passed to `backtrace_snprint`
will be overwritten inside `backtrace_frame_resolve` call.

Part of #5665

NO_DOC=internal
NO_CHANGELOG=internal
NO_TEST=internal

* Ensure correct stack top for OOM error message.
* x86/x64: Check for jcc when using xor r,r in emit_loadi().
* Save trace recorder state around VM event call.
* Fix io.close() error message.
* Fix io.close().
* Cleanup math function compilation and fix inconsistencies.

Closes #3840
Closes #6782
Part of #7230

NO_DOC=LuaJIT submodule bump
NO_TEST=LuaJIT submodule bump

If create_index('...', {parts = {...}}) is called without the `collation`
option, then use collation from the space format.

Closes #5104

@TarantoolBot document
Title: Index options are inherited from the space format
Root document: https://www.tarantool.io/en/doc/latest/reference/reference_lua/box_space/create_index/

If `is_nullable` and `collation` options are not set, then they are
inherited from the space format, if applicable for that field type.

Local spaces can be written to on any replica, even on a read-only one.
This makes sense, because local space data isn't replicated, so it can't
lead to a conflict or violate consistency anyhow.

However, when used together with synchronous replication, local spaces
can't be written to by anyone but the synchro queue owner:
```
tarantool> box.info.synchro.queue.len
---
- 1
...
tarantool> box.space.loc:replace{2}
---
- error: Found uncommitted sync transactions from other instance with id 1
...
```
Fix this and allow to put transactions touching local spaces to the
synchro queue even if it is claimed by someone else.

Note, we can't let local transactions bypass the synchro queue completely.
This would lead to consistency loss in a case when synchro queue
contains a transaction [sync_row, local_row], and another transcation
[local_row], probably based on the sync transaction, bypasses the limbo.

Closes #7592

NO_DOC=bugfix

txn_commit() and txn_commit_try_async() share the common logic of
determining if a transaction should go to the synchronous transaction
queue and then putting the transaction there.

Factor the common places out to a helper.

Part-of #7592

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

If you shut down the replica and delete all xlog files on it, and it
reconnect to the master, may occur the restore error. Because there are
intermediate xlog files between the snap on the replica and the last
xlog on the master. When restoring xlog files on the replica, it can be
occurred that prev_signature < signature and this will lead to the crash
of the master.

Closes #5158

NO_DOC=bugfix

It appears that we have to wait for commit completion on master before
start waiting for it on replica.

Closes tarantool/tarantool-qa#284

NO_DOC=test fix
NO_CHANGELOG=test fix

Added a description for the --format flag
of the tarantoolctl cat command.

Fixes #7099

NO_DOC=bugfix
NO_TEST=help message fix

Two more places where replica didn't wait until data is replicated.

Follow-up tarantool/tarantool-qa#274

NO_CHANGELOG=test fix
NO_DOC=test fix

Existence of txn_commit_ro_stmt assumes there is txn_rollback_ro_stmt,
but it does not exist. Instead, currently we use txn_rollback_stmt, which
has nothing to do with 'ro' statements, and will just rollback the
currently running normal statement.

As an example, consider box_index_iterator. It calls txn_begin_ro_stmt,
which does not produce any artefacts in an existing `struct txn`. But in
case of the iterator creation fail it calls txn_rollback_stmt, which will
rollback the currently being executed 'rw' statement.

Drop txn_commit_ro_stmt and introduce txn_end_ro_stmt, which is called
regardless of 'ro' statement result.

NO_DOC=bugfix

Closes #5501

`box_schema_version` symbol is currently exposed via `exports`, but it is
not part of the public API: export it to public API and Lua.

Retain `box.internal.schema_version` for backward compatibility and add
deprecation warning issued only once.

Closes #7904

@TarantoolBot document
Title: export `box_schema_version` to public API and Lua
`box_schema_version`, which returns the server's current monotonic schema
version, is now part of Tarantool public API, and is also accessible from
Lua:
```lua
box.info.schema_version
-- 81
```

Schema version is a number that is designed for simple detection of changes
in the database schema. It is guaranteed that if space/index/tuple API of
database is changed (any space/index is added/deleted or any name of
space/index/field is changed) - this value will grow. Another (weak)
guarantee is that the value usually remains the same if there were no
changes in system spaces (with id < 512), so usually it is changed rather
seldom. The value may grow when any other change in system spaces takes
place, but the exact behavior is unspecified and an application should not
rely on it. Using of this value inside of a transaction that makes such
changes is unspecified either.

IPROTO_SCHEMA_VERSION section of IPROTO response is set to this number.

The requirement of the schema version is that it monotonically grows: in
order to protect from wrapping, change its type and the type of everything
depending on it to `uint64_t`.

Needed for #7904

NO_CHANGELOG=enhancement
NO_DOC=enhancement
NO_TEST=enhancement

The patch introduces after and fetch_pos options to net.box select.
New options are equivalents of the same options of index:select.
Behavior of `is_async`, `buffer` and `skip_header` options will be
described in documentation. If server does not support pagination,
select with any pagination option will throw an error.

Closes #7637

NO_CHANGELOG=see later commits
NO_DOC=see later commits

The patch adds new iproto keys needed for pagination and extends
tx_process_select, used by IPROTO. IPROTO and NETBOX_IPROTO versions
are updated, iproto feature pagination is introduced, new keys are:

0x2e - IPROTO_AFTER_POSITION - start iteration after passed
iterator position. It has type MP_STR.

0x2f - IPROTO_AFTER_TUPLE - start iteration after passed tuple.
It has type MP_ARRAY.

0x1f - IPROTO_FETCH_POSITION - send position of last fetched tuple
in response. It has type MP_BOOL.

0x35 - IPROTO_POSITION - iterator position, sent in response if
IPROTO_FETCH_POSITION is true. It has type MP_STR.

Part of #7637

NO_CHANGELOG=see later commits
NO_DOC=see later commits

Currently AArch64 version of coro_transfer stores x19-x30 and d8-d15
registers to the stack, but only after that it updates the stack pointer.
If a SIGALRM signal is delivered during the execution of coro_transfer,
the signal handler will use the stack starting from current sp, thus
corrupting the saved registers.

Fix this by updating the stack pointer at the beginning of coro_transfer.
x2 register is still required, because `str sp, [x0, #0]` is invalid in
the A64 instruction set.

Closes #7484
Closes #7523

NO_DOC=bugfix
NO_TEST=Hard to create a stable reproducer,
        mostly covered by existing tests.

NO_DOC=fix protocol in link
NO_TEST=fix protocol in link
NO_CHANGELOG=fix protocol in link

The test fails with:

  master | 2022-11-11 09:03:32.093 [4128822] main/103/default.lua F>
  can't initialize storage: unlink, called on fd 30, aka unix/:(socket),
  peer of unix/:(socket): Address already in use

Looks like it happens, because both test cases share the socket path.
The fix is the same as in commit 3f86cd04 ("test: fix flaky
'test_ignore_with_force_recovery'") - use different socket paths.

Follow-up commit b2dab5f4 ("memtx: log bad row on snapshot recovery
error").

NO_DOC=test fix
NO_CHANGELOG=test fix

* ci: add workflow for legacy CMake GNU Make build
* build: fix build with the original Makefile
* ci: drop integration for macOS with disabled GC64
* ci: add 'self-hosted' tag to runs-on
* ci: use Ninja generator in CI workflows
* ci: use CMAKE_EXTRA_PARAMS in LuaJIT integration
* test: replace result variable in MakeLuaPath.cmake
* ci: merge Linux and macOS workflows
* ci: merge x86_64 and ARM64 workflows
* ci: remove arch prefix for macOS M1 workflow
* ci: remove excess parallel level setup
* ci: use out of source build in GitHub Actions
* test: fix tarantool suite for out of source build
* test: introduce MakeLuaPath.cmake helper
* test: introduce utils.profilename helper

NO_DOC=LuaJIT submodule bump
NO_TEST=LuaJIT submodule bum
NO_CHANGELOG=ci

When replica heartbeat is read while another heartbeat is en route, it's
delivery is delayed for a full replication_timeout.

This can be seen in replication/qsync_basic.test.lua occasional hangs on
these lines:
```
--
-- gh-5100: replica should send ACKs for sync transactions after
-- WAL write immediately, not waiting for replication timeout or
-- a CONFIRM.
--
-- on replica:
box.cfg{replication_timeout = 1000, replication_synchro_timeout = 1000}
test_run:switch('default')
box.cfg{replication_timeout = 1000, replication_synchro_timeout = 1000}
<factored-out>
-- Now commit something sync. It should return immediately even
-- though the replication timeout is huge.
box.space.sync:replace{4}

```

Change that so a pending heartbeat is sent immediately upon status
message return from tx thread.

Closes #7869

NO_TEST=hard to test
NO_DOC=bugfix
NO_CHANGELOG=not user-visible

Fun fact: our self-hosted macOS runner has the same name as the one
provided by GitHub. Hence sometimes when no self-hosted runners are
available, the public GitHub one is chosen.

This patch enforces LuaJIT integration workflow to use only self-hosted
runner by explicitly specifying this in runs-on section.

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

Reviewed-by: Yaroslav Lobankov <y.lobankov@tarantool.org>
Signed-off-by: Igor Munkin <imun@tarantool.org>

We log bad row on xlog recovery error. We should log bad row on snapshot
recovery error ase well to ease debugging.

Closes #7917

NO_DOC=bug fix

fh:pread() will raise a clear error on wrong arguments.

Closes #4963

NO_DOC=bugfix
NO_CHANGELOG=bugfix

If we find a newer story in the secondary index, the statement of which
deletes a story added in the same transaction, there can be two cases: if
the secondary index is not unique, then it is conflicted by primary index
(since it's `cmp_def` is extended with the primary index's
`key_def`, every history chain in the secondary index has a corresponding
chain in the primary index). Otherwise, the prepared story must also
conflict the newer one.

Summarizing the above: remove the unnecessary check.

Closes #7761

NO_DOC=bugfix

Gap trackers are stored in the story at the top of the history chain (see
`memtx_tx_story_{un}link_top_light`): following this logic, during rollback
we need to rebind them to the older story. We already maintain this in
56cf737c, since a story is not retained iff it has a newer story (which
means it does not store trackers) or if it has an older story (in which
case rebinding is done in `memtx_tx_story_{un}link_top_light`): add an
assertion that a story deleted during rollback does not have any gap
trackers and a test for this case.

Follow-up 56cf737c

NO_CHANGELOG=internal
NO_DOC=internal

`memtx_tx_story_unlink_both` is called in two separate contexts: on space
delete and on rollback. In the former case we need to simply unlink the
story, while in the latter case we need to rebind read and gap trackers,
and, perhaps do some other logic in the future. Calling
`memtx_tx_story_unlink_both` in the former context can trigger assertion:
split the function and its helpers into two separate functions for each
case, grouping the common logic into third `*_common` functions.

Closes #7757

NO_DOC=bugfix

stailq_shift expects list to be non empty. Let's add assertion.

This patch addresses coverity complains 1526815 and 1526816 (dup).
https://scan7.scan.coverity.com/reports.htm#v43693/p13437/fileInstanceId=129122176&defectInstanceId=18601010&mergedDefectId=1526815

Closes #7913

NO_DOC=code health
NO_CHANGELOG=code health
NO_TEST=code health

We potentially pass negative size to safe_write in say_default if total
is negative. We can't hit this issue currently because formatting
functions currently do not return -1. Yet let's check total value where
appropriate.

By the way fix restoring errno in log_vsay. format_log_entry_full is
introduced for this purpuse as SNPRINT macro returns in case of error.

Also add assertion to safe_write.

This patch addresses coverity complain 1527154
(https://scan7.scan.coverity.com/reports.htm#v43693/p13437/fileInstanceId=129123623&defectInstanceId=18608452&mergedDefectId=1527154)

Part of #7913

NO_DOC=code health
NO_CHANGELOG=code health
NO_TEST=code health

After revisiting the list of currently disabled compiler warnings it seems
like the following are unjustifiably disabled:
* `-Wunknown-pragmas`;
* `-Wstrict-aliasing`;
* `-Wunused-value`;
* `-Wchar-subscripts`;
* `-Wformat-truncation`.

`-Wunused-value` appeared to be disabled due to a bug in Clang (see also
llvm/llvm-project#14119): according to the LLVM release dates, the last
affected Clang version was 3.1 and it was hopefully fixed in Clang 3.2.
No currently alive (or recently EOLed) distributions offer such old Clang
versions.

Strict aliasing violation warnings coming from libev cannot be disabled
with `system_header` pragma on GNU compilers older than 8, so we have to
leave to leave it disabled for the whole project in this case.

Otherwise, enable the warnings listed above for the sake of code health.

Closes #7862

NO_CHANGELOG=build
NO_DOC=build
NO_TEST=build

Fix strict aliasing rules violation by explicitly calling `memcpy`.

libev is treated in a special way: -Wstrict-aliasing warning are ignored
there, because:
1. libev authors claim that this is a false positive and their code is
correct in respect to aliasing rules: see
* http://lists.schmorp.de/pipermail/libev/2010q1/000943.html
* http://lists.schmorp.de/pipermail/libev/2015q2/002540.html
2. it's a common practice to ignore compiler warning in libev: see
* http://pod.tst.eu/http://cvs.schmorp.de/libev/ev.pod#COMPILER_WARNINGS
* https://github.com/tarantool/tarantool/blob/dec0e0221e183fa972efa65bb0fb658112f2196f/cmake/BuildLibEV.cmake#L6-L12

Ignoring `-Wstrict-aliasing` for libev using diagnostic pragmas does not
work, apparently, due to a bug in GNU: use the `system_header` pragma
instead — ignoring all compiler warnings coming from libev is a win-win
anyways.

Needed for #7862

NO_CHANGELOG=build
NO_DOC=build
NO_TEST=build

The path buffer's size is `PATH_MAX`, whilst we write two paths into
it, the latter (`dent->d_name`) also having a maximum size of `PATH_MAX`,
which triggers `-Wformat-truncation` compiler warning: check the return
value of `snprintf` and mimic a system error if the path got truncated.

Needed for #7862

NO_CHANGELOG=build
NO_DOC=build
NO_TEST=build

There is a big caveat in current implementation of position - it is
represented as a string with unprintable chars in Lua - if user will
need to save it to another database or pass to another client,
unprintable chars can be affected. That is why we decided to redesign
position entity - now it is just a key with MP_ARRAY header, encoded to
base64 format.

NO_CHANGELOG=redesign of unreleased feature
NO_DOC=later

Part of #7639

When pagination is used with offset and returns no tuples while several
tuples were actually skipped by offset, iterator has non-NULL position,
that is why select returns non-empty position. Let's check if resulting
tuples were actually found before return position.

NO_CHANGELOG=bugfix for unreleased feature
NO_DOC=bugfix

Part of #7639

The module API header (module.h) is generated automatically by
extracting everything between '\cond public' and '\endcond public'
from headers listed in the api_headers CMake variable. We're
planning to extend the module API in Tarantool EE. To achieve that,
this commit includes all files listed in the new CMake variable
EXTRA_API_HEADERS into api_headers.

Needed for https://github.com/tarantool/tarantool-ee/issues/265

NO_DOC=build
NO_TEST=build
NO_CHANGELOG=build

Since this feature is compiler-dependent, we need to skip the test if the
server was build with a compiler that does not support the
`-fmacro-prefix-map` flag.

Follow-up #7808

NO_CHANGELOG=test
NO_DOC=test
NO_TEST=test

Currently, `tarantool.build.compiler` stores the `CMAKE_C_COMPILER` and
`CMAKE_CXX_COMPILER` strings from CMake, which are basically paths to the
compiler used for build, and are usually set to `/usr/bin/cc` and
`/usr/bin/c++` accordingly, which does not yield any useful information:
instead, set it to `${CMAKE_C_COMPILER_ID}-${CMAKE_C_COMPILER_VERSION}`,
assuming the same compiler is used for building C and C++.

Closes #7888

@TarantoolBot document
Title: change of information shown in compiler information
Compiler information provided by `tarantool --version` and
`tarantool.build.compiler` now show
`${CMAKE_C_COMPILER_ID}-${CMAKE_C_COMPILER_VERSION}` instead of paths to C
and C++ compilers used to build tarantool, for instance:

```console
tarantool> tarantool.build.compiler

---
- Clang-14.0.0.14000029
...

tarantool> tarantool.build.compiler

---
- GNU-12.2.0
...
```

The `-Wa,--debug-prefix-map` compiler flag breaks GNU-based LTO, and also
we cannot reliably test this feature.

Follow-up 256da010

NO_CHANGELOG=bugfix
NO_DOC=bugfix
NO_TEST=bugfix

This patch renames index_def_is_valid() to index_def_check() and makes
it return 0 or -1 because that function sets a diag.

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring