Runtime tuple formats are reusable, which means that a tuple format
returned by runtime_tuple_format_new may not be brand new, but actually
be used by a Lua object. As a result, if we call any function that may
trigger Lua GC between runtime_tuple_format_new and tuple_format_ref,
the tuple format may be deleted, leading to a use-after-free bug. This
is what happens in lbox_tuple_format_new. Fix this issue by taking a
reference to the format before pushing a cdata object to the Lua stack
in lbox_push_tuple_format.

The issue was fixed in the master branch by commit 28ec245d ("lua:
fix heap-use-after-free bug in tuple format constructor"). This isn't
a clean cherry-pick because the code changed quite a bit.

Closes #8889

NO_DOC=bug fix
NO_TEST=difficult to reproduce, found by ASAN

(cherry picked from commit 4123061b)

* ci: support coveralls
* cmake: add code coverage support
* test: run flake8 static analysis via CMake
* test: fix E741 errors by pycodestyle
* test: fix E722 errors by pycodestyle
* test: fix E711 errors by pycodestyle
* test: fix E502 errors by pycodestyle
* test: fix E501 errors by pycodestyle
* test: fix E305 errors by pycodestyle
* test: fix E303 errors by pycodestyle
* test: fix E302 errors by pycodestyle
* test: fix E301 errors by pycodestyle
* test: fix E275 errors by pycodestyle
* test: fix E251 errors by pycodestyle
* test: fix E231 errors by pycodestyle
* test: fix E203 errors by pycodestyle
* test: fix E201 and E202 errors by pycodestyle
* test: suppress E131 errors by pycodestyle
* test: fix E128 errors by pycodestyle
* test: fix E122 errors by pycodestyle
* gdb: fix Python <assert> statement usage

NO_DOC=LuaJIT submodule bump
NO_TEST=LuaJIT submodule bump
NO_CHANGELOG=LuaJIT submodule bump

The `__sanitizer_start_switch_fiber()` function takes a pointer as the
first argument to store the current fake stack if there is one (it is
necessary when stack-use-after-return detection is enabled). When leaving a
fiber definitely, NULL must be passed so that the fake stack is destroyed.

Before this patch, NULL was passed for dead fibers, however this is wrong
for dead fibers that are recycled and resumed. In such cases ASAN destroys
the fake stack, and the fiber crashes trying to use it in `fiber_yield()`
upon return from `coro_transfer()`.

Closes tarantool/tarantool-qa#321

NO_DOC=bugfix
NO_TEST=tested by test-release-asan workflow

(cherry picked from commit 72a6abee)

Fixed the implementation of the box console.
Before this fix, result of `\set language` is shared between clients
via `console.connect`, despite the fact that clients have different
`box.session.id`. Now the parameter of the selected language is stored
by each client in his own `box.session.storage`.

Fixes #8817

NO_DOC=bugfix

(cherry picked from commit e4fda4b7)

Interactive console test helper could be used for remote connection too,
for that purpose prompt needs to be changed accordingly to connection
type.

This patch introduces getter and setter for the prompt. Prompt is
configured per session, so it is advised to create a new session
for each test case (e.g. with before_each()).

NO_CHANGELOG=test helper change
NO_DOC=test helper change

(cherry picked from commit ed86a729)

The ./test/luatest_helpers/interactive_tarantool.lua module is not
a luatest helper. So moving it to the ./test/ dir and removing empty
./test/luatest_helpers/.

NO_DOC=testing stuff
NO_TEST=testing stuff
NO_CHANGELOG=testing stuff

(cherry picked from commit 5493db7c)

A basic (and pretty useless) example:

```lua
local it = require('test.luatest_helpers.interactive_tarantool')

local child = it.new()

child:execute_command('6 * 7')
local res = child:read_response()
t.assert_equals(res, 42)

child:close()
```

The module also contains `:read_line()`, `:assert_line()` helpers for
testing output directly: for example, when we need to catch a print()
from a background fiber. It provides has constants related to terminal's
control sequences.

A real usage can be seen in a next commit.

Part of #7169

NO_DOC=no user visible changes
NO_TEST=not applicable, it is a testing helper
NO_CHANGELOG=no user visible changes

(cherry picked from commit a9d96007)

As the underlying problem behind this injection is fixed in #7357 it can
be removed and `-i` flag could be used as initially intended.

Closes #7554
Requires #7357
NO_DOC=refactoring
NO_CHANGELOG=refactoring

(cherry picked from commit 16d6e9d2)

The interactive mode has been ignored when stdin was not a tty and is no
more. Now results of another command can be handled by tarantool.
Before the patch:
```
$ echo 42 | tarantool -i
LuajitError: stdin:1: unexpected symbol near '42'
fatal error, exiting the event loop
```

After the patch:
```
$ echo 42 | tarantool -i
Tarantool 2.5.0-130-ge3cf64a6c
type 'help' for interactive help
tarantool> 42
---
- 42
...

```

Closes #5064

NO_DOC=bugfix

(cherry picked from commit 9965e3fe)

Bump test-run to new version with the following improvements:

- luatest: fix detect tarantool crash at exit [1]
- Fix bug when lua script name truncated by dot [2]
- Raise an error and log it if test timeouts are set incorrectly [3]
- Pin PyYAML version to 5.3.1 [4]
- Add ability to set path to executable file [5]
- Migrate tarantoolctl from tarantool repository [6]
- Fix test-run crash when default server is crashed [7]
- Disable reproduce content printing [8]

[1] tarantool/test-run@be693d1
[2] tarantool/test-run@a6405f1
[3] tarantool/test-run@d34ecb0
[4] tarantool/test-run@704420e
[5] tarantool/test-run@0a70001
[6] tarantool/test-run@ad43d8f
[7] tarantool/test-run@b31329e
[8] tarantool/test-run@31f0ced

NO_DOC=test
NO_TEST=test
NO_CHANGELOG=test

(cherry picked from commit f4511948)

Run release build LTO testing inside a Docker container created from the
`tarantool/testing:ubuntu-jammy-clang16` image with Clang 16 installed.

Closes #318

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

(cherry picked from commit 9d0cb54f)

Run release build testing inside a Docker container created from the
`tarantool/testing:ubuntu-jammy-clang16` image with Clang 16 installed.

Closes #317

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

(cherry picked from commit 9134dabd)

NO_DOC=build
NO_TEST=build
NO_CHANGELOG=build

(cherry picked from commit bb74d6c9)

Add code owners for CI-related script and
for github automation directory.

NO_CHANGELOG=no code changes
NO_TEST=no code changes
NO_DOC=no code changes

(cherry picked from commit 9234763a)

In order to improve tests of Tarantool core
assign dedicated team to perform review of each
and every change.

NO_CHANGELOG=no code changes
NO_TEST=no code changes
NO_DOC=no code changes

(cherry picked from commit 532bada7)

To make changelog preparation less stressful let's pass each and every
change to changelogs/ directly through the doc team.

NO_CHANGELOG=no code changes
NO_TEST=no code changes
NO_DOC=no code changes

(cherry picked from commit 2970bd57)

In #7309 a truncation of a space that was referenced by foreign
key from some other space was prohibited.

It appeared that this solution is too bothering since a user can't
truncate a space even if he truncated referring space before that.

Fix it by allowing space truncate if referring spaces are empty.
Also allow drop of the primary index in the same case with the
same reason: logically the index along with all space data is
not needed for consistency if there's no referring data.

Note that by design space truncate is implemented quite similar
to space drop. Both delete all indexes, from secondary to primary.
Since this patch allows deletion of the primary index (which is
the action that actually deletes all data from the space), this
patch changes the result of space drop too: the space remains
alive with no indexes, while before this patch it remained alive
with no secondary indexes but with present primary. In both cases
the behaviour is quite strange and must be fixed in #4348. To
make tests pass I had to perform drop in box.atomic manually.

Closes #8946

NO_DOC=bugfix

(cherry picked from commit 983a7ec2)

All LuaJIT related LSan warnings were suppressed in the scope of the
commit 985548e4 ("asan: suppress all
LSAN warnings related to LuaJIT"), since all compiler flags tweaks were
enclosed in LuaJIT CMake machinery. As a result of the commit in LuaJIT
submodule tarantool/luajit@a86e376 ("build: introduce LUAJIT_USE_ASAN
option") ASan and LSan support has been finally added to LuaJIT runtime,
so it was decided to remove LSan suppressions for LuaJIT functions.
Unfortunately, it was not so easy as it looked like.

At first, Lua global state is not closed properly at Tarantool instance
exit (see <tarantool_free> in src/main.cc and <tarantool_lua_free> in
src/lua/init.c for more info), so LSan false-positive leaks are detected
(for more info, see #3071). Hence, the original LSan suppression for
lj_BC_FUNCC is returned back (temporarily) until the aforementioned
issue is not resolved.

Furthermore, the internal LuaJIT memory allocator is not instrumented
yet, so to find any memory faults it's worth building LuaJIT with system
provided memory allocator (i.e. enable LUAJIT_USE_SYSMALLOC option).
However, again, since Tarantool doesn't finalize Lua universe the right
way, so running Tarantool testing routine with LUAJIT_USE_SYSMALLOC
enabled generates false-positive LSan leaks. Return back here to enable
LUAJIT_USE_SYSMALLOC, when #3071 is resolved.

Last but not least, the default value of fiber stack size is 512Kb, but
several tests in test/PUC-Rio-Lua-5.1-test suite in LuaJIT submodule
(e.g. some cases with deep recursion in errors.lua or pm.lua) have
already been tweaked according to the limitations mentioned in #5782,
but the crashes still occur while running LuaJIT tests with ASan support
enabled. Experiments once again confirm the notorious quote that "640 Kb
ought to be enough for anybody".

Anyway, LuaJIT tests are added to <test-release-asan> target in .test.mk
and LUAJIT_TEST_ENV is extended with required ASan and LSan options.

Follows up #5878

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

(cherry picked from commit bacf4e56)

* ci: introduce testing workflow with sanitizers
* build: introduce LUAJIT_USE_ASAN option
* test: introduce test:done TAP helper
* memprof: remove invalid assertions
* ci: clean up workflow for exotic builds

Closes #5878

NO_DOC=LuaJIT submodule bump
NO_TEST=LuaJIT submodule bump
NO_CHANGELOG=LuaJIT submodule bump

Applier thread uses lsregion to allocate the messages for tx thread. The
messages are freed upon return to the applier thread using a
corresponding lsr_id.

Due to a typo, one of the lsregion allocations was made with a postfix
increment of lsr_id instead of the prefix one. Essentially, part of a
new message was allocated with an old lsr_id, and might be freed early
by a return of a previous message.

Fix this.

Closes #8848

NO_DOC=bugfix
NO_TEST=covered by asan in #8901
NO_CHANGELOG=bugfix

(cherry picked from commit 0d5bd6b7)

This improves diff hunk name detection. Needed for checkpatch to
correctly detect if patched code belongs to a function.

NO_DOC=git
NO_TEST=git
NO_CHANGELOG=git

(cherry picked from commit 642584fd)

This patch fixes an issue in generate_column_metadata(). Prior to this
patch, the number of variable-only expressions was counted incorrectly
when temporary memory was allocated on region to store their positions.
However, although this allocation was incorrect, this did not lead to
any problems due to the specifics of the region allocations.

This patch fixes this by removing the temporary memory allocation.

Closes #8763

NO_DOC=no user-visible changes
NO_TEST=no user-visible changes
NO_CHANGELOG=no user-visible changes

(cherry picked from commit d4f143ad)

It is possible for interval to have days, hours, minutes and seconds
larger than INT_MAX (or less than INT_MIN). Before this patch, msgpack
decoding had failed to parse intervals with msgpack int64 and uint64.
int64_t should be enough to store any value allowed for datetime
intervals.

Closes #8887

NO_DOC=small bug fix

(cherry picked from commit 01c7ae11)

* FFI: Fix recording of union initialization.
* Fix maxslots when recording BC_VARG, part 2.
* Fix maxslots when recording BC_VARG.
* Fix BC_UCLO insertion for returns.
* ci: update job concurrency group definition

Part of #8825

NO_DOC=LuaJIT submodule bump
NO_TEST=LuaJIT submodule bump

Before this patch, one couldn't create new datetime interval with
boundary value from Lua. At the same time, it was possible to create
such interval from Lua through addition and subtraction. C range
verification allow to create boundary value intervals, error message
also implies that they should be allowed. (See #8878 for more info.)

Closes #8878

NO_DOC=small bug fix

(cherry picked from commit b2a001cc)

A read source iterator stores statements in a vy_history object using
vy_history_append_stmt(). If a statement can be referenced, it's
reference counter is incremented. If it can't, i.e. it belongs to a
memory source, it's stored in a vy_history object without referencing.

This works fine because memory sources are append-only. A problem arises
only when we get to scanning disk sources. Since we yield while reading
disk, a dump task may complete concurrently dropping the memory sources
and possibly invalidating statements stored in the iterator history.
Although we drop the history accumulated so far and restart the
iteration from scratch in this case, there's still an issue that can
result in a use-after-free bug in vy_read_iterator_next().

The problem is that we access the current candidate for the next
statement while evaluating a disk source after a disk read. If 'next'
refers to a referenced statement, it's fine, but if it refers to a
statement from a memory source, it may cause use-after-free because
the memory source may be dropped during a disk read.

To fix this issue, let's make vy_history_append_stmt() copy statements
coming from memory sources. This should be fine performance-wise because
we copied memory statements eventually in vy_history_apply() anyway,
before returning them to the user.

Note that we also have to update vy_read_iterator_restore_mem() because
it implicitly relied on the fact that 'next' coming from a memory source
can't be freed by vy_mem_iterator_restore(), which cleans up the memory
source history. Now, it isn't true anymore so we have to temporarily
take a reference to 'next' explicitly.

Closes #8852

NO_DOC=bug fix
NO_TEST=tested by ASAN

(cherry picked from commit 0e5a3cc2)

This commit follows up the previous one (i.e. the backport of the commit
ff57f990 ("cmake: introduce FIBER_STACK_SIZE option")), since the
commit 98f62cd0 ("test: make fiber_stack.test tap-compatible") and
its siblings have not been backported to 2.10 in scope of #7618.

NO_DOC=fixup for d296f732
NO_CHANGELOG=fixup for d296f732

In scope of the commit 82f4b4a3 ("lib/core/fiber: Increase default
stack size") the default value of fiber stack size is increased up to
512 Kb (you can find the reasons in the aforementioned commit message
and in https://github.com/tarantool/tarantool/issues/3418 description).

Some of the tests in test/PUC-Rio-Lua-5.1-test suite in LuaJIT repo
(e.g. some cases with deep recursion in errors.lua or pm.lua) have
already been tweaked according to the limitations mentioned in
https://github.com/tarantool/tarantool/issues/5782, but the crashes
still occurs while running LuaJIT tests with ASan support enabled.

To make the testing routine more convenient, FIBER_STACK_SIZE option is
introduced to Tarantool CMake machinery. One can provide the size either
by raw digits (i.e. in bytes) or using Kb/Mb suffixes for convenience.

A couple of important nits:
* If the given value is not a multiple of 4Kb, CMake machinery adjusts
  it up to the nearest one greater than this value.
* If the adjusted value is less than 512Kb, configuration fails with the
  corresponding CMake fatal error.

Follows up #3418
Relates to #5782

@TarantoolBot document
Title: introduce FIBER_STACK_SIZE configuration option

To make managing of the default fiber stack size more convenient, the
corresponding CMake option is added.

**NB**: The stack size can't be less than 512Kb and if the given value
is not a multiple of 4Kb, CMake machinery adjusts it up to the nearest
one greater than this value.

(cherry picked from commit ff57f990)

This reverts commit 8aa2bb19.

In scope of the aforementioned commit changes from commit e4fda4b7
("box: fix shared lang between connected clients") are backported, but
the test in this patch uses test.interactive_tarantool.lua helper, that
is missing in release/2.10 branch. Applying all the related commits from
the master listed below doesn't fix the problem:
* commit a9d96007 ("test: add a helper for testing interactive
  mode")
* commit 5493db7c ("test: mv interactive_tarantool.lua to ./test/
  dir")
* commit ed86a729 ("test: add prompt setter to interactive helper")

Hence, simply revert the poisoned commit from the branch.

NO_DOC=revert
NO_TEST=revert
NO_CHANGELOG=revert

Test uses a popen module that starts tarantool process in background
mode. Tarantool process started in background mode forks a new process
and closes a parent, after that popen loses a PID of the started process
and `ph:kill()` and `ph:terminate()` doesn't work anymore. It leads to
non-terminated tarantool processes after running the test.

Patch fixes that by running `kill` using os.execute with a PID of
tarantool process written to a pid file.

Follows up #6128

NO_CHANGELOG=fix test
NO_DOC=fix test

(cherry picked from commit 88686227)

Fixed the implementation of the box console.
Before this fix, result of `\set language` is shared between clients
via `console.connect`, despite the fact that clients have different
`box.session.id`. Now the parameter of the selected language is stored
by each client in his own `box.session.storage`.

Fixes #8817

NO_DOC=bugfix

(cherry picked from commit e4fda4b7)

When a space is altered, we abort all in-progress transactions and delete
all stories related to that space: the problem is we don't delete the
stories' read gaps, which are also linked to the stories' transactions,
which get cleaned up on transaction destruction — this, in turn, results in
heap-use-after-free. To fix this, clean up stories' read gap in
`memtx_on_space_delete` — we don't do this in `memtx_tx_story_delete` since
it expects the story to not have any read gaps (see
`memtx_tx_story_gc_step`).

Tested this patch manually against Nick Shirokovskiy's experimental
small-ASAN integration branch.

Closes #8781

NO_DOC=bugfix
NO_TEST=<already covered by existing tests, but was not detectable by ASAN>

(cherry picked from commit e1ed31bb)

* test: fix flaky <unit-jit-parse.test.lua>
* Fix use-def analysis for vararg functions.
* Fix use-def analysis for BC_VARG.
* Fix TNEW load forwarding with instable types.
* Fix memory probing allocator to check for valid end address, too.
* Another fix for lua_yield() from C hook.
* Fix lua_yield() from C hook.
* x64: Fix 64 bit shift code generation.
* Fix canonicalization of +-0.0 keys for IR_NEWREF.
* test: add utility for parsing `jit.dump`
* test: split utils.lua into several modules
* test: rewrite lj-49-bad-lightuserdata test in C
* test: rewrite misclib-sysprof-capi test in C
* test: rewrite misclib-getmetrics-capi test in C
* test: introduce utils.h helper for C tests
* test: introduce module for C tests
* test: fix setting of {DY}LD_LIBRARY_PATH variables
* build: fix build with LUAJIT_USE_GDBJIT enabled
* ci: update the branch name for Tarantool 2.10

Closes #8718
Part of #7900
Part of #8516

NO_DOC=LuaJIT submodule bump
NO_TEST=LuaJIT submodule bump

Previous attempt to fix flakiness in commit 6a2c73f8 ("test: fix
flakiness in gh_6128_background_mode_test") used a constant buffer size
in check_err_msg function. Tarantool 2.10 has a bit bigger log before a
desired message that other versions of Tarantool and it leads to a this
resulted in a truncated message ("entering the even" instead of
"entering the event loop"). Patch replaces check_err_msg()
implementation to grep_log used in luatest, it reads the whole log.

Also patch renames check_err_msg to check_msg, because "entering the
event loop" is not an error message.

Follows up #6128

NO_CHANGELOG=fix test
NO_DOC=fix test

(cherry picked from commit 1c8e7124)

Test runs an external process with tarantool that writes to a log file.
Then test reads that log file and searches a string with required
message in it (see function check_err_msg). Test was flaky on macOS and
I suspect it was happening due to a high log level - timeout was not
enough to wait message in the log file.

Patch decreases a log level to a default value and replaces io
functions with the similar alternatives in a fio module. Using
fio functions allows to not block fibers.

NO_CHANGELOG=test fix
NO_DOC=test fix

(cherry picked from commit 47380bb7)

The xlog reader Lua module uses the xlog_cursor_next_row, which decodes
the row header with xrow_header_decode. The latter silently ignores any
unknown fields, which complicates catching bugs when garbage is written
to a row header by mistake, for example, see #8783.

Let's parse a row header without using xrow_header_decode in the xlog
reader module, like we parse a row body, and output all unknown/invalid
keys as is.

To do that, we have to extend the xlog cursor API with the new method
xlog_cursor_next_row_raw that returns a pointer to the position in the
tx buffer where the next xrow is stored without advancing it. To avoid
a memory leak in case the caller fails to parse an xrow returned by this
function, we also have to move the call to xlog_tx_cursor_destroy from
xlog_tx_cursor_next_row to xlog_cursor_next_tx.

While we are at it,
 - Don't raise an error if a key type encountered in a row body is
   invalid (not an integer). Instead, silently ignore such keys.
 - Remove the useless body MsgPack validness check because we already
   check it after decoding the header.
 - Add error injection based tests to check all the corner cases.

NO_DOC=bug fix

(cherry picked from commit 8a25d170)

To avoid garbage written to xlog.

Closes #8783

NO_DOC=bug fix
NO_TEST=next commit

(cherry picked from commit f058cee7)

<luaT_pushtuple> function created a new GCfunc object for a tuple __gc
handler (i.e. <lbox_tuple_gc>), having no upvalues, on each call when
tuple object for Lua world is created.

The change introduces the reference to the single GCfunc object created
on Tarantool startup and stored to Lua registry. Using this approach in
scope of <luaT_pushtuple> is aimed to reduce Lua GC memory usage.

Part of #5201

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Reviewed-by: Vladimir Davydov <vdavydov@tarantool.org>
Reviewed-by: Sergey Kaplun <skaplun@tarantool.org>
Signed-off-by: Igor Munkin <imun@tarantool.org>
(cherry picked from commit 8d7d1460)

There's case when a transaction is rolled back from prepared
state. This happens when WAL fails, synchronized replication
confirmation failure or perhaps in other similar cases. By design
other RW transactions and transactions with READ_COMMITTED
isolation level are allowed to read prepared state. All these
transactions must be aborted in case of rollback of prepared
transaction since they definitely have read no more possible
database state.

This patch implements this abortion.

Closed #8654

NO_DOC=bugfix

(cherry picked from commit 54986902)

Rollback is rather complicated part if MVCC implementation that
is meant to handle two kinds of rollback:
* rollback from in-progress state, if box.rollback() is called.
* rollback from prepared state, when WAL fails.
Unfortunately the last one was not properly tested and surely
has at least one flaw. When an inserting transaction becomes
prepared its stories could be linked as deleted (via del_stmt
pointer) by other in-progress transactions in order to maintain
correct visibility. The problem is that in case of rollback of
such prepared transaction those links remained. Broken links
breaks the chain structure, and some older changes (that were
linked as deleted before that hapless preparation) can become
visible to other transaction.

Refactor, simplify a bit that part of code and fix the issue
described above; cover with tests.

Closes #8648

NO_DOC=bugfix

(cherry picked from commit 85569d9c)