It has been decided to drop required devx team review for tests.

NO_DOC=codeowners
NO_TEST=codeowners
NO_CHANGELOG=codeowners

(cherry picked from commit 58f5ffe4)

Header "unit.h" contains `ok` and `is` macros used to check test cases.
The problem is such simple names can be used in C++ STL library headers
(it's OK because such short names can be hidden in a namespace), so when
including, for example, header "vector" after "unit.h", build can fail
because function declaration or definition in the C++ header will turn
into a macro invocation. I faced this problem building Tarantool on
MacOS with SDK of 14.4 version.

NO_TEST=fix build
NO_CHANGELOG=fix build
NO_DOC=fix build

(cherry picked from commit 025ba32f)

Currently, exclude_null option doesn't affect functional indexes at all.
It seems that we just forgot to check if tuple should be inserted to the
index - the patch simply adds missing check in replace and build_next
methods of functional memtx_tree index.

Closes #9732

NO_DOC=bugfix

(cherry picked from commit c56998fa)

In order to prevent the garbage collection of the discarded connection,
asynchronous requests must reference the connection object. We must
reference the connection object rather than the transport object, because
our garbage collection hook is attached to the former.

Closes #9629

NO_DOC=<bugfix>

(cherry picked from commit fb5bf51c)

This patch adds necessary cmake configurations for the
<evread.lua> module, so it can be used later to implement
human-readable error reporting in profile parsers.

Part of #9217

NO_DOC=LuaJIT submodule
NO_TEST=covered by the LuaJIT tests
NO_CHANGELOG=build

(cherry picked from commit e01fe8f7)

This commit increases a time to wait of the process termination. It may
take longer than 5 seconds, when tarantool is built with an address
sanitizer. The address sanitizer generates a report at the process
termination and it is not always a fast thing.

NO_DOC=test fix
NO_CHANGELOG=see NO_DOC

(cherry picked from commit 5260bc2a)

This patch reduces the number of tests in the in2.test.lua test file.
This patch also reduces the number of inserted values. This shouldn't
affect the test since it's not really an original Tarantool test, but it
will reduce the execution time of this test. Currently this test often
fails due to a timeout.

NO_DOC=test
NO_CHANGELOG=test

(cherry picked from commit 0ae908cd)

* test: set dependencies in BuildTestCLib macro
* Add 'cc' file type for saving bytecode.
* Fix C file generation in jit.bcsave.
* Throw any errors before stack changes in trace stitching.
* Fix recording of __concat metamethod.
* Check frame size limit before returning to a lower frame.
* build: purge sysprof.collapse module
* build: fix tool components handling
* memprof: refactor `heap_chunk` data structure
* memprof: remove unused arguments
* memprof: introduce the `--human-readable` option
* profilers: introduce event reader module
* ci: extend tarantool integration testing

Part of #9595
Part of #5994
Follows up #8700
Needed for #9217

NO_DOC=LuaJIT submodule bump
NO_TEST=LuaJIT submodule bump

After the commit 13ac5daf ("ci: fix step parameters for
reusable runs") it is now possible to create Tarantool
integration workflows for any repository in the Tarantool
organization. Considering this, we don't need to run Tarantool
tests under the old target for the LuaJIT integration in
.test.mk and we can leave only LuaJIT tests in this target for
the sake of exotic Tarantool builds testing.

NO_DOC=CI
NO_TEST=CI
NO_CHANGELOG=CI

Due to a regression introduced in c13b3a31, the worker fiber is started
synchronously, while it should be started asynchronously, in order for the
`wait_connected = false` option of `connect` to work correctly. We already
explicitly wait from Lua for the connection to become active via
`wait_state` when `wait_connected = true`.

Closes #9489

NO_DOC=<bugfix>

(cherry picked from commit 0502a1f5)

Fixed a bug when it was possible that the `on_relay_thread_start`
trigger handler would continue to use `data` allocated on the stack
of `box_collect_confirmed_vclock` after returning from it.

Closes #9505

NO_DOC=bugfix
NO_TEST=asan

(cherry picked from commit 40bd0eb1)

Bump version of the `codeql-action` action to v3 for fixing an annoying
warning that appears in GitHub WebUI:

    Node.js 16 actions are deprecated.
    Please update the following actions to use Node.js 20

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

(cherry picked from commit 5040fba9)

Bump version of the `upload-artifact` and `download-artifact` actions
to v4 for fixing an annoying warning that appears in GitHub WebUI:

    Node.js 16 actions are deprecated.
    Please update the following actions to use Node.js 20

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

(cherry picked from commit b012117d)

Bump version of actions/checkout to v4.
Bump fixes an annoying warning that appears in Github WebUI:

Node.js 16 actions are deprecated. Please update the following actions
to use Node.js 20: actions/checkout@v3. For more information see:
https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/.

NO_CHANGELOG=ci
NO_DOC=ci
NO_TEST=ci

(cherry picked from commit 8aff2ede)

In commit 13ac5daf ("ci: fix step parameters for reusable
runs") the integration workflow was made reusable, but
concurrency group pattern modification that was done for other
workflows made reusable in the same patch was forgotten. This
patch fixes the mentioned issue.

NO_DOC=CI
NO_TEST=CI
NO_CHANGELOG=CI

The #8350 was introduced by the commit b42302f5 ("lua-yaml: enable
aliasing for objects returned by __serialize") so the patch is
effectively reversed.

The idea is to call all object __serialize methods recursively before
finding references. The new serialization pass stores the mapping from
the original object to the serialized representation.

After this, the reference analysis pass and the encoding pass use this
mapping to replace original objects with the serialized representation.

As result, the reference analysis has a complete information about
objects and no references are missed.

Closes #8350
Closes #8310
Closes #8321

NO_DOC=bugfix

Co-authored-by: Nikolay Shirokovskiy <nshirokovskiy@tarantool.org>
(cherry picked from commit 610f5fb7)

Now the error message is allocated by `malloc' if it doesn't fit into the
static buffer.

Closes #4975

NO_DOC=bugfix

(cherry picked from commit c8da06ca)

It brings newer codespell version: 2.1.0. Ubuntu Focal offers 1.16.0.

Fixes tarantool/checkpatch#70

NO_DOC=CI adjustment
NO_CHANGELOG=see NO_DOC
NO_TEST=see NO_DOC

(cherry picked from commit 21a779e1)

This patch fixes three issues:
1. It changes the condition for workflows so they can be run not
only from the Tarantool repository but from any repository in the
Tarantool organization.

2. Reusable workflows substitute the `${{ github.workflow }}`
context variable with the name of their top-level workflow. This
behavior causes concurrency group clashes when several reusable
workflows are called from a single top-level workflow. This patch
adds an additional constant part to the concurrency group pattern
to solve the issue.

3. The checkout actions use the reference from the repository in
which the top-level workflow is located instead of the one where
the reusable workflow is located. This patch solves the issue by
passing the reference explicitly.

NO_DOC=CI
NO_TEST=CI
NO_CHANGELOG=CI

Some workflows are not relevant for integration testing.
This patch disables them.

NO_DOC=CI
NO_TEST=CI
NO_CHANGELOG=CI

If we run a static build with ASAN enabled via Clang 16, the build will
fail unless `libresolv` is in the white list of static dependencies.
It looks like it is a peculiarity of Clang 16 and higher.

Fixes #9740

NO_DOC=build
NO_TEST=build
NO_CHANGELOG=build

(cherry picked from commit 8f6bc366)

There are two reasons for this changeset:

* The positive one: Tarantool supports -b and -j options to use LuaJIT
  modules since the commit bf8b76a4
  ("lua: proxy -j and -b flags"), so the related tests from lua-Harness
  suite can be partially (since -O option is still not implemented in
  Tarantool) enabled.

* The negative one: Tarantool diff-based tests for CLI interfaces are
  hard to maintain, if any change occurs in LuaJIT modules, since the
  aforementioned tests implement dumb comparison of the output, produced
  by the current CLI version against the expected one, managed by the
  .result file. Hence, to rule the tests related to the LuaJIT CLI
  interface in a more convenient way, the corresponding tests should be
  moved from the tests in the Tarantool repository to the tests in the
  LuaJIT repository.

The recent LuaJIT bump landed to the master in the scope of commit
0dcf6759 ("luajit: bump new version")
enables the nice checks implemented in the lua-Harness suite; this patch
removes the barely maintainable diff-based tests from this repository.

Follows up #5541

NO_DOC=test
NO_CHANGELOG=test

(cherry picked from commit 137e9156)

* cmake: introduce AddTestLib macro
* test: prepare lauxilarily libs for LuaJIT-tests
* test: separate LuaJIT helpers from ffi_util.inc
* test: enable <ffi_arith_ptr.lua> in LuaJIT-tests
* test: enable <ffi_bitfield.lua> in LuaJIT-tests
* test: enable <ffi_call.lua> in LuaJIT-tests
* test: enable <ffi_callback.lua> in LuaJIT-tests
* test: enable <ffi_const.lua> in LuaJIT-tests
* test: enable <ffi_convert.lua> in LuaJIT-tests
* test: enable <ffi_enum.lua> in LuaJIT-tests
* test: enable <ffi_gcstep_recursive.lua>
* test: enable <ffi_jit_arith.lua> in LuaJIT-tests
* test: enable <ffi_jit_call.lua> in LuaJIT-tests
* test: enable <ffi_jit_conv.lua> in LuaJIT-tests
* test: enable <ffi_lex_number.lua> in LuaJIT-tests
* test: enable <ffi_metatype.lua> in LuaJIT-tests
* test: enable <ffi_new.lua> in LuaJIT-tests
* test: enable <ffi_parse_array.lua> in LuaJIT-tests
* test: enable <ffi_parse_basic.lua> in LuaJIT-tests
* test: enable <ffi_parse_cdef.lua> in LuaJIT-tests
* test: enable <ffi_parse_struct.lua> LuaJIT test
* test: enable <ffi_tabov.lua> LuaJIT test
* test: enable <lightud.lua> LuaJIT test
* test: enable <api_call.lua> LuaJIT test
* test: enable <catch_wrap.lua> LuaJIT test
* test: enable <catch_cpp.lua> LuaJIT test
* test: introduce routine to build error message
* test: enable CLI-related lua-Harness tests back

Closes #7834
Part of #9398
Follows up #5541

NO_DOC=LuaJIT submodule bump
NO_TEST=LuaJIT submodule bump
NO_CHANGELOG=add new tests

Disable the app-tap/iconv.test.lua test on RED OS due to missing
character encodings (UTF-16, UTF-16BE, etc).

NO_DOC=test
NO_TEST=test
NO_CHANGELOG=test

Enable LTO in the rpm spec file for RedOS 7+. Otherwise, we are getting
a compilation error.

NO_DOC=build
NO_TEST=build
NO_CHANGELOG=build

Google fuzzing efforts revealed yet another bound condition
we don't handle well in the `tnt_strptime` function:

- for format `%m%g%W`;
- and input string `07001`.

We failed with assertion failure:
```
  | datetime_strptime_fuzzer: ./src/lib/core/datetime.c:148: \
_Bool tm_to_datetime(struct tnt_tm *, struct datetime *): \
Assertion `mday >= 1 && mday <= 31' failed.
```

Closes #8525

NO_TEST=updated fuzzer corpus
NO_CHANGELOG=internal
NO_DOC=internal

(cherry picked from commit 4043664d)

I got four fails on the given tests in a row on debug-asan job in CI for
tarantool-ee.

It seems, tarantool-ee is more sensitive to small timeouts, when the
address sanitizer slows down the execution. Or I'm just lucky.

Anyway, the given tests don't really need small timeouts: increasing it
doesn't break any test logic, doesn't increase duration of the test in a
successful case and doesn't increase it in case of a failure.

The tests are more stable after the change: I verified it locally by
running each of the tests in parallel many times on tarantool built with
enabled address sanitizer.

See the following commits for details about the given test cases and the
problems behind.

* commit 1fcfb8c2 ("app: start init script event loop explicitly")
* commit 786eb2ac ("main: don't break graceful shutdown on init
  script exit")

Follows up #9266
Follows up #9411

NO_DOC=test adjustment
NO_CHANGELOG=see NO_DOC

(cherry picked from commit dfca3c6c)

By design, the error references its cause, see error_set_prev() in diag.c.
Referencing the error from error_set_prev() in error.lua is wrong.
Introduced by commit 2e3c81de ("error: use int64_t as reference
counter").

Closes #9694

NO_DOC=bugfix
NO_TEST=memory leak

(cherry picked from commit d08dbcbf)

Add the fedora_39.yml and fedora_39_aarch64.yml workflow files to build
Tarantool packages for x86_64 and aarch64 platforms.

Closes #9705

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

This commit adds a new configuration option wal_retention_period and
function stubs for it. It's needed to avoid rebootstrap on anonymous
replicas, as Tarantool doesn't save xlog for them.

The new option takes a floating point number that sets the period
for every xlog file during which this xlog file cannot be deleted
by garbage collector. The default value is 0, which means no delay.
The option can be set dynamically.

Note:
- The delay is applied after xlog closing
- During instance restart delay becomes box.cfg.wal_retention_period -
  last modification time of xlog.
- The minimum vclock (same as xlog file name) can be found with
  box.info.gc().wal_retention_vclock.

The option value is stored and used in C code, so we define
configuration callbacks in EE: cfg_set_wal_retention_period.

Needed for tarantool/tarantool-ee#513

NO_DOC=experimental
NO_CHANGELOG=experimental

(cherry picked from commit e04c162e)

This commit introduces new methods for vclock library.

Sometimes we need to take into account the 0-th component of vclock,
as it's done in the following commit, that's why vclock_min/max are
added. vclockset_foreach is just a macros, which allows to iterate
over vclockset comfortably.

NO_DOC=internal
NO_CHANGELOG=internal

(cherry picked from commit 3bf55ab9)

Add the almalinux_9.yml and almalinux_9_aarch64.yml workflow files to
build Tarantool packages for x86_64 and aarch64 platforms.

Closes #9434

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

Add the almalinux_8.yml and almalinux_8_aarch64.yml workflow files to
build Tarantool packages for x86_64 and aarch64 platforms.

Closes #9706

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

Enable in the rpm spec file LTO for AlmaLinux 9+.

Part of #9434

NO_DOC=build
NO_TEST=build
NO_CHANGELOG=build

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

(cherry picked from commit d756931d)

* test: fix initialization in lj-549-lua-load.test.c
* codehealth: add `nd` to the codespell ignore list
* LJ_GC64: Always snapshot functions for non-base frames.
* Avoid assertion in case of stack overflow from stitched trace.
* Fix recording of __concat metamethod.
* Avoid out-of-range number of results when compiling select(k, ...).
* Consider slots used by upvalues in use-def analysis.
* Only emit proper parent references in snapshot replay.
* Optimize table.new() with constant args to (sinkable) IR_TNEW.
* Followup fix for embedded bytecode loader.
* Fix embedded bytecode loader.
* LJ_GC64: Fix HREFK optimization.
* Fix unsinking of IR_FSTORE for NULL metatable.
* Fix zero stripping in %g number formatting.
* Follow-up fix for stack overflow handling cleanup.
* Cleanup stack overflow handling.
* Improve error reporting on stack overflow.
* sysprof: disable runtime host symtab updates
* codehealth: fix the typo
* Simplify handling of instable types in TNEW/TDUP load forwarding.
* Respect jit.off() on pending trace exit.
* Limit exponent range in number parsing.
* Emit sunk IR_NEWREF only once per key on snapshot replay.

Closes #7937
Closes #8140
Part of #9145
Part of #9595

NO_DOC=LuaJIT submodule bump
NO_TEST=LuaJIT submodule bump

Bump test-run to new version with the following improvements:

- Bump luatest to 1.0.0-5-gf31fe34 [1]
- get_iproto_port: remove duplicates [2]
- requirements: bump gevent to 22.10.2 [3]
- Fix decoding error when reading server's log file [4]

[1] tarantool/test-run@bfcc9e8
[2] tarantool/test-run@da98d7f
[3] tarantool/test-run@bc1c473
[4] tarantool/test-run@434cbec

NO_DOC=test
NO_TEST=test
NO_CHANGELOG=test

(cherry picked from commit d75cd423)

This function returns a key_def part by a field number. However, currently
it returns NULL for parts that contain a JSON path to indexed data. Fix it.

Needed for tarantool/tarantool-ee#671

NO_DOC=bugfix
NO_CHANGELOG=not visible in CE

(cherry picked from commit cb0264c3)

We've had numerous problems with transaction boundaries in replication.
They were mostly caused by various cases when either the beginning or
end of the transaction happened to be a local row. Local rows are not
replicated, so the peer saw "corrupted" transactions with either no
beginning or no end flag, even though the transaction contents were
fine.

The problem with starting a transaction with a local row was solved in
commit f41d1ddd ("wal: fix tx boundaries"), and that fix seems to
continue working fine to this day.

The problem with ending transactions with a local row was first fixed
in commit 25382617 ("replication: append NOP as the last tx row"),
however there were problems with this approach: when a user tried to
write to local spaces on a replica from a replication trigger, it made
it impossible to ever start replicating from replica back to master.

Another fix was proposed: in commit f96782b5 ("relay: send rows
transactionally") we made relay read a full transaction into memory and
then send it all at once mangling with transanction start and end flags
when necessary.

After that the NOPs were removed in commit f5e52b2c ("box: get rid
of dummy NOPs after transactions ending with local rows"), since relay
became capable of fixing transaction boundaries itself.

Turns out the assumption that relay always sees a full transaction and
may correctly set transaction boundaries is wrong: when a replica
reconnects to master we set its starting vclock[0] to the one master has
at the moment of reconnect, so when recovery reads local rows with lsns
less than vclock[0] it silently skips them without showing them to
relay. When such skipped rows contain the is_commit flag for a currently
sent transaction we get the same problem as described before.

Let's make recovery track whether it has pushed any transaction rows to
relay or not, and if yes, recover rows with is_commit flag regardless of
whether the rows were already applied. To prevent recovering the same
data twice, recovery replaces such row contents with NOPs. Basically the
row is "recovered" only for the sake of showing its is_commit flag to
relay. Relay will skip the row anyway, since it remains local.

Follow-up #8958
Closes #9491

NO_DOC=bugfix

(cherry picked from commit 60d45765)

It doesn't make sense to assert that replica_id is correct in a row
after using that replica id to make some decisions based on it. Let's
switch the order of operations: first assert that replica_id is correct,
then compare row lsn with the already recovered one.

In-scope-of #9491

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

(cherry picked from commit 4ec51b4b)