The #8350 was introduced by the commit b42302f5 ("lua-yaml: enable
aliasing for objects returned by __serialize") so the patch is
effectively reversed.

The idea is to call all object __serialize methods recursively before
finding references. The new serialization pass stores the mapping from
the original object to the serialized representation.

After this, the reference analysis pass and the encoding pass use this
mapping to replace original objects with the serialized representation.

As result, the reference analysis has a complete information about
objects and no references are missed.

Closes #8350
Closes #8310
Closes #8321

NO_DOC=bugfix

Co-authored-by: Nikolay Shirokovskiy <nshirokovskiy@tarantool.org>
(cherry picked from commit 610f5fb7)

Now the error message is allocated by `malloc' if it doesn't fit into the
static buffer.

Closes #4975

NO_DOC=bugfix

(cherry picked from commit c8da06ca)

It brings newer codespell version: 2.1.0. Ubuntu Focal offers 1.16.0.

Fixes tarantool/checkpatch#70

NO_DOC=CI adjustment
NO_CHANGELOG=see NO_DOC
NO_TEST=see NO_DOC

(cherry picked from commit 21a779e1)

This patch fixes three issues:
1. It changes the condition for workflows so they can be run not
only from the Tarantool repository but from any repository in the
Tarantool organization.

2. Reusable workflows substitute the `${{ github.workflow }}`
context variable with the name of their top-level workflow. This
behavior causes concurrency group clashes when several reusable
workflows are called from a single top-level workflow. This patch
adds an additional constant part to the concurrency group pattern
to solve the issue.

3. The checkout actions use the reference from the repository in
which the top-level workflow is located instead of the one where
the reusable workflow is located. This patch solves the issue by
passing the reference explicitly.

NO_DOC=CI
NO_TEST=CI
NO_CHANGELOG=CI

Some workflows are not relevant for integration testing.
This patch disables them.

NO_DOC=CI
NO_TEST=CI
NO_CHANGELOG=CI

If we run a static build with ASAN enabled via Clang 16, the build will
fail unless `libresolv` is in the white list of static dependencies.
It looks like it is a peculiarity of Clang 16 and higher.

Fixes #9740

NO_DOC=build
NO_TEST=build
NO_CHANGELOG=build

(cherry picked from commit 8f6bc366)

There are two reasons for this changeset:

* The positive one: Tarantool supports -b and -j options to use LuaJIT
  modules since the commit bf8b76a4
  ("lua: proxy -j and -b flags"), so the related tests from lua-Harness
  suite can be partially (since -O option is still not implemented in
  Tarantool) enabled.

* The negative one: Tarantool diff-based tests for CLI interfaces are
  hard to maintain, if any change occurs in LuaJIT modules, since the
  aforementioned tests implement dumb comparison of the output, produced
  by the current CLI version against the expected one, managed by the
  .result file. Hence, to rule the tests related to the LuaJIT CLI
  interface in a more convenient way, the corresponding tests should be
  moved from the tests in the Tarantool repository to the tests in the
  LuaJIT repository.

The recent LuaJIT bump landed to the master in the scope of commit
0dcf6759 ("luajit: bump new version")
enables the nice checks implemented in the lua-Harness suite; this patch
removes the barely maintainable diff-based tests from this repository.

Follows up #5541

NO_DOC=test
NO_CHANGELOG=test

(cherry picked from commit 137e9156)

* cmake: introduce AddTestLib macro
* test: prepare lauxilarily libs for LuaJIT-tests
* test: separate LuaJIT helpers from ffi_util.inc
* test: enable <ffi_arith_ptr.lua> in LuaJIT-tests
* test: enable <ffi_bitfield.lua> in LuaJIT-tests
* test: enable <ffi_call.lua> in LuaJIT-tests
* test: enable <ffi_callback.lua> in LuaJIT-tests
* test: enable <ffi_const.lua> in LuaJIT-tests
* test: enable <ffi_convert.lua> in LuaJIT-tests
* test: enable <ffi_enum.lua> in LuaJIT-tests
* test: enable <ffi_gcstep_recursive.lua>
* test: enable <ffi_jit_arith.lua> in LuaJIT-tests
* test: enable <ffi_jit_call.lua> in LuaJIT-tests
* test: enable <ffi_jit_conv.lua> in LuaJIT-tests
* test: enable <ffi_lex_number.lua> in LuaJIT-tests
* test: enable <ffi_metatype.lua> in LuaJIT-tests
* test: enable <ffi_new.lua> in LuaJIT-tests
* test: enable <ffi_parse_array.lua> in LuaJIT-tests
* test: enable <ffi_parse_basic.lua> in LuaJIT-tests
* test: enable <ffi_parse_cdef.lua> in LuaJIT-tests
* test: enable <ffi_parse_struct.lua> LuaJIT test
* test: enable <ffi_tabov.lua> LuaJIT test
* test: enable <lightud.lua> LuaJIT test
* test: enable <api_call.lua> LuaJIT test
* test: enable <catch_wrap.lua> LuaJIT test
* test: enable <catch_cpp.lua> LuaJIT test
* test: introduce routine to build error message
* test: enable CLI-related lua-Harness tests back

Closes #7834
Part of #9398
Follows up #5541

NO_DOC=LuaJIT submodule bump
NO_TEST=LuaJIT submodule bump
NO_CHANGELOG=add new tests

Disable the app-tap/iconv.test.lua test on RED OS due to missing
character encodings (UTF-16, UTF-16BE, etc).

NO_DOC=test
NO_TEST=test
NO_CHANGELOG=test

Enable LTO in the rpm spec file for RedOS 7+. Otherwise, we are getting
a compilation error.

NO_DOC=build
NO_TEST=build
NO_CHANGELOG=build

Google fuzzing efforts revealed yet another bound condition
we don't handle well in the `tnt_strptime` function:

- for format `%m%g%W`;
- and input string `07001`.

We failed with assertion failure:
```
  | datetime_strptime_fuzzer: ./src/lib/core/datetime.c:148: \
_Bool tm_to_datetime(struct tnt_tm *, struct datetime *): \
Assertion `mday >= 1 && mday <= 31' failed.
```

Closes #8525

NO_TEST=updated fuzzer corpus
NO_CHANGELOG=internal
NO_DOC=internal

(cherry picked from commit 4043664d)

I got four fails on the given tests in a row on debug-asan job in CI for
tarantool-ee.

It seems, tarantool-ee is more sensitive to small timeouts, when the
address sanitizer slows down the execution. Or I'm just lucky.

Anyway, the given tests don't really need small timeouts: increasing it
doesn't break any test logic, doesn't increase duration of the test in a
successful case and doesn't increase it in case of a failure.

The tests are more stable after the change: I verified it locally by
running each of the tests in parallel many times on tarantool built with
enabled address sanitizer.

See the following commits for details about the given test cases and the
problems behind.

* commit 1fcfb8c2 ("app: start init script event loop explicitly")
* commit 786eb2ac ("main: don't break graceful shutdown on init
  script exit")

Follows up #9266
Follows up #9411

NO_DOC=test adjustment
NO_CHANGELOG=see NO_DOC

(cherry picked from commit dfca3c6c)

By design, the error references its cause, see error_set_prev() in diag.c.
Referencing the error from error_set_prev() in error.lua is wrong.
Introduced by commit 2e3c81de ("error: use int64_t as reference
counter").

Closes #9694

NO_DOC=bugfix
NO_TEST=memory leak

(cherry picked from commit d08dbcbf)

Add the fedora_39.yml and fedora_39_aarch64.yml workflow files to build
Tarantool packages for x86_64 and aarch64 platforms.

Closes #9705

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

This commit adds a new configuration option wal_retention_period and
function stubs for it. It's needed to avoid rebootstrap on anonymous
replicas, as Tarantool doesn't save xlog for them.

The new option takes a floating point number that sets the period
for every xlog file during which this xlog file cannot be deleted
by garbage collector. The default value is 0, which means no delay.
The option can be set dynamically.

Note:
- The delay is applied after xlog closing
- During instance restart delay becomes box.cfg.wal_retention_period -
  last modification time of xlog.
- The minimum vclock (same as xlog file name) can be found with
  box.info.gc().wal_retention_vclock.

The option value is stored and used in C code, so we define
configuration callbacks in EE: cfg_set_wal_retention_period.

Needed for tarantool/tarantool-ee#513

NO_DOC=experimental
NO_CHANGELOG=experimental

(cherry picked from commit e04c162e)

This commit introduces new methods for vclock library.

Sometimes we need to take into account the 0-th component of vclock,
as it's done in the following commit, that's why vclock_min/max are
added. vclockset_foreach is just a macros, which allows to iterate
over vclockset comfortably.

NO_DOC=internal
NO_CHANGELOG=internal

(cherry picked from commit 3bf55ab9)

Add the almalinux_9.yml and almalinux_9_aarch64.yml workflow files to
build Tarantool packages for x86_64 and aarch64 platforms.

Closes #9434

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

Add the almalinux_8.yml and almalinux_8_aarch64.yml workflow files to
build Tarantool packages for x86_64 and aarch64 platforms.

Closes #9706

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

Enable in the rpm spec file LTO for AlmaLinux 9+.

Part of #9434

NO_DOC=build
NO_TEST=build
NO_CHANGELOG=build

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

(cherry picked from commit d756931d)

* test: fix initialization in lj-549-lua-load.test.c
* codehealth: add `nd` to the codespell ignore list
* LJ_GC64: Always snapshot functions for non-base frames.
* Avoid assertion in case of stack overflow from stitched trace.
* Fix recording of __concat metamethod.
* Avoid out-of-range number of results when compiling select(k, ...).
* Consider slots used by upvalues in use-def analysis.
* Only emit proper parent references in snapshot replay.
* Optimize table.new() with constant args to (sinkable) IR_TNEW.
* Followup fix for embedded bytecode loader.
* Fix embedded bytecode loader.
* LJ_GC64: Fix HREFK optimization.
* Fix unsinking of IR_FSTORE for NULL metatable.
* Fix zero stripping in %g number formatting.
* Follow-up fix for stack overflow handling cleanup.
* Cleanup stack overflow handling.
* Improve error reporting on stack overflow.
* sysprof: disable runtime host symtab updates
* codehealth: fix the typo
* Simplify handling of instable types in TNEW/TDUP load forwarding.
* Respect jit.off() on pending trace exit.
* Limit exponent range in number parsing.
* Emit sunk IR_NEWREF only once per key on snapshot replay.

Closes #7937
Closes #8140
Part of #9145
Part of #9595

NO_DOC=LuaJIT submodule bump
NO_TEST=LuaJIT submodule bump

Bump test-run to new version with the following improvements:

- Bump luatest to 1.0.0-5-gf31fe34 [1]
- get_iproto_port: remove duplicates [2]
- requirements: bump gevent to 22.10.2 [3]
- Fix decoding error when reading server's log file [4]

[1] tarantool/test-run@bfcc9e8
[2] tarantool/test-run@da98d7f
[3] tarantool/test-run@bc1c473
[4] tarantool/test-run@434cbec

NO_DOC=test
NO_TEST=test
NO_CHANGELOG=test

(cherry picked from commit d75cd423)

This function returns a key_def part by a field number. However, currently
it returns NULL for parts that contain a JSON path to indexed data. Fix it.

Needed for tarantool/tarantool-ee#671

NO_DOC=bugfix
NO_CHANGELOG=not visible in CE

(cherry picked from commit cb0264c3)

We've had numerous problems with transaction boundaries in replication.
They were mostly caused by various cases when either the beginning or
end of the transaction happened to be a local row. Local rows are not
replicated, so the peer saw "corrupted" transactions with either no
beginning or no end flag, even though the transaction contents were
fine.

The problem with starting a transaction with a local row was solved in
commit f41d1ddd ("wal: fix tx boundaries"), and that fix seems to
continue working fine to this day.

The problem with ending transactions with a local row was first fixed
in commit 25382617 ("replication: append NOP as the last tx row"),
however there were problems with this approach: when a user tried to
write to local spaces on a replica from a replication trigger, it made
it impossible to ever start replicating from replica back to master.

Another fix was proposed: in commit f96782b5 ("relay: send rows
transactionally") we made relay read a full transaction into memory and
then send it all at once mangling with transanction start and end flags
when necessary.

After that the NOPs were removed in commit f5e52b2c ("box: get rid
of dummy NOPs after transactions ending with local rows"), since relay
became capable of fixing transaction boundaries itself.

Turns out the assumption that relay always sees a full transaction and
may correctly set transaction boundaries is wrong: when a replica
reconnects to master we set its starting vclock[0] to the one master has
at the moment of reconnect, so when recovery reads local rows with lsns
less than vclock[0] it silently skips them without showing them to
relay. When such skipped rows contain the is_commit flag for a currently
sent transaction we get the same problem as described before.

Let's make recovery track whether it has pushed any transaction rows to
relay or not, and if yes, recover rows with is_commit flag regardless of
whether the rows were already applied. To prevent recovering the same
data twice, recovery replaces such row contents with NOPs. Basically the
row is "recovered" only for the sake of showing its is_commit flag to
relay. Relay will skip the row anyway, since it remains local.

Follow-up #8958
Closes #9491

NO_DOC=bugfix

(cherry picked from commit 60d45765)

It doesn't make sense to assert that replica_id is correct in a row
after using that replica id to make some decisions based on it. Let's
switch the order of operations: first assert that replica_id is correct,
then compare row lsn with the already recovered one.

In-scope-of #9491

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

(cherry picked from commit 4ec51b4b)

This patch fixes the following error when building tarantool statically
with Clang compiler on a Linux system:

    error: argument unused during compilation: '-static-libstdc++'

Fixes #9646

NO_DOC=build issue
NO_TEST=build issue
NO_CHANGELOG=build issue

(cherry picked from commit 64afe49b)

It has been decided to have only release and debug builds for testing
on two latest macOS versions (at this momemnt, 13 and 14). Static and
LTO builds are dropped.

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

(cherry picked from commit 47940311)

The ChristopherHX/github-act-runner agent evaluates expression
`join(matrix.*, ', ')` to `Object` instead of `a, b, c`. So let's
have the facility to specify the job name manually to avoid the issue.

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

(cherry picked from commit 4007552a)

This patch fixes the following issue:

    Error: Unable to retrieve job ID by provided job name

The `${{ github.job }} (${{ join(matrix.*, ', ') }})` expression was
evaluated to `freebsd (Object)` instead of proper job name. It looks
like the ChristopherHX/github-act-runner agent has a bug somewhere.
Now the job name is made manually to bypass the issue.

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

(cherry picked from commit 9717c31f)

- Drop testing for FreeBSD 12 since FreeBSD 14 is available
- Add testing for FreeBSD 14

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

(cherry picked from commit 9ee99c5f)

- Drop testing for macOS 12 since macOS 14 is available
- Add testing for macOS 14

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

(cherry picked from commit 19f9786f)

The issue [1] is closed, and it looks like the problem doesn't exist
now. So removing the respective workaround from the action.

Also, removing the workaround with upgrading packages as it is not
needed anymore because now `brew` just gives a warning instead of
failing when the package is already installed.

[1] https://github.com/gevent/gevent/issues/1721

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

(cherry picked from commit 3142fb66)

From time to time, the step to upload job artifacts to S3 fails due to
the following error:

    Error: Unable to retrieve job ID by provided job name

It happens in the `s3-upload-artifact` action that tries to get job ID
by the name via the `get-job-id` action. Job ID is needed to compose the
proper artifact name. `get-job-id` performs a request to GitHub for the
list of workflow jobs and finds the job by its name and then gets its
ID.

For some reason, sometimes GitHub doesn't give the complete job list and
there is no chance to find the job and get its ID. At least, it has been
noticed for the osx.yml workflow with 36 parallel jobs.

It looks like the issue resides on the GitHub side and the simplest way
to resolve this is to ignore the failure and not fail the whole workflow
due to this GitHub bug.

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

(cherry picked from commit 5137ea59)

When the `freebsd` workflow has become a matrix one in 0bd15468
("ci: mv freebsd testing from per-commit to nightly"), we forgot
to do the corresponding changes in the `Upload artifacts to S3` step.
So adding missing changes.

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

(cherry picked from commit 5fa3d5a2)

When `${{ matrix.tarantool-branch}}` is `release/3.0` or similar, the
step to save build artifacts will fail with the following error:

    Error: Artifact name is not valid: osx-13-x86_64-release/3.0-debug.
    Contains the following character:  Forward slash /

This commit fixes the issue.

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

(cherry picked from commit 64363b43)

<box/net.box_reconnect_after_gh-3164.test.lua> and
<vinyl/tx_gap_lock.test.lua> checking some objects that should be
collected may fail because these objects are referenced as the upvalues
for a function inlined on some JIT trace. The objects aren't leaked;
tests don't consider the JIT semantics. See more details in [1].

This patch disables JIT for the first test.

The second test itself isn't affected since the commit
tarantool/test-run@33fbace8307e64e66b0b2178ac952b2697673917 ("Add
default server restart to each test run"). Because of this commit, the
`tx.gap_locks` can't be affected by the use of the `create_iterator()`
function in other tests.
But to avoid similar situations in the future, the JIT is disabled for
the `res.next()` function in test utils.

[1]: https://github.com/tarantool/tarantool/wiki/LuaJIT-function-inlining

Closes tarantool/tarantool-qa#233
Closes tarantool/tarantool-qa#276

NO_DOC=fix flaky test
NO_CHANGELOG=see NO_DOC

(cherry picked from commit 1a5e3bf3)

The patch updates curl module to the version 8.5.0 [1][2] that brings a
number of functional fixes and security fix for CVE-2023-46219 (HSTS
long file name clears contents), see description in [2], and updates
CMake module for building curl library.

Changes in CMake module:

- Option `CURL_DISABLE_HEADERS_API` was added and disabled by default [4].
- Option `CURL_DISABLE_BINDLOCAL` was added and disabled by default [5].
- Option `CURL_DISABLE_INSTALL` was added and disabled by default [6].

1. https://curl.se/changes.html#8_5_0
2. https://github.com/curl/curl/compare/curl-8_4_0...curl-8_5_0
3. https://curl.se/docs/CVE-2023-46219.html
4. https://github.com/curl/curl/commit/33493db2af2dc6d9910f5d7c702aae6f63b8a6a6
5. https://github.com/curl/curl/commit/20bb363f25151febe9026b41b8ad65df6db20b68
6. https://github.com/curl/curl/commit/aace27b0965c10394544d1dacc9c2cb2fe0de3d3

NO_DOC=libcurl submodule bump
NO_TEST=libcurl submodule bump

(cherry picked from commit b8d733df)

The commit "lib: add ability to disable auths individually" [1][2] in
Curl 8.3.0 removes CURL_DISABLE_CRYPTO_AUTH and introduces CMake options
for a number crypto protocols.

The patch reflects this change in Tarantool's build infrastructure.

1. https://github.com/curl/curl/commit/e92edfbef64448ef461117769881f3ed776dec4e
2. https://github.com/curl/curl/pull/11490

Follows up #9086

NO_CHANGELOG=third_party
NO_DOC=third_party
NO_TEST=third_party

(cherry picked from commit c9daf9bf)

The commit "cmake: improve OpenLDAP builds" [1][2] in Curl deletes CMake
option CURL_USE_OPENLDAP.

The patch reflects this change in Tarantool's build infrastructure.

1. https://github.com/curl/curl/commit/751e168d93b4a58f3fbbe2908c0041ae2f934329
2. https://github.com/curl/curl/pull/12024

Follows up #9086

NO_CHANGELOG=third_party
NO_DOC=third_party
NO_TEST=third_party

(cherry picked from commit 7e3dc177)

`httpc` module has two GC-finalizers: the first one for a Lua http
client (C function `luaT_httpc_cleanup`) and the second one for a Lua
http chunked requests (C function `luaT_httpc_io_cleanup`) introduced in
commit 417c6cb7 ("httpc: introduce stream input/output interface").
In a C implementation HTTP requests depends on structures of HTTP client
and there is a problem with destroying Lua objects in `httpc` module -
these GC-finalizers are not synchronized. This could lead to at least
two problems:

There is a race with GC-finalization that leads to use-after-free errors
when HTTP client is collected before collecting HTTP request. In a
stacktrace the problem looks as below:

```
0x55ca7d47652e in crash_collect+256
0x55ca7d476f6a in crash_signal_cb+100
0x7fb876c42520 in __sigaction+80
0x55ca7d641e51 in curl_slist_free_all+35
0x55ca7d441498 in httpc_request_delete+45
0x55ca7d4653f1 in httpc_io_destroy+27
0x55ca7d4674bc in luaT_httpc_io_cleanup+36
0x55ca7d4e00c7 in lj_BC_FUNCC+70
0x55ca7d4f8364 in gc_call_finalizer+668
0x55ca7d4f8946 in gc_finalize+1387
0x55ca7d4f91e2 in gc_onestep+864
0x55ca7d4f9716 in lj_gc_fullgc+276
...
```

Lua object `http.client` could be GC-collected when chunked HTTP request
is alive. This will lead to an error "IllegalParams: io: request must be
io" because we call a method when Lua object is already a `nil`.

```lua
local url = 'https://bronevichok.ru/'
local c = require('http.client').new()
local r = c:get(url, {chunked = true})
c = nil
collectgarbage()
collectgarbage()
r:read(1) -- IllegalParams: io: request must be io
```

The patch introduces two functions: `httpc_env_finish` and
`curl_env_finish`, that prepares curl and httpc environments for
destruction. HTTP client's GC finalizer now calls `httpc_env_finish`
instead of `httpc_env_destroy`, this prevents from destroying memory
that could be in use by HTTP requests. Additionally `httpc_env_finish`
sets a flag `cleanup`. HTTP environment destroying is called when flag
`cleanup` is set and a there are no active HTTP requests. The main idea
of the patch is a synchronization of destructors for HTTP client and
HTTP chunked requests. Unfortunately, GC will eventually collect HTTP
client object after calling its `__gc`. To prevent this we put a
reference to a Curl's userdata in Lua objects with HTTP chunked requests
and HTTP default client.

Fixes #9346
Fixes #9453

NO_DOC=bugfix

(cherry picked from commit 17e9c6ff)