The goal is to make sure that no files except box.cc can change
instance_vclock_storage directly. That leads to all sorts of hacks
which in turn lead to bugs - #9916 is a good example.

Now applier on final join only sends rows into the journal. The
journal then is handled by box.cc where vclock is properly
updated.

Part of #10113

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

The function writes a single xrow into the journal in a blocking
way. It isn't so simple, so makes sense to keep as a function,
especially given that it will be used more in the next commit.

Part of #10113

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Recovery journal uses word "recovery" to say that it works with
xlogs. For snapshot recovery there is bootstrap_journal. Lets use
it during local snapshot recovery.

The reasoning is that while right now there is no difference, in
next commits the recovery_journal will do more.

Part of #10113

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Storing vclock of the instance in replicaset.vclock wasn't right.
It wasn't vclock of the whole replicaset. It was local to this
instance. There is no such thing as "replicaset vclock".

The patch moves it to box.h/cc.

Part of #10113

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Applier during the registration waiting (for registering a new ID
or a name) could keep doing the master txns received before the
registration was started. They could still be inside WAL doing a
disk write, when the replica sends a register request.

Before this commit, it could cause an assertion failure in debug
and a double LSN error in release.

The reason was that during the registration waiting the applier
treated all incoming txns as "final join" txns. I.e. it wasn't
checking if those txns were already received, but not committed
yet.

During normal subscribe process the appliers (potentially
multiple) protect themselves from that by keeping track of the
vclocks which are already applied and also being applied right now
(replicaset.applier.vclock).

Such protection ensures that receiving same row from 2 appliers
wouldn't result into its double write. It also protects from the
case when a txn was received, goes to WAL, but then the applier
reconnects, resubscribes, and gets the same txn again - it
shouldn't be applied.

The patch makes so that the registration waiting after recovery
works like subscribe. Registration during recovery would mean
bootstrap via join. And outside of recovery it means the instance
is already running.

Closes #9916

NO_DOC=bugfix

As this fiber is made system in the commit bf620650 ("box: finish
client fibers on shutdown") we don not need the existing protection from
cancelling. So first remove it. Now make it managed on shutdown.

Note that we may have issues as we finish this fiber too early. The
tasks scheduled but not executed at this moment will never be executed.
So the tasks that be scheduled after fiber is finished. Now when we
don't use worker fiber for swim gc this will not cause leaks. And
leaking fd on Tarantool shutdown in fio is not a problem.

Closes #9722

NO_CHANGELOG=internal
NO_DOC=internal

Let's make sure box raft worker fiber is finished on Tarantool shutdown as
we are going to free fibers stacks. If fiber is not finished it's stack
may have references to objects on heap. Thus as fiber stack will be freed
we will have FP memory leaks under ASAN.

Part of #9722

NO_TEST=rely on existing tests
NO_CHANGELOG=internal
NO_DOC=internal

Let's make sure swim worker fiber is finished on Tarantool shutdown as
we are going to free fibers stacks. If fiber is not finished it's stack
may have references to objects on heap. Thus as fiber stack will be freed
we will have FP memory leaks under ASAN.

Let's make swim gc do not yield using asynchronuos deletion. This way we
will not use worker fiber for swim deletion. We are going to stop this
worker fiber before all swim object are collected.

Part of #9722

NO_CHANGELOG=internal
NO_DOC=internal

Let's make sure net.box system fiber is finished on Tarantool shutdown as
we are going to free fibers stacks. If fiber is not finished it's stack
may have references to objects on heap. Thus as fiber stack will be freed
we will have FP memory leaks under ASAN.

Part of #9722

NO_CHANGELOG=internal
NO_DOC=internal

Such fibers are cancelled during Tarantool shutdown the same way as
cancelled client fibers. This is internal API.

Part of #9722

NO_CHANGELOG=internal
NO_DOC=internal

Fiber pool shutdown is finishing all idle fibers. Any message processing
is finished earlier on client fiber shutdown.

We need some changes in shutdown order to make fiber pool shutdown.
First we need to move stopping of iproto threads from free to shutdown
step. The issue is we want to destroy "tx" endpoint which requires all
pipes connected to it to be destroyed first. There are pipes in iproto
threads that connected to "tx". Currently we destroy pipes in free step
and at this point as there is no event loop in tx thread
`cbus_endpoint_destroy` can't receive notifications that pipes are
destroyed.

Originally we put stopping of iproto threads to the free step because we
don't have client fibers shutdown. So it was convenient to have working
`net_pipe` so that client fibers can use iproto API without adding extra
logic to them. Now I guess it make sense to stop client fibers before
iproto shutdown. This is the second change in shutdown order.

There is another reason why we have iproto shutdown before client fiber
shutdown. In the process of iproto shutdown we close connections first
and then cancel all requests in progress. This way client do not receive
unexpected `FiberIsCancelled` errors in the process of server shutdown.
After the patch it not so. Well we may close connections as an extra
step before client fibers shutdown. But let's leave it this way.  Good
clients can subscribe to servere shutdown and prepare for it.  Otherwise
they may receive `FiberIsCancelled` for theier request which looks
sensible.

It is also makes sense now to move watcher and client fiber shutdown
to `box_shutdown` as we can both use watcher and create client fibers
without starting a box.

While at it also drop a note in code why we shutdown watcher before even
fiber clients shutdown.

Part of #9722

NO_CHANGELOG=internal
NO_DOC=internal

As we are going to check for memory leaks in ASAN we need to wait while
memtx gc finishes freeing tuples of dropped primary index or keys of
dropped functional indexes.

Part of #9722

NO_CHANGELOG=internal
NO_DOC=internal

It it same as ERROR_INJECT_YIELD() but if fiber is cancelled it will
execute given code.

Part of #9722

NO_TEST=internal
NO_CHANGELOG=internal
NO_DOC=internal

There may be more than one fiber waiting on `vy_scheduler::dump_cond`:

```
box.snapshot
  vinyl_engine_wait_checkpoint
    vy_scheduler_wait_checkpoint

space.create_index
  vinyl_space_build_index
    vy_scheduler_dump
```

To avoid hang, we should use `fiber_cond_broadcast`.

Closes #10233

NO_DOC=bug fix

This patch bumped small to the new version that does not trigger
UBSan with *_entry* macros and should support new oss-fuzz builder.

New commits:

* rlist: make its methods accept const arguments
* lsregion: introduce lsregion_to_iovec method
* rlist: make foreach_enrty_* macros not to use UB

Fixes: #10143

NO_DOC=small submodule bump
NO_TEST=small submodule bump
NO_CHANGELOG=small submodule bump

Changed default tarantool `offsetof` macro implementation so it don't
access members of null pointer in typeof that triggers UBsan.

Needed for #10143

NO_DOC=bugfix
NO_CHANGELOG=minor
NO_TEST=tested manually with fuzzer

This patch completely relaxes UUID checks and accepts an arbitrary
128-bit sequence as an UUID for binary data. String representations
still should match the grammars in RFC 4122, Section 3 [1] and RFC 9562,
Section 4 [2].

[1]: https://datatracker.ietf.org/doc/html/rfc4122#section-3
[2]: https://datatracker.ietf.org/doc/html/rfc9562#name-uuid-format

Closes #5444

@TarantoolBot document
Title: uuid: relaxed UUID validation

[The UUID module documentation][1] mentions that Tarantool generates
UUIDs following the rules for RFC 4122,[version 4, variant 1][2]. It is
worth mentioning that the user can store an arbitrary 128-bit sequence
as an UUID for binary data. String representations still should match
the grammars in RFC 4122, [Section 3][3], and RFC 9562, [Section 4][4].

[1]: https://www.tarantool.io/en/doc/latest/reference/reference_lua/uuid/
[2]: https://en.wikipedia.org/wiki/Universally_unique_identifier#Version_4_(random)
[3]: https://datatracker.ietf.org/doc/html/rfc4122#section-3
[4]: https://datatracker.ietf.org/doc/html/rfc9562#name-uuid-format

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

* Correct fix for stack check when recording BC_VARG.
* test: remove inline suppressions of _TARANTOOL
* FFI: Fix ffi.alignof() for reference types.
* FFI: Fix sizeof expression in C parser for reference types.
* FFI: Allow ffi.metatype() for typedefs with attributes.
* FFI: Fix ffi.metatype() for non-raw types.
* Maintain chain invariant in DCE.
* build: introduce option LUAJIT_ENABLE_TABLE_BUMP
* ci: add tablebump flavor for exotic builds
* test: allow `jit.parse` to return aborted traces
* Handle all types of errors during trace stitching.
* Use generic trace error for OOM during trace stitching.
* Check for IR_HREF vs. IR_HREFK aliasing in non-nil store check.
* cmake: set cmake_minimum_required only once
* cmake: fix warning about minimum required version
* ci: add a workflow for testing with AVX512 enabled
* test: introduce a helper read_file
* OSX/iOS/ARM64: Fix generation of Mach-O object files.
* OSX/iOS/ARM64: Fix bytecode embedding in Mach-O object file.
* build: introduce LUAJIT_USE_UBSAN option
* ci: enable UBSan for sanitizers testing workflow
* cmake: add the build directory to the .gitignore
* Prevent sanitizer warning in snap_restoredata().
* Avoid negation of signed integers in C that may hold INT*_MIN.
* Show name of NYI bytecode in -jv and -jdump.

Closes #9924
Closes #8473

NO_DOC=LuaJIT submodule bump
NO_TEST=LuaJIT submodule bump

In this case join will just hang. Instead let's raise an error in case
of Lua API and panic in case of C API.

Closes #10196

NO_DOC=minor

When fiber is accessed from Lua we create a userdata object and keep the
reference for future accesses. The reference is cleared when fiber is
stopped. But if fiber is joinable is still can be found with
`fiber.find`. In this case we create userdata object again.
Unfortunately as fiber is already stopped we fail to clear the
reference. The trigger memory that clear the reference is also leaked.
As well as fiber storage if it is accessed after fiber is stopped.

Let's add `on_destroy` trigger to fiber and clear the references there.

Note that with current set of LSAN suppressions the trigger memory leak
of the issue is not reported.

Closes #10187

NO_DOC=bugfix

The module is renamed from `internal.config.utils.schema` to
`experimental.config.utils.schema` without changes.

It is useful for validation of configuration data in roles and
applications.

Also, it provides a couple of methods that aim to simplify usual tasks
around processing of hierarchical configuration data. For example,

* get/set a nested value
* apply defaults from the schema
* filter data based on annotations from the schema
* transform a hierarchical data using a function
* merge two hierarchical values
* parse environment variable according to its type in the schema

See https://github.com/tarantool/doc/issues/4279 for an in-depth
description.

Fixes #10117

NO_DOC=https://github.com/tarantool/doc/issues/4279

We just don't free functional index keys on functional index drop now.
Let's approach keys deletion as in the case of primary index drop ie
let's drop these keys in background.

We should set `use_hint` to `true` in case of MEMTX_TREE_VTAB_DISABLED
tree index methods because `memtx_tree_disabled_index_vtab` uses
`memtx_tree_index_destroy<true>`. Otherwise we get read outside of index
structure for stub functional index on destroy for introduced `is_func`
field (which is reported by ASAN).

Closes #10163

NO_DOC=bugfix

The patch updates curl module to the version 8.8.0 [1] plus
a number of commits in a range curl-8_8_0..30de937bda0f because
it includes a fix for a regression [2] caught on the previous bump.
The new version brings a number of functional fixes.

Previous changelog entry has been removed because duplicate
entries about bumps in release changelog confuses end users.

Closes #9612

1. https://curl.se/changes.html#8_8_0
2. https://github.com/curl/curl/issues/13740

NO_DOC=libcurl submodule bump
NO_TEST=libcurl submodule bump

The patch updates curl module to the version 8.7.1 [1][2] that
brings a number of functional and security fixes, and updates
CMake module for building curl library.

Security fixes:

- CVE-2024-2004: Usage of disabled protocol. (low)
- CVE-2024-2398: HTTP/2 push headers memory-leak. (medium)
- CVE-2024-2379: QUIC certificate check bypass with wolfSSL. (low)
- CVE-2024-2466: TLS certificate check bypass with mbedTLS. (medium)

Changes in CMake module:

- Option `USE_OPENSSL_QUIC` was added and disabled by default [3]

Previous changelog entry has been removed because duplicate
entries about bumps in release changelog confuses end users.

The bump was blocked by a regression in libcurl [4][5].

1. https://curl.se/changes.html#8_7_1
2. https://github.com/curl/curl/compare/curl-8_6_0...curl-8_7_1
3. https://github.com/curl/curl/commit/8e741644a229c3791963b4f5cae1dcfccba842dd
4. https://curl.se/mail/lib-2024-03/0059.html
5. https://github.com/curl/curl/issues/13260

NO_DOC=libcurl submodule bump
NO_TEST=libcurl submodule bump

The patch updates curl module to the version 8.6.0 [1][2] that
brings a number of functional fixes, and updates CMake module for
building curl library.

Changes in CMake module:

- Option `ENABLE_CURL_MANUAL` was added and disabled by default [3]
- Option `BUILD_LIBCURL_DOCS` was added and disabled by default [3]

The patch follows up commit 9bdf2bab ("httpc: fix reading data
in a chunked request") where curl submodule was updated to
a version based on 8.5.0 release with applied patch with fix [4].

Previous changelog entry has been removed because duplicate
entries about bumps in release changelog confuses end users.

This bump was blocked by a regression in libcurl [5].

1. https://curl.se/changes.html#8_6_0
2. https://github.com/curl/curl/compare/curl-8_5_0...curl-8_6_0
3. https://github.com/curl/curl/commit/a808aab06851d4364ab1773c664df3d906a497a9
4. https://github.com/curl/curl/commit/cdd905a9854305657ebbe645095e1189dcda28c7
5. https://github.com/curl/curl/commit/b8c003832d730bb2f4b9de4204675ca5d9f7a903

NO_DOC=libcurl submodule bump
NO_TEST=libcurl submodule bump

etcd configuration section allows to connect to TLS-encrypted etcd
cluster, providing a way to pass `ssl.ssl_key`. But it is not enough
when etcd server have client cert auth enabled and has a CA file, since
it requires a ssl_cert as well. Actually, propagating ssl_cert is
already a part of the EE connect code [1], we just missing the top-level
config option.

Fixes https://github.com/tarantool/tarantool-ee/issues/827

1. https://github.com/tarantool/tarantool-ee/blame/1138443c46e7a6e1bb855277bc6cb3333240131c/src/box/lua/config/source/etcd.lua#L103

@TarantoolBot document
Title: config: add missing ssl.ssl_cert for etcd

etcd configuration section already allows to set `ssl.ssl_key`. Now it
also allows to pass `ssl.ssl_cert`.

listen() on Mac used to take SOMAXCONN as the backlog size. It is
just 128, which is too small when connections are incoming too
fast. They get rejected.

Increase of the queue size wasn't possible, because the limit was
hardcoded. But now sio takes the runtime limit from
kern.ipc.somaxconn sysctl setting.

One weird thing is that when set too high, it seems to have no
effect, like if nothing was changed. Specifically, values above
32767 are not doing anything, even though stay visible in
kern.ipc.somaxconn.

It seems listen() on Mac internally might be using 'short' or
int16_t to store the queue size and it gets broken when anything
above INT16_MAX is used. The code truncates the queue size to this
value if the given one is too high.

Closes #8130

NO_DOC=bugfix
NO_TEST=requires root privileges for testing

This patch is the follow-up for the commit
49946a72 ("ci: send perf statistics to InfluxDB").
Since secrets are unavailable for fork repositories, the sending step
fails due to a missed InfluxDB URL and token. This patch allows to run
this step only for on push events or PRs from the main repository
itself.

NO_DOC=CI
NO_CHANGELOG=CI
NO_TEST=CI

Tarantool has hardcoded list of versions it can downgrade to. This list
should consist of all the released versions less than Tarantool version.
This workflow helps to make sure we update the list before release.

It is run on pushing release tag to the repo, checks the list and fails
if it misses some released version less than current. In this case we
are supposed to update downgrade list (with required downgrade code) and
update the release tag.

Closes #8319

NO_TEST=ci
NO_CHANGELOG=ci
NO_DOC=ci

This patch adds an additional steps in the <perf_micro.yml> workflow to
aggregate and send aggregated data to InfluxDB via curl.

Also, this patch adds the corresponding environment variables to be used
during workflow to preserve the original commit hash and branch name.

NO_DOC=CI
NO_CHANGELOG=CI
NO_TEST=CI

This patch adds a helper script to aggregate the benchmark results from
JSON files to the format parsable by the InfluxDB line protocol [1].

All JSON files from the <perf/output> directory are benchmark results
and aggregated into the <perf/output/summary.txt> file that can be
posted to the InfluxDB. The results are aggregated via the new target
test-perf-aggregate, which is run only if some JSON files with results
are missed.

[1]: https://docs.influxdata.com/influxdb/v2/reference/syntax/line-protocol/

NO_DOC=perf test
NO_CHANGELOG=perf test
NO_TEST=perf test

This file can be used to compare the results of Lua benchmarks. Since it
has a general purpose, it is moved to the <perf/tools> directory.

NO_DOC=perf test
NO_CHANGELOG=perf test
NO_TEST=perf test

This patch saves the output of the performance tests in the JSON format
to be processed later. The corresponding directory is added to the
<.gitignore>.

NO_DOC=perf test
NO_CHANGELOG=perf test
NO_TEST=perf test

This patch considers the number of iterations as the number of items
proceeded by the corresponding benchmark, so it may be used for the
`items_per_second` counter.

NO_DOC=perf test
NO_CHANGELOG=perf test
NO_TEST=perf test

The output now contains items per second without the mean time in
seconds. The number of iterations is reduced to 40 to avoid running the
test too long. The `wal_mode` option (default is "none") is set via
command line flags, as far as the number of nodes (default is 10). Also,
the master nodes are set up via the `popen()` command without using any
Makefile.

Also, two new options are introduced:
* The `--output` option allows you to specify the output file.
* The `--output_format` option means the format for the printed output.
  The default is "console". It prints items proceeded per second to the
  stdout. The "json" format contains all the information about the
  benchmark in a format similar to Google Benchmark's.

Usually, these options should be used together to dump machine-readable
results for the benchmarks.

NO_DOC=perf test
NO_CHANGELOG=perf test
NO_TEST=perf test

This patch renames subtests in column scan to avoid the usage of `,`
(the separator) in the tag name for the InfluxDB report.

NO_DOC=perf test
NO_CHANGELOG=perf test
NO_TEST=perf test

The output now contains items per second instead of time in seconds.

Also, two new options are introduced:
* The `--output` option allows you to specify the output file.
* The `--output_format` option means the format for the printed output.
  The default is "console". It prints rows proceeded per second to the
  stdout. The "json" format contains all the information about the
  benchmark in a format similar to Google Benchmark's.

Usually, these options should be used together to dump machine-readable
results for the benchmarks.

NO_DOC=perf test
NO_CHANGELOG=perf test
NO_TEST=perf test

The output now contains items per second instead of time in nanoseconds.

Also, two new options are introduced:
* The `--output` option allows you to specify the output file.
* The `--output_format` option means the format for the printed output.
  The default is "console". It just prints the number of iterations
  proceeded per second to the stdout. The "json" format contains all the
  information about the benchmark in a format similar to Google
  Benchmark's.

Usually, these options should be used together to dump machine-readable
results for the benchmarks.

NO_DOC=perf test
NO_CHANGELOG=perf test
NO_TEST=perf test