If addition of a tuple is rolled back while the corresponding
story is needed for something else (for example it stores a read
set of another transaction) - the story cannot be deleted.
Now there's a special flag `rollbacked` that is set to true
for such stories, and the flag must be considered in places
where history chains are scanned. That approach also requires
psn to be set for rolled-back transactions, which surprisingly
not as simple as it to say. All that makes the code complicated
and hard to maintain.

There's another approach for managing rolled back stories: simply
set their del_psn to a low enough value (lower than any existing
transaction's PSN) and (if necessary) push them to the end of
history chain. Such a story would be invisible to any transaction
due to already existing mechanisms, that's what is needed.

In order to provide "low enough" del_psn it will be natural to
assign real PSN starting from some predefined value, so any value
below that predefined value will be less that any existing PSN and
thus "low enough".

Implement this more simple approach.

Part of #8648
Part of #8654

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

(cherry picked from commit ba394a58)

That's strange, but in this test in a group of simple test cases
there are test cases that checks replaces, updates and deletes,
but occasionally there's no test case that checks inserts.

Fix it and add simple test cases for inserts.

No logical changes.

Part of #8648
Part of #8654

NO_DOC=new test case
NO_CHANGELOG=new test case

(cherry picked from commit 37b4561f)

The description of CURLOPT_UPLOAD [1] is confusing:

    If you use PUT to an HTTP 1.1 server, you can upload data without
    knowing the size before starting the transfer if you use chunked
    encoding. You enable this by adding a header like
    "Transfer-Encoding: chunked" with CURLOPT_HTTPHEADER.

In fact, libcurl adds this header itself. Actually we need to avoid
adding this header by libcurl with CURLOPT_INFILESIZE [2].

The CURLOPT_UPLOAD documentation has been updated [3].

1. https://github.com/curl/curl/blob/555bacd6d522bcca497573765056354f4d5d7b33/docs/libcurl/opts/CURLOPT_UPLOAD.3
2. https://curl.se/libcurl/c/httpput.html
3. https://github.com/curl/curl/pull/11300

Closes #8744

NO_DOC=bugfix

(cherry picked from commit e2e5ef4b)

This reverts commit 1c0a0c90.

Cherry-picked by mistake. Tarantool 2.11 still depends on libgomp.

NO_DOC=revert
NO_TEST=revert
NO_CHANGELOG=revert

It was disabled because ligomp.a on some AArch64 systems (e.g. CentOS)
is compiled without PIC support. Now Tarantool doesn't depend on ligomp,
so it's safe to turn on PIC for AArch64 systems.

Follow-up #7536

NO_DOC=build
NO_TEST=build

(cherry picked from commit 5c1968ca)

It was disabled because dlsym tests failed when PIC is on (#7640).
However, later it turned out that the issue is not related to PIC, and
LuaJIT was turned off for such tests by commit 67e79b15 ("test: turn
LuaJIT off in tests with dlsym"). Now it's safe to turn on PIC for FreeBSD.

Follow-up #7536

NO_DOC=build
NO_TEST=build

(cherry picked from commit 05eba830)

It was decided to include the `osx_debug.yml` and `osx_release.yml`
workflows to the default tests run (without the `full-ci` label).
Now we can get test results for macOS faster and without an extra
load on CI.

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

(cherry picked from commit de404cb0)

Commit/rollback triggers are run asynchronously, upon receiving the
write status from WAL. We can't run them in the original fiber that
submitted the WAL request, because it would open a time window between
writing a transaction to WAL and committing it in tx, which could lead
to violating the cascading rolback principles. As a result,
commit/rollback triggers run with admin privileges.

The issue was already solved for confirming async transaction, but
session and user are still not correct, when the transaction is
confirmed by the limbo. Let's fix this issue by temporarily setting
session and credentials to the original fiberfor running
commit/rollback triggers.

Closes #8742

NO_DOC=bugfix

(cherry picked from commit 8cd0cd09)

Currently some transactions on synchronous space fail to complete with
the `ER_CURSOR_NO_TRANSACTION` error, when on_rollback/on_commit triggers
are set.

This is caused due to the fact, that some rollback/commit triggers
require in_txn fiber variable to be set but it's not done when a
transaction is completed from the limbo. Callbacks, which are used to
work with iterators (`lbox_txn_pairs` and `lbox_txn_iterator_next`),
acquire tnx statements from the current transactions, but they cannot
do that, when this transaction is not assigned to the current fiber, so
`ER_CURSOR_NO_TRANSACTION` is thrown.

Let's assign in_txn variable when we complete transaction from the limbo.
Moreover, let's add assertions, which check whether in_txn() is correct,
in order to be sure, that `txn_complete_success/fail` always run with
in_txn set.

Closes #8505

NO_DOC=bugfix

(cherry picked from commit 6fadc8a0)

See the docbot request for details.

Closes #7932

@TarantoolBot document
Title: correct recovery of mixed transactions

In this patch implemented correct recovery of mixed transactions. To do
this, set  `box.cfg.force_recovery` to `true`. If you need to revert to
the old behavior, don't set the `force_recovery` option.

What to do when one node feeds the other a xlog with mixed transactions?
Let there be two nodes (`node#1` and `node#2`). And let the data be
replicated from `node#1` to `node#2`. Suppose that at some point in time,
`node#1` is restoring data from an xlog containing mixed transactions. To
replicate data from `node#1` to `node#2`, do the following:
1. Stop `node#2` and delete all xlog files from it
2. Restart `node#1` by setting `force_recovery` to `true`
3. Make `node#2` rejoin to `node#1` again.

(cherry picked from commit 2b1c8713)

A new macros have been introduced because OOM errors are not handled in the
code.

Needed for #7932

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

(cherry picked from commit 53857148)

box.cfg.force_recovery used to be needed only during box.cfg() in
a few places, but its usage is going to extend.

In future commits about cluster/replicaset/instance names it will
be needed to allow rename. It won't be entirely legal (hence can't
be done without any flags), but won't be fully illegal either.

The "valid" rename will be after upgrading, when an
old cluster updated to a new version and wants to start using the
names. Then it will have to set force_recovery, set the names,
sync the instances, drop force_recovery. One-time action to allow
old installations use the new feature - the names.

Part of #5029

NO_DOC=refactoring
NO_CHANGELOG=refactoring
NO_TEST=already covered

(cherry picked from commit ae4c96c7)

After the commit a392eb76 ("box: destroy its modules even if box.cfg
not done") wal_free() on exit is called even if box.cfg() isn't finished
yet, and thus wal isn't yet enabled. This leads to a bug when wal would
unconditionally write a 00...0.xlog regardless of the actual xlog
directory contents or the actual vclock.

Let's fix this by not writing an empty xlog on exit if wal writer's
vclock is zero. It means the writer either wasn't enabled yet, or was
enabled, but hasn't written anything. In both cases writing the empty
xlog is extraneous.

Closes #8704

NO_DOC=bugfix

(cherry picked from commit f78908c9)

There was a problem in applier_wait_bootstrap_leader_uuid_is_set():
 * it didn't set an error cause when the ballot watcher was dead,
   leading to a segfault when trying to diag_add() to the cause.
 * it didn't expect the ballot watcher to exit without an error
   before the bootstrap leader uuid is known, hanging forever.

The first issue could be reproduced by trying to bootstrap a replicaset
consisting of both "old" (Tarantool 2.10 or less) and "new" instances.
Or by bootstrapping a replicaset consisting of "new" instances and
stopping some of them in a specific manner.

The second issue could be reproduced only by manually broadcasting an
empty "internal.ballot" event.

Fix both issues. The first one by setting an ER_UNKNOWN error when
the ballot watcher fiber is dead. And the second one by checking if the
ballot watcher has died during waiting for the ballot update.

The test only covers the first issue: the second one can only happen
with manual intervention and is hard to test: it involves broadcasting
an empty "internal.ballot" from the replica while it's still connecting
to remote nodes during an initial `box.cfg{}` call.

Closes #8757

NO_DOC=bugfix

(cherry picked from commit 1c25efb4)

Closes #8716

NO_DOC=bugfix

(cherry picked from commit 3d0afc60)

`string` with JOIN option is only available since CMake 3.12, but we have
developers using CMake 3.1: implement a utility `string_join` function to
remove this dependency.

Closes #5881

NO_CHANGELOG=<build fix>
NO_DOC=<build fix>
NO_TEST=<build fix>

(cherry picked from commit 55298308)

Reported by ASAN. The issue was fixed in the master branch in commit
b9550f19 ("box: support space and index names in IPROTO requests").

NO_TEST=asan
NO_DOC=bug fix
NO_CHANGELOG=minor

The xrow_decode_* functions are written in such a way that they ignore
unknown IPROTO keys. This is required for connectivity between different
Tarantool version. However, there's bug in the code connected with the
value type checking: we fail if the key is >= iproto_key_MAX. This
worked fine as long as we added new IPROTO keys in the middle of the key
space, without bumping iproto_key_MAX, but this assumption broke when we
added IPROTO_AUTH_TYPE. The issue is exacerbated by the fact that
IPROTO_AUTH_TYPE is used by IPROTO_ID, which is sent unconditionally on
connect. Let's fix the value type check and add some tests.

Notes:
 - xrow_decode_heartbeat turns out to be unused. Drop it.
 - Fix the net.box helpers response_body_decode and netbox_decode_table
   to handle unknown keys and empty body. This is needed to properly
   decode a response to an injection in tests.
 - Testing unknown keys in replication requests would be complicated.
   Instead we add a bunch of unit tests.
 - Convert the xrow unit test to TAP.

Closes #8745

NO_DOC=bug fix

(cherry picked from commit ee0660b8)

Response body decoding of DML and call/eval requests is very ad-hoc and
hard to extend: introduce a new `response_body_decode` helper that decodes
the response body similarly to `xrow_decode_dml` — this will allow to
separate decoding from processing.

Needed for #8147

NO_CHANGELOG=refactoring
NO_DOC=refactoring
NO_TEST=refactoring

(cherry picked from commit 1d6043fa)

Fix the `box-tap/gh-5602-environment-vars-cfg.test.lua` diff test after
disabling stdout bufferization in test-run.

NO_DOC=test
NO_TEST=test
NO_CHANGELOG=test

(cherry picked from commit d2d06824)

Bump test-run to new version with the following improvements:

- lint: update flake8 to 5.* [1]
- app: write test output directly to files [2]
- app: disable stdout bufferization [3]
- luatest: fix luatest command description [4]
- luatest: split stdout and stderr [5]
- luatest: write stdout to a file directly [6]
- luatest: drop a dead code [7]
- luatest: disable stdout buffering [8]
- Prettify messages about log files [9]

[1] https://github.com/tarantool/test-run/commit/4916351
[2] https://github.com/tarantool/test-run/commit/7b3c003
[3] https://github.com/tarantool/test-run/commit/48ccad8
[4] https://github.com/tarantool/test-run/commit/2756eb2
[5] https://github.com/tarantool/test-run/commit/3dab8dc
[6] https://github.com/tarantool/test-run/commit/0c24557
[7] https://github.com/tarantool/test-run/commit/036e6b6
[8] https://github.com/tarantool/test-run/commit/e65a5f8
[9] https://github.com/tarantool/test-run/commit/7a1ec4f

NO_DOC=test
NO_TEST=test
NO_CHANGELOG=test

(cherry picked from commit f0d94f8b)

Bump test-run to new version with the following improvements:

- Drop setting OMP_NUM_THREADS

Part of #3389

NO_DOC=testing stuff
NO_TEST=testing stuff
NO_CHANGELOG=testing stuff

(cherry picked from commit f65900ab)

The memtx engine has two quota consumers: `index_slab_cache` and
`slab_cache`. They both acquire SLAB_SIZE of space on first tuple
insert in "small" allocator mode, so the quota less than the slab
size times two exceeds right there.

It was decided to introduce a theoretically minimal size of the
quota and round-up any value given in box.cfg if it's less than
the lower bound.

Closes #7389

NO_DOC=bugfix

(cherry picked from commit de0731bd)

Fix the `test-sdk` job in the `submodule_update.yml` workflow according
to the new name policy for supported branches (2.10 -> release/2.10,
2.11 -> release/2.11).

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

(cherry picked from commit 83158695)

README.md in the `report-job-status` action was fixed in accordance
with the new name policy for supported branches (2.10 -> release/2.10,
2.11 -> release/2.11).

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

(cherry picked from commit fa33c5e5)

The condition for uploading test artifact to S3 was fixed in accordance
with the new name policy for supported branches (2.10 -> release/2.10,
2.11 -> release/2.11).

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

(cherry picked from commit 76c79e4e)

The condition for workflow concurrency was fixed in accordance with
the new name policy for supported branches (2.10 -> release/2.10,
2.11 -> release/2.11).

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

(cherry picked from commit 0c117197)

The push trigger in the submodule_update.yml workflow was fixed
in accordance with the new name policy for supported branches
(2.10 -> release/2.10, 2.11 -> release/2.11).

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

(cherry picked from commit 9518f322)

The push trigger for branches matching to the `[0-9].[0-9]+` pattern
was removed due to changing the name policy for supported branches
(2.10 -> release/2.10, 2.11 -> release/2.11).

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

(cherry picked from commit ba0a9f07)

- Rename the workflow into `publish-module-api-doc` to be consistent
  with other workflow names.
- Refactor workflow triggers.
- Run the workflow only on push to the tarantool/tarantool repository
  or on pull request if the 'notest' label is not set.
- Simplify condition for publishing generated API documentation.

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

(cherry picked from commit c9e0074d)

Follows up #8194

NO_CHANGELOG=ci
NO_DOC=ci
NO_TEST=ci

(cherry picked from commit 07d686dc)

After this patch, the workaround for LuaJIT profiling tests to
avoid runner's shutdown due to no space left on the disk is going
to be used in all relevant workflows. Previously, we applied it
only for the coverage test, but we noticed that other jobs may be
affected as well after tarantool/tarantool#8737 is merged.

Follows up tarantool/tarantool#7472

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

(cherry picked from commit b41fc204)

Corrupted snap files that are used in the test were generated manually
using a now old Tarantool version that has an outdated system schema.
In the scope of #7149 DDL was forbidden until the system schema is
upgraded. The problem is luatest tries to grant super privileges to
the guest user (which is a DDL operation) after starting a test instance
unless they are already granted. Since the snap files don't store the
required privileges, luatest fails.

To fix this issue, let's generate corrupted snap files right in the test
using error injection.

Closes #8702

NO_DOC=test
NO_CHANGELOG=test

(cherry picked from commit 67598073)

The patch fixes two mistakes that were made while writing the test.
Firstly, we should replace metrics module with an empty table before
every test case - otherwise, metrics can be collected before the test.
Secondly, all attempts to check if a required amount of metrics was
collected is pointless, even with margin - some environments are
incredibly slow. So let's check if some metrics were collected, and check
if there are not too many of them.

Also, the patch increases some timeouts to minimize the probability of
fail due to slow environment.

Follows up #8192
Closes tarantool/tarantool-qa#312

NO_CHANGELOG=test
NO_DOC=test

(cherry picked from commit d187b142)

Bump the metrics submodule to 1.0.0-2-gea83227 version.

NO_DOC=metrics submodule bump
NO_TEST=metrics submodule bump
NO_CHANGELOG=metrics submodule bump

(cherry picked from commit 2780e8e0)

Some replication tests (linearizable_test.lua and
bootstrap_strategy_test.lua) used default test-runner to test box.cfg{}
calls which are expected to fail.

Since box.cfg{} is going to be prohibited on default test runner, let's
move such test cases into properly initialized servers.

NO_DOC=test
NO_CHANGELOG=test

(cherry picked from commit 8b10902d)

Direct call and configuration of the runner instance is prohibited. Now
if you need to test something with specific configuration use a server
instance please (see luatest.Server module).

In-scope-of tarantool/luatest#245

NO_DOC=ban calling box.cfg
NO_TEST=ban calling box.cfg
NO_CHANGELOG=ban calling box.cfg

(cherry picked from commit fc3426d8)

Drop the IPROTO_FEATURE_NAMES table and use box.iproto.feature in
iproto_feature_resolve so that we don't have to update it manually every
time we add a new feature.

Follow-up #8443
Follow-up commit b3fb883b ("iproto: export IPROTO constants to Lua
automatically")

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

(cherry picked from commit 606e50c4)

There are no substantial gaps in the remaining IPROTO constant enums so
there's no need in iproto_constant struct. Instead we can generate
string arrays, as we usually do. This is more flexible because it allows
us to look up a name by code. It's also consistent with iproto_type and
iproto_key names.

The only tricky part here is the iproto_flag enum because it contains
bit masks. To generate names for the flags, we add the auxiliary enum
iproto_flag_bit that contains bit numbers.

Follow-up #8443
Follow-up commit b3fb883b ("iproto: export IPROTO constants to Lua
automatically")

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

(cherry picked from commit 75632133)

Currently, we fill iproto_type_strs only for command codes exported to
box.stat while for the rest of command codes we have a switch-case in
the iproto_type_name function. This is ugly and error-prone because we
can easily forget to update iproto_type_name when we add a new command
code. Let's generate iproto_type_strs automatically just like we
generate iproto_key_strs.

There are a few things that should be noted here:
 - We don't generate strings for IPROTO_TYPE_ERROR and IPROTO_UNKNOWN
   because the former has a big code while the latter has a negative
   code. The only place where we need the strings is exporting IPROTO
   constants to Lua so now we just export these special codes explicitly
   there.
 - We don't generate strings for IPROTO codes reserved for vinyl because
   they aren't exported to Lua and use a different naming convention.
   As before, we have a switch-case in iproto_type_name for them.
 - We remove IPROTO_RESERVED_TYPE_STAT_MAX because it isn't a reserved
   code. Instead we define IPROTO_TYPE_STAT_MAX explicitly in the
   iproto_type enum as IPROTO_ROLLBACK + 1. This allows us to remove
   the condition that skips "RESERVED" constants from the code that
   exports IPROTO constants to Lua.
 - Before this change iproto_type_strs didn't have names for OK,
   CALL_16, and NOP, because they aren't shown in box.stat. After this
   change the names are present so we have to filter out the stat items
   explicitly in the rmean_foreach callback.

Generating iproto_type_strs makes iproto_type_constants useless so we
drop it in the scope of this patch and start using iproto_type_strs
to populate box.iproto.type.

Follow-up #8443
Follow-up commit b3fb883b ("iproto: export IPROTO constants to Lua
automatically")

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

(cherry picked from commit 42dc000e)