NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

This patch fixes three issues:
1. It changes the condition for workflows so they can be run not
only from the Tarantool repository but from any repository in the
Tarantool organization.

2. Reusable workflows substitute the `${{ github.workflow }}`
context variable with the name of their top-level workflow. This
behavior causes concurrency group clashes when several reusable
workflows are called from a single top-level workflow. This patch
adds an additional constant part to the concurrency group pattern
to solve the issue.

3. The checkout actions use the reference from the repository in
which the top-level workflow is located instead of the one where
the reusable workflow is located. This patch solves the issue by
passing the reference explicitly.

NO_DOC=CI
NO_TEST=CI
NO_CHANGELOG=CI

Some workflows are not relevant for integration testing.
This patch disables them.

NO_DOC=CI
NO_TEST=CI
NO_CHANGELOG=CI

* test: fix initialization in lj-549-lua-load.test.c
* codehealth: add `nd` to the codespell ignore list
* LJ_GC64: Always snapshot functions for non-base frames.
* Avoid assertion in case of stack overflow from stitched trace.
* Fix recording of __concat metamethod.
* Avoid out-of-range number of results when compiling select(k, ...).
* Consider slots used by upvalues in use-def analysis.
* Only emit proper parent references in snapshot replay.
* Optimize table.new() with constant args to (sinkable) IR_TNEW.
* Followup fix for embedded bytecode loader.
* Fix embedded bytecode loader.
* LJ_GC64: Fix HREFK optimization.
* Fix unsinking of IR_FSTORE for NULL metatable.
* Fix zero stripping in %g number formatting.
* Follow-up fix for stack overflow handling cleanup.
* Cleanup stack overflow handling.
* Improve error reporting on stack overflow.
* sysprof: disable runtime host symtab updates
* codehealth: fix the typo
* Simplify handling of instable types in TNEW/TDUP load forwarding.
* Respect jit.off() on pending trace exit.
* Limit exponent range in number parsing.
* ARM64: Allow building with unwinding disabled.
* Emit sunk IR_NEWREF only once per key on snapshot replay.

Closes #7937
Closes #8140
Part of #9145
Part of #9595

NO_DOC=LuaJIT submodule bump
NO_TEST=LuaJIT submodule bump

Closes #8632

NO_DOC=gdb extension
NO_CHANGELOG=gdb extension
NO_TEST=gdb extension

In Linux systems based on Debian, libraries are installed in paths
depending on the architecture. For example, /usr/lib/x86_64-linux-gnu/,
/usr/lib/aarch64-linux-gnu/. Some packages may be installed in these
paths, but Tarantool does not look for libraries installed in these
paths. This patch solves the problem.

Also remove redundant OS depended `if` branches.

Fix #9580

NO_DOC=bugfix
NO_TEST=bugfix

Bump test-run to new version with the following improvements:

- Bump luatest to 1.0.0-5-gf31fe34 [1]
- get_iproto_port: remove duplicates [2]
- requirements: bump gevent to 22.10.2 [3]
- Fix decoding error when reading server's log file [4]

[1] tarantool/test-run@bfcc9e8
[2] tarantool/test-run@da98d7f
[3] tarantool/test-run@bc1c473
[4] tarantool/test-run@434cbec

NO_DOC=test
NO_TEST=test
NO_CHANGELOG=test

Switch the legacy `box.iproto.override()' interface to the newly introduced
event triggers. This change is mostly not user-visible, except:
 - Now it can be called before `box.cfg{}';
 - Now request type can be set as a string;
 - Some changes in error messages;
 - The "overriding does not support ... request type" error is logged,
   rather than raised;
 - The internal trigger is visible via the `trigger' module.

If some request type is overridden by both interfaces (legacy
`box.iproto.override()' and new `trigger.set()'), the order of invocation
of the handlers is unspecified.

Closes #8138

NO_DOC=internal

This patch allows to override IPROTO request handlers by setting triggers
on the corresponding events after the initial `box.cfg{}' call.

Part of #8138

@TarantoolBot document
Title: Document iproto override using event triggers
Product: Tarantool
Since: 3.1
Root document: New page - https://www.tarantool.io/en/doc/latest/reference/reference_lua/trigger/

Since Tarantool 3.1 there are 2 ways to override iproto request handlers:

 1. Using `box.iproto.override()`, introduced in Tarantool 2.11:
    https://www.tarantool.io/en/doc/latest/reference/reference_lua/box_iproto/override/



 2. Using universal trigger registry: tarantool/doc#3988

To override an iproto request handler for the given request type, one can
set a trigger (or multiple triggers) on the corresponding event.

There are 2 types of iproto-overriding events:
 1. set by request type id, e.g.:
     - box.iproto.override[1]
     - box.iproto.override[-1]
 2. set by request type name (the name must be in the lowercase), e.g.:
     - box.iproto.override.select
     - box.iproto.override.unknown

Override-by-id allows to set a handler for a particular request type, that
is not known by the given version of Tarantool. This is not possible with
override-by-name, where a type name must be known by Tarantool. Also there
are a special type name "unknown" and a type id box.iproto.type.UNKNOWN
(== -1) that allow to set a single handler for all unknown request types.

Multiple triggers can be associated with a single event. The triggers are
called in reverse order of their installation, however triggers set by id
are called before triggers set by name.

If a trigger returns `false`, the next trigger in the list is called, or a
system handler if there are no more triggers. If a trigger returns `true`,
no more triggers or system handlers are called.

If some request type is overridden by both interfaces (legacy
`box.iproto.override()' and new `trigger.set()'), the order of invocation
of those handlers is unspecified.

Co-authored-by: Andrey Saranchin <Andrey22102001@gmail.com>

This patch allows to override IPROTO request handlers by setting triggers
on the corresponding events before the initial `box.cfg{}' call.
For triggers that are set after the first `box.cfg{}', see next commit.

There are 2 types of iproto-overriding events:
 1. set by request type id, e.g.:
     - box.iproto.override[1]
     - box.iproto.override[-1]
 2. set by request type name (the name must be in the lowercase), e.g.:
     - box.iproto.override.select
     - box.iproto.override.unknown

Override-by-id allows to set a handler for a particular request type, that
is not known by the given version of Tarantool. This is not possible with
override-by-name, where a type name must be known by Tarantool. Also there
are a special type name "unknown" and a type id box.iproto.type.UNKNOWN
(== -1) that allow to set a single handler for all unknown request types.

Multiple triggers can be associated with a single event. The triggers are
called in reverse order of their installation, however triggers set by id
are called before triggers set by name.

If a trigger returns `false`, the next trigger in the list is called, or a
system handler if there are no more triggers. If a trigger returns `true`,
no more triggers or system handlers are called.

Part of #8138

NO_DOC=see next commit
NO_CHANGELOG=see next commit

Co-authored-by: Andrey Saranchin <Andrey22102001@gmail.com>

So far only one type of a handler can be set for a particular request type,
but the following commit will bring two more handlers. This patch prepares
`struct iproto_req_handler` for this extension to simplify next commits.

Part of #8138

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

This patch adds `iproto_key_translation' to the MsgPack decoding context,
that is used to decode xrow header and body in the space recovery triggers.

NO_DOC=Minor improvement
NO_CHANGELOG=Minor improvement

It works like func_adapter_push_msgpack, but with an additional MsgPack
decoding context. The context is required to support translation of
first-level `MP_MAP` keys.

Needed for #8138

NO_DOC=internal
NO_CHANGELOG=internal

Since we are going to move iproto override mechanism to the trigger
registry, we need to allow to set core triggers to on change event
because we need to notify iproto threads when a handler is overridden.
Let's add a new method that accepts a core trigger and sets it to an
internal trigger list.

Needed for #8138

NO_DOC=internal
NO_CHANGELOG=internal

Test box-luatest/iproto_request_handlers_overriding_test.lua has two
test cases with the same name. That is why a test case that was declared
first is replaced by the second one and that's why it is not launched.
The commit renames one of the cases.

NO_CHANGELOG=test
NO_DOC=test

This function returns a key_def part by a field number. However, currently
it returns NULL for parts that contain a JSON path to indexed data. Fix it.

Needed for tarantool/tarantool-ee#671

NO_DOC=bugfix
NO_CHANGELOG=not visible in CE

Closes #9657

@TarantoolBot document
Title: The `sharding.rebalancer_mode` option

The `sharding.rebalancer_mode` option can have one of three values:
`manual`, `auto` and `off`. Default value is `auto`.

If the option is set to `manual`, one of the replicasets must have the
`rebalancer` sharding role. The rebalancer will be in this replicaset.

If the option value is `auto` and there are no replicasets with the
sharding role `rebalancer`, the replicaset with rebalancer will
be selected automatically among all replicasets. If the value of the
parameter is `auto` and one of the replicasets has the sharding role
`rebalancer`, then the rebalancer will be in that replicaset.

If the option value is `off`, rebalancing will be disabled regardless of
whether a replicaset with the sharding role `rebalancer` exists or no
such replicaset exists.

This commit adds a new configuration option wal_retention_period and
function stubs for it. It's needed to avoid rebootstrap on anonymous
replicas, as Tarantool doesn't save xlog for them.

The new option takes a floating point number that sets the period
for every xlog file during which this xlog file cannot be deleted
by garbage collector. The default value is 0, which means no delay.
The option can be set dynamically.

Note:
- The delay is applied after xlog closing
- During instance restart delay becomes box.cfg.wal_retention_period -
  last modification time of xlog.
- The minimum vclock (same as xlog file name) can be found with
  box.info.gc().wal_retention_vclock.

The option value is stored and used in C code, so we define
configuration callbacks in EE: cfg_set_wal_retention_period.

Needed for tarantool/tarantool-ee#513

NO_DOC=EE
NO_CHANGELOG=EE

This commit introduces new methods for vclock library.

Sometimes we need to take into account the 0-th component of vclock,
as it's done in the following commit, that's why vclock_min/max are
added. vclockset_foreach is just a macros, which allows to iterate
over vclockset comfortably.

NO_DOC=internal
NO_CHANGELOG=internal

We've had numerous problems with transaction boundaries in replication.
They were mostly caused by various cases when either the beginning or
end of the transaction happened to be a local row. Local rows are not
replicated, so the peer saw "corrupted" transactions with either no
beginning or no end flag, even though the transaction contents were
fine.

The problem with starting a transaction with a local row was solved in
commit f41d1ddd ("wal: fix tx boundaries"), and that fix seems to
continue working fine to this day.

The problem with ending transactions with a local row was first fixed
in commit 25382617 ("replication: append NOP as the last tx row"),
however there were problems with this approach: when a user tried to
write to local spaces on a replica from a replication trigger, it made
it impossible to ever start replicating from replica back to master.

Another fix was proposed: in commit f96782b5 ("relay: send rows
transactionally") we made relay read a full transaction into memory and
then send it all at once mangling with transanction start and end flags
when necessary.

After that the NOPs were removed in commit f5e52b2c ("box: get rid
of dummy NOPs after transactions ending with local rows"), since relay
became capable of fixing transaction boundaries itself.

Turns out the assumption that relay always sees a full transaction and
may correctly set transaction boundaries is wrong: when a replica
reconnects to master we set its starting vclock[0] to the one master has
at the moment of reconnect, so when recovery reads local rows with lsns
less than vclock[0] it silently skips them without showing them to
relay. When such skipped rows contain the is_commit flag for a currently
sent transaction we get the same problem as described before.

Let's make recovery track whether it has pushed any transaction rows to
relay or not, and if yes, recover rows with is_commit flag regardless of
whether the rows were already applied. To prevent recovering the same
data twice, recovery replaces such row contents with NOPs. Basically the
row is "recovered" only for the sake of showing its is_commit flag to
relay. Relay will skip the row anyway, since it remains local.

Follow-up #8958
Closes #9491

NO_DOC=bugfix

It doesn't make sense to assert that replica_id is correct in a row
after using that replica id to make some decisions based on it. Let's
switch the order of operations: first assert that replica_id is correct,
then compare row lsn with the already recovered one.

In-scope-of #9491

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Roles are now started and stopped at the "post_apply" stage rather than
at the "apply" stage. This allows require('config'):get() to correctly
return the configuration that is being applied.

Closes #9649

NO_DOC=bugfix

This commit adds a test, that covers the integration of tarantool's
config module with vshard and verifies correctness of changes in #9514
and tarantool/vshard#458. It checks, that we're able to upgrade several
vshard clusters without downtime.

NO_DOC=test
NO_CHANGELOG=test

We stop client fibers in the process of Tarantool shutdown in order to
be sure that subsystems that will be shutdown later are not being used.
See the commit bf620650 ("box: finish client fibers on shutdown").

But there is one more way for client code to be executed - in watcher
callback. So let's shutdown watcher too. After shutdown watcher API is
usable so we can next shutdown client fibers but notifications are
stopped.

Part of #8423

NO_CHANGELOG=internal
NO_DOC=internal

On Tarantool shutdown we cancel client fibers and then wait their
finishing (see the commit bf620650 ("box: finish client fibers on
shutdown"). If we allow to spawn new fibers after shutdown is started
then Tarantool shutdown can easily be hanged if spawned fiber does not
finish in time. This can be the case if client fiber ignores cancel
flag at the time of creating new fiber. I guess fiber does not need
creating another fiber to handle cancelling thus it is better to disable
it.

Part of #8423

NO_CHANGELOG=internal
NO_DOC=internal

This patch changes when extras.post_apply() is executed. Previously, it
would only run if a _post_apply() was successful and all previous reload
steps completed properly. It will now be executed, even if some reload
steps failed.

Note that if an error appears during startup, Tarantool will stop and
extras.post_apply() will not be executed.

Also note that if the startup or restart is successful, there will be no
change in behavior due to this patch.

Needed for https://github.com/tarantool/tarantool-ee/issues/643

NO_DOC=will be added in EE
NO_CHANGELOG=will be added in EE

This patch fixes the following error when building tarantool statically
with Clang compiler on a Linux system:

    error: argument unused during compilation: '-static-libstdc++'

Fixes #9646

NO_DOC=build issue
NO_TEST=build issue
NO_CHANGELOG=build issue

Currently hot standby in not cancellable. So if for example it is
executed in init script then server shutdown hang. The issue is shutdown
waits init script fiber finishing.

Part of #8423

NO_CHANGELOG=bug is not yet released
NO_DOC=bugfix

The declarative configuration has the `credentials` section that
describes users and their privileges. It is OK to have privileges for a
space/function/sequence that does not exist. Such a privilege will lead
to an alert that states that the privilege will be granted, when the
object is created.

The problem that is fixed by this commit is that such an alert was not
dropped, when the object is created and the relevant privileges are
granted.

There are several ways to solve the problem. Let's look on them.

1. When a privilege is granted, drop an alert if any.
2. After the config-database privilege synchronization, revisit alerts
   to drop all obsolete ones.
3. Drop all the alerts regarding missed privileges before the
   config-database privilege synchronization and issue actual alerts
   afterwards.

The first way is the simplest, but it doesn't cover one specific
scenario: an object rename.

Let's assume that the object T has privileges declared in the
configuration and the object doesn't exist. There is an alert regarding
it. Now, object S is renamed to T. Let's assume that S had some or all
the privileges needed for T according to the configuration.

In the given scenario, we don't need to grant some or all of the
privileges and, so, the first solution doesn't work. We don't reach the
code that grants the privileges and, so, dropping alerts at this point
has no effect.

The second and the third solutions are similar and mainly differs in how
complicated the code is. The third one is implemented here with idea of
simplifying the code.

The internal `aboard` module has the following changes.

1. The `aboard` module now ignores underscored fields of an alert on its
   serialization to allow a caller to store a machine-readable
   information in them.
2. The new method `:drop_if()` is added to perform a conditional alert
   drop.

Several unit test cases are updated, because now we always need
initialized `config._aboard` for testing of the credentials applier.

Fixes #9574

NO_DOC=bugfix

Before this patch `config:info().alerts` reports only last alert of a
certain type, while others are skipped. It is a regression from commit
fa97cc0c ("config: introduce droppable alerts").

The idea of the fix is to eliminate a key from all the alerts that are
not to be dropped later except on configuration reloading.

Fixes #9586

NO_DOC=bugfix

It encapsulates all the alerts manipulation in one place, splits it from
the main config logic and simplifies reading of the relevant code.

Several tests are updated to use the public API instead of the internal
one.

Part of #9574
Part of #9586

NO_DOC=refactoring, no user-visible changes
NO_CHANGELOG=see NO_DOC
NO_TEST=see NO_DOC

We call `gc_shutdown` after all client shutdown fibers is finished. So
shutdown is quite simple, we just need to finish checkpoint system
fibers.

There may be a checkpoint in progress during shutdown. It can by done in
`checkpoint_daemon` fiber or in client fiber. Anyway we just need to
make sure it is cancellable. In case of memtx we run checkpoint in
distinct cord but thanks to `cord_cojoin` semantics cancel is propagated
to the cord's main fiber. So let's add a yield to the `checkpoint_f` to
make it cancellable.

Now we don't need `checkpoint_cancel` at all. At the moment of
memtx_engine_shutdown (actually freeing resources) the cord is sure to
be terminated.

Part of #8423

NO_CHANGELOG=internal
NO_DOC=internal

This function is outlined from tuple_field_map_create_plain() and
tuple_format_iterator_next(). Also it will be used by the memcs engine,
which is part of Tarantool Enterprise Edition.

Needed for tarantool/tarantool-ee#628

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Also fix compilation of rope.h as C++, because now it is included into:
 1. src/box/xrow_update_field.h:88
 2. src/box/xrow_update.h:37
 3. src/box/tuple.h:1548
 4. src/box/tuple_compare.cc:33

Needed for tarantool/tarantool-ee#628

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Running of automatically generated Lua programs sometimes
failed due to errors "table index is nil" and
"table index is NaN" (3%). The error was caused by `nil` and
`NaN` values used as a table indices and key values,
but it is prohibited.

Examples of errors are the following:

```
local t = {}
t[nil] = 42 -- table index is nil.
t[0 / 0] = 42 -- table index is NaN.
local a = { [nil] = 0 } -- table index is nil.
local a = { [0 / 0] = 0 } -- table index is NaN.
```

The patch fixes serializer, index or key value is always not
a `nil` and not a `NaN`: it is either a generated value or
a default number.

NO_CHANGELOG=testing
NO_DOC=testing
NO_TEST=testing

Running of automatically generated Lua programs sometimes failed
with an error "'<name>' expected near 'and'" (0.07%).

Example of the error is the following:

```lua
local t = {}
t.or = 0  -- "'<name>' expected near 'or'"
t.nil = 0 -- "'<name>' expected near 'nil'"
t.not = 0 -- "'<name>' expected near 'not'"
```

This error was caused by reserved Lua keywords used as table indices.
The patch fixes serializer, if the automatically generated index
name is equal to the reserved keyword then postfix "_1" is added
to index name.

NO_CHANGELOG=testing
NO_DOC=testing
NO_TEST=testing

Running of automatically generated Lua programs sometimes failed
with an error "attempt to compare" (5%). The error is caused by
comparison of non-comparable Lua values: string and boolean,
number and function, boolean and number etc. In Lua 5.1 and
LuaJIT 2.1 (which follow the semantics and behaviour of Lua 5.1),
it is not possible to overload Lua metamethods __le and __lt for
these base types. Therefore, an approach with metamethods doesn't
help there.

See an LuaJIT implementation in `src/lj_meta.c:lj_meta_comp()`
and result of comparison values with different types and
metamethods `__le` and `__lt`:

```
$ cat compare.lua
local mt_table = {}
mt_table.__le = function() assert() end
mt_table.__lt = function() assert() end

local res
local str = 'str1'
debug.setmetatable(str, mt_table)

-- __le
res = str <= 10
-- __lt
res = str < 10
$ luajit compare.lua
luajit: compare.lua:10: attempt to compare string with number
stack traceback:
        compare.lua:10: in main chunk
        [C]: at 0x55908425f2e0
$ lua5.1 compare.lua
lua5.1: compare.lua:10: attempt to compare string with number
stack traceback:
        compare.lua:10: in main chunk
        [C]: ?
$ lua5.2 compare.lua
lua5.2: compare.lua:2: assertion failed!
stack traceback:
        [C]: in function 'assert'
        compare.lua:2: in function '__le'
        compare.lua:10: in main chunk
        [C]: in ?
$
```

To fix errors triggered by comparison of non-comparable values
introduced a metamethods `__le` and `__lt` and comparison operators
are wrapped by a Lua function `only_numbers_cmp` that is
implemented in `preamble.lua`. The function perform comparison
only when both values have type 'number', otherwise it returns
`false`.

NO_CHANGELOG=testing
NO_DOC=testing
NO_TEST=testing

Running of automatically generated Lua programs sometimes failed
with an errors like:

- "'for' initial value must be a" (12%)
- "'for' limit must be a" (4.5%)
- "'for' step must be a" (0.4%)

The patch fixes these errors.

NO_CHANGELOG=testing
NO_DOC=testing
NO_TEST=testing

Running of automatically generated Lua programs sometimes failed
with an error like "ambiguous syntax (function call x new statement)"
(0.3%). The patch fixes these errors.

NO_CHANGELOG=testing
NO_DOC=testing
NO_TEST=testing

Running of automatically generated Lua programs sometimes failed
with an errors like:

- "unexpected symbol near" (0.3%)
- "bad argument" (0.2%)
- "'then' expected near" (0.02%)

NO_CHANGELOG=testing
NO_DOC=testing
NO_TEST=testing