This function returns a key_def part by a field number. However, currently
it returns NULL for parts that contain a JSON path to indexed data. Fix it.

Needed for tarantool/tarantool-ee#671

NO_DOC=bugfix
NO_CHANGELOG=not visible in CE

Closes #9657

@TarantoolBot document
Title: The `sharding.rebalancer_mode` option

The `sharding.rebalancer_mode` option can have one of three values:
`manual`, `auto` and `off`. Default value is `auto`.

If the option is set to `manual`, one of the replicasets must have the
`rebalancer` sharding role. The rebalancer will be in this replicaset.

If the option value is `auto` and there are no replicasets with the
sharding role `rebalancer`, the replicaset with rebalancer will
be selected automatically among all replicasets. If the value of the
parameter is `auto` and one of the replicasets has the sharding role
`rebalancer`, then the rebalancer will be in that replicaset.

If the option value is `off`, rebalancing will be disabled regardless of
whether a replicaset with the sharding role `rebalancer` exists or no
such replicaset exists.

This commit adds a new configuration option wal_retention_period and
function stubs for it. It's needed to avoid rebootstrap on anonymous
replicas, as Tarantool doesn't save xlog for them.

The new option takes a floating point number that sets the period
for every xlog file during which this xlog file cannot be deleted
by garbage collector. The default value is 0, which means no delay.
The option can be set dynamically.

Note:
- The delay is applied after xlog closing
- During instance restart delay becomes box.cfg.wal_retention_period -
  last modification time of xlog.
- The minimum vclock (same as xlog file name) can be found with
  box.info.gc().wal_retention_vclock.

The option value is stored and used in C code, so we define
configuration callbacks in EE: cfg_set_wal_retention_period.

Needed for tarantool/tarantool-ee#513

NO_DOC=EE
NO_CHANGELOG=EE

This commit introduces new methods for vclock library.

Sometimes we need to take into account the 0-th component of vclock,
as it's done in the following commit, that's why vclock_min/max are
added. vclockset_foreach is just a macros, which allows to iterate
over vclockset comfortably.

NO_DOC=internal
NO_CHANGELOG=internal

We've had numerous problems with transaction boundaries in replication.
They were mostly caused by various cases when either the beginning or
end of the transaction happened to be a local row. Local rows are not
replicated, so the peer saw "corrupted" transactions with either no
beginning or no end flag, even though the transaction contents were
fine.

The problem with starting a transaction with a local row was solved in
commit f41d1ddd ("wal: fix tx boundaries"), and that fix seems to
continue working fine to this day.

The problem with ending transactions with a local row was first fixed
in commit 25382617 ("replication: append NOP as the last tx row"),
however there were problems with this approach: when a user tried to
write to local spaces on a replica from a replication trigger, it made
it impossible to ever start replicating from replica back to master.

Another fix was proposed: in commit f96782b5 ("relay: send rows
transactionally") we made relay read a full transaction into memory and
then send it all at once mangling with transanction start and end flags
when necessary.

After that the NOPs were removed in commit f5e52b2c ("box: get rid
of dummy NOPs after transactions ending with local rows"), since relay
became capable of fixing transaction boundaries itself.

Turns out the assumption that relay always sees a full transaction and
may correctly set transaction boundaries is wrong: when a replica
reconnects to master we set its starting vclock[0] to the one master has
at the moment of reconnect, so when recovery reads local rows with lsns
less than vclock[0] it silently skips them without showing them to
relay. When such skipped rows contain the is_commit flag for a currently
sent transaction we get the same problem as described before.

Let's make recovery track whether it has pushed any transaction rows to
relay or not, and if yes, recover rows with is_commit flag regardless of
whether the rows were already applied. To prevent recovering the same
data twice, recovery replaces such row contents with NOPs. Basically the
row is "recovered" only for the sake of showing its is_commit flag to
relay. Relay will skip the row anyway, since it remains local.

Follow-up #8958
Closes #9491

NO_DOC=bugfix

It doesn't make sense to assert that replica_id is correct in a row
after using that replica id to make some decisions based on it. Let's
switch the order of operations: first assert that replica_id is correct,
then compare row lsn with the already recovered one.

In-scope-of #9491

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Roles are now started and stopped at the "post_apply" stage rather than
at the "apply" stage. This allows require('config'):get() to correctly
return the configuration that is being applied.

Closes #9649

NO_DOC=bugfix

This commit adds a test, that covers the integration of tarantool's
config module with vshard and verifies correctness of changes in #9514
and tarantool/vshard#458. It checks, that we're able to upgrade several
vshard clusters without downtime.

NO_DOC=test
NO_CHANGELOG=test

We stop client fibers in the process of Tarantool shutdown in order to
be sure that subsystems that will be shutdown later are not being used.
See the commit bf620650 ("box: finish client fibers on shutdown").

But there is one more way for client code to be executed - in watcher
callback. So let's shutdown watcher too. After shutdown watcher API is
usable so we can next shutdown client fibers but notifications are
stopped.

Part of #8423

NO_CHANGELOG=internal
NO_DOC=internal

On Tarantool shutdown we cancel client fibers and then wait their
finishing (see the commit bf620650 ("box: finish client fibers on
shutdown"). If we allow to spawn new fibers after shutdown is started
then Tarantool shutdown can easily be hanged if spawned fiber does not
finish in time. This can be the case if client fiber ignores cancel
flag at the time of creating new fiber. I guess fiber does not need
creating another fiber to handle cancelling thus it is better to disable
it.

Part of #8423

NO_CHANGELOG=internal
NO_DOC=internal

This patch changes when extras.post_apply() is executed. Previously, it
would only run if a _post_apply() was successful and all previous reload
steps completed properly. It will now be executed, even if some reload
steps failed.

Note that if an error appears during startup, Tarantool will stop and
extras.post_apply() will not be executed.

Also note that if the startup or restart is successful, there will be no
change in behavior due to this patch.

Needed for https://github.com/tarantool/tarantool-ee/issues/643

NO_DOC=will be added in EE
NO_CHANGELOG=will be added in EE

This patch fixes the following error when building tarantool statically
with Clang compiler on a Linux system:

    error: argument unused during compilation: '-static-libstdc++'

Fixes #9646

NO_DOC=build issue
NO_TEST=build issue
NO_CHANGELOG=build issue

Currently hot standby in not cancellable. So if for example it is
executed in init script then server shutdown hang. The issue is shutdown
waits init script fiber finishing.

Part of #8423

NO_CHANGELOG=bug is not yet released
NO_DOC=bugfix

The declarative configuration has the `credentials` section that
describes users and their privileges. It is OK to have privileges for a
space/function/sequence that does not exist. Such a privilege will lead
to an alert that states that the privilege will be granted, when the
object is created.

The problem that is fixed by this commit is that such an alert was not
dropped, when the object is created and the relevant privileges are
granted.

There are several ways to solve the problem. Let's look on them.

1. When a privilege is granted, drop an alert if any.
2. After the config-database privilege synchronization, revisit alerts
   to drop all obsolete ones.
3. Drop all the alerts regarding missed privileges before the
   config-database privilege synchronization and issue actual alerts
   afterwards.

The first way is the simplest, but it doesn't cover one specific
scenario: an object rename.

Let's assume that the object T has privileges declared in the
configuration and the object doesn't exist. There is an alert regarding
it. Now, object S is renamed to T. Let's assume that S had some or all
the privileges needed for T according to the configuration.

In the given scenario, we don't need to grant some or all of the
privileges and, so, the first solution doesn't work. We don't reach the
code that grants the privileges and, so, dropping alerts at this point
has no effect.

The second and the third solutions are similar and mainly differs in how
complicated the code is. The third one is implemented here with idea of
simplifying the code.

The internal `aboard` module has the following changes.

1. The `aboard` module now ignores underscored fields of an alert on its
   serialization to allow a caller to store a machine-readable
   information in them.
2. The new method `:drop_if()` is added to perform a conditional alert
   drop.

Several unit test cases are updated, because now we always need
initialized `config._aboard` for testing of the credentials applier.

Fixes #9574

NO_DOC=bugfix

Before this patch `config:info().alerts` reports only last alert of a
certain type, while others are skipped. It is a regression from commit
fa97cc0c ("config: introduce droppable alerts").

The idea of the fix is to eliminate a key from all the alerts that are
not to be dropped later except on configuration reloading.

Fixes #9586

NO_DOC=bugfix

It encapsulates all the alerts manipulation in one place, splits it from
the main config logic and simplifies reading of the relevant code.

Several tests are updated to use the public API instead of the internal
one.

Part of #9574
Part of #9586

NO_DOC=refactoring, no user-visible changes
NO_CHANGELOG=see NO_DOC
NO_TEST=see NO_DOC

We call `gc_shutdown` after all client shutdown fibers is finished. So
shutdown is quite simple, we just need to finish checkpoint system
fibers.

There may be a checkpoint in progress during shutdown. It can by done in
`checkpoint_daemon` fiber or in client fiber. Anyway we just need to
make sure it is cancellable. In case of memtx we run checkpoint in
distinct cord but thanks to `cord_cojoin` semantics cancel is propagated
to the cord's main fiber. So let's add a yield to the `checkpoint_f` to
make it cancellable.

Now we don't need `checkpoint_cancel` at all. At the moment of
memtx_engine_shutdown (actually freeing resources) the cord is sure to
be terminated.

Part of #8423

NO_CHANGELOG=internal
NO_DOC=internal

This function is outlined from tuple_field_map_create_plain() and
tuple_format_iterator_next(). Also it will be used by the memcs engine,
which is part of Tarantool Enterprise Edition.

Needed for tarantool/tarantool-ee#628

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Also fix compilation of rope.h as C++, because now it is included into:
 1. src/box/xrow_update_field.h:88
 2. src/box/xrow_update.h:37
 3. src/box/tuple.h:1548
 4. src/box/tuple_compare.cc:33

Needed for tarantool/tarantool-ee#628

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Running of automatically generated Lua programs sometimes
failed due to errors "table index is nil" and
"table index is NaN" (3%). The error was caused by `nil` and
`NaN` values used as a table indices and key values,
but it is prohibited.

Examples of errors are the following:

```
local t = {}
t[nil] = 42 -- table index is nil.
t[0 / 0] = 42 -- table index is NaN.
local a = { [nil] = 0 } -- table index is nil.
local a = { [0 / 0] = 0 } -- table index is NaN.
```

The patch fixes serializer, index or key value is always not
a `nil` and not a `NaN`: it is either a generated value or
a default number.

NO_CHANGELOG=testing
NO_DOC=testing
NO_TEST=testing

Running of automatically generated Lua programs sometimes failed
with an error "'<name>' expected near 'and'" (0.07%).

Example of the error is the following:

```lua
local t = {}
t.or = 0  -- "'<name>' expected near 'or'"
t.nil = 0 -- "'<name>' expected near 'nil'"
t.not = 0 -- "'<name>' expected near 'not'"
```

This error was caused by reserved Lua keywords used as table indices.
The patch fixes serializer, if the automatically generated index
name is equal to the reserved keyword then postfix "_1" is added
to index name.

NO_CHANGELOG=testing
NO_DOC=testing
NO_TEST=testing

Running of automatically generated Lua programs sometimes failed
with an error "attempt to compare" (5%). The error is caused by
comparison of non-comparable Lua values: string and boolean,
number and function, boolean and number etc. In Lua 5.1 and
LuaJIT 2.1 (which follow the semantics and behaviour of Lua 5.1),
it is not possible to overload Lua metamethods __le and __lt for
these base types. Therefore, an approach with metamethods doesn't
help there.

See an LuaJIT implementation in `src/lj_meta.c:lj_meta_comp()`
and result of comparison values with different types and
metamethods `__le` and `__lt`:

```
$ cat compare.lua
local mt_table = {}
mt_table.__le = function() assert() end
mt_table.__lt = function() assert() end

local res
local str = 'str1'
debug.setmetatable(str, mt_table)

-- __le
res = str <= 10
-- __lt
res = str < 10
$ luajit compare.lua
luajit: compare.lua:10: attempt to compare string with number
stack traceback:
        compare.lua:10: in main chunk
        [C]: at 0x55908425f2e0
$ lua5.1 compare.lua
lua5.1: compare.lua:10: attempt to compare string with number
stack traceback:
        compare.lua:10: in main chunk
        [C]: ?
$ lua5.2 compare.lua
lua5.2: compare.lua:2: assertion failed!
stack traceback:
        [C]: in function 'assert'
        compare.lua:2: in function '__le'
        compare.lua:10: in main chunk
        [C]: in ?
$
```

To fix errors triggered by comparison of non-comparable values
introduced a metamethods `__le` and `__lt` and comparison operators
are wrapped by a Lua function `only_numbers_cmp` that is
implemented in `preamble.lua`. The function perform comparison
only when both values have type 'number', otherwise it returns
`false`.

NO_CHANGELOG=testing
NO_DOC=testing
NO_TEST=testing

Running of automatically generated Lua programs sometimes failed
with an errors like:

- "'for' initial value must be a" (12%)
- "'for' limit must be a" (4.5%)
- "'for' step must be a" (0.4%)

The patch fixes these errors.

NO_CHANGELOG=testing
NO_DOC=testing
NO_TEST=testing

Running of automatically generated Lua programs sometimes failed
with an error like "ambiguous syntax (function call x new statement)"
(0.3%). The patch fixes these errors.

NO_CHANGELOG=testing
NO_DOC=testing
NO_TEST=testing

Running of automatically generated Lua programs sometimes failed
with an errors like:

- "unexpected symbol near" (0.3%)
- "bad argument" (0.2%)
- "'then' expected near" (0.02%)

NO_CHANGELOG=testing
NO_DOC=testing
NO_TEST=testing

Running of automatically generated Lua programs sometimes failed due
to semantic errors like:

- "attempt to index" (16%)
- "attempt to perform arithmetic on" (12%)
- "attempt to call" (11%)
- "attempt to concatenate" (3%)
- "attempt to get length of" (1%)

Example of an error "attempt to index" is the following:

```
Name0 = 'Name';
function Name0:Name4()
end
```

These errors are happened because Lua objects have no appropriate
metamethods and therefore these operations are not applicable:

- "attempt to index": missed `__index` metamethod
- "attempt to call": missed `__call` metamethod
- "attempt to perform arithmetic on": missed metamethods that
  implements arithmetic operations
- "attempt to get length of": missed `__len` metamethod
- "attempt to concatenate": missed `__concat` metamethod

The patch defines a Lua metatables [1] and set this metatables
to a Lua values.

1. https://www.lua.org/manual/5.1/manual.html#2.8

NO_CHANGELOG=testing
NO_DOC=testing
NO_TEST=testing

Parsing of automatically generated Lua programs sometimes failed
with an error "cannot use '...' outside a vararg function near '...'".
These errors breaks about 1.4% of generated programs.

Example is the following:

```lua
counter_0 = 0
counter_1 = 0
Name0:Name0 {  };
repeat
  if counter_0 > 5 then break end
    counter_0 = counter_0 + 1
until function ()
  if counter_1 > 5 then return end
  counter_1 = counter_1 + 1
  return ..., 'Name'
end
```

Vararg expressions, denoted by three dots ('...'), can only
be used when directly inside a vararg functions, see
Lua 5.1 Reference Manual, 2.5.9 – Function Definitions [1].

The patch fixes serializer, now ellipsis is used when function has
variable number of arguments.

1. https://www.lua.org/manual/5.1/manual.html#2.5.9

NO_CHANGELOG=testing
NO_DOC=testing
NO_TEST=testing

Protocol Buffers library has a name `protobufd` when CMAKE_BUILD_TYPE is
equal to "Debug". In other cases the name is `protobuf`. The patch fixes
the library name.

Follows up commit b11072a6 ("cmake: add dependencies for LuaJIT and
SQL fuzzers").

Follows up #4823

NO_CHANGELOG=build
NO_DOC=build
NO_TEST=build

I want to receive review request notifications about the changes and let
people know who is responsible to accept them.

NO_CHANGELOG=no code changes
NO_TEST=no code changes
NO_DOC=no code changes

The test creates a space with 1 million rows and 100 columns storing
unsigned integer values (both row and column count are configurable)
and sums values over the first and the last columns.

By default the test uses the iterator C API but one may switch to the
column scanner C API and/or raw read view C API (both features are
exclusive to the Enterprise Edition).

It's also possible to specify the engine to use (default is memtx).

Closes tarantool/tarantool-ee#659

NO_DOC=perf test
NO_TEST=perf test
NO_CHANGELOG=perf test

In the process of graceful shutdown it is convenient to first finish
all client (non system) fibers. Otherwise we should be ready for any
subsystem to handle request from client fiber during or after subsystem
shutdown. This would make code more complex.

We first cancel client fibers and then wait for their finishing. The
fiber may not respond to cancel and hang which cause shutdown hang
but this is the approach we choose for iproto shutdown already.

Note that as a result of this approach application will panic if
it is shutdown during execution of initialization script (in
particular if this script is doing box.cfg).

There are changes in application/test to adopt to client fibers
shutdown:

- make code cancellable (only to pass existing tests, we did not
  investigate all the possible places that should be made such).

- make console stop sending echo to client before client fibers
  shutdown. Otherwise as console server fiber is client one we will send
  message that fiber is cancelled on shutdown which breaks a lot of
  existing tests. This approach is on par with iproto shutdown.

- some tests (7743, replication-luatest/shutdown, replication/anon,
  replication/force_recovery etc etc) test shutdown during execution of
  init script. Now panic is expected so change them accordingly.

- some tests (8530, errinj_vylog) use injection that block client
  fiber finishing. In that tests we don't need graceful shutdown so
  let's just kill tarantool instead.

- we change test in vinyl/errinj for gh-3225. We don't really need
  to check when vinyl reader is blocked as it executes small tasks
  (we assume reading syscall will not hang). Also change test for
  vinyl dump shutdown by slowing dump down instead of blocking it
  entirely. This is required to finish in time client fibers in
  the test.

- other similar changes

Also we can drop code from replication shutdown which is required to
handle client requests during/after shutdown.

Part of #8423

NO_CHANGELOG=internal
NO_DOC=internal

tarantoolctl has been removed. Systemd, sysvinit and logrotate scripts
based on it were also removed. All this functionality is covered by the
tt utility.

Closes #9443

@TarantoolBot document
Title: utils: tarantoolctl has been removed.

Tarantoolctl has been removed. You now need to use the `tt` utility.
Documentation for the `tt` utility can be found at
[tarantool.io](https://www.tarantool.io/ru/doc/latest/reference/tooling/tt_cli/)

It has been decided to have only release and debug builds for testing
on two latest macOS versions (at this momemnt, 13 and 14). Static and
LTO builds are dropped.

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

These options are needed to properly set up reconnection of etcd
watchers.

Needed for https://github.com/tarantool/tarantool-ee/issues/647

NO_DOC=will be added in EE

This patch is a follow-up for the commit
b99a3541 ("lua: fix loader behaviour in
case of syntax error").

This test uses full matching of the error message and assumes that the
temporary directory is short enough, so the full patch is less than 60
symbols (size of `lua_Debug.short_src`). But for macOS, the temporary
directory has a longer path, so the error message is shrunk and isn't
equal to the expected string.

This patch replaces full matching with matching of the last part of the
error message.

Relates to tarantool/tarantool-ee#585

NO_DOC=test fix
NO_CHANGELOG=test fix

The types added by commit 2e0f9a3c ("box: introduce fixed-size numeric
field types") can't be indexed. Fix it.

Perf results for `tuple_compare' benchmark with FIELD_TYPE_INTEGER:

NO_WRAP
$ taskset 0x2 ~/benchmark/tools/compare.py benchmarks \
  ./tuple.perftest.ref ./tuple.perftest.exp \
  --benchmark_min_warmup_time=10 \
  --benchmark_repetitions=30 \
  --benchmark_report_aggregates_only=true \
  --benchmark_filter=tuple_compare$
[...]
Comparing ./tuple.perftest.ref to ./tuple.perftest.exp
Benchmark                                    Time             CPU      Time Old      Time New       CPU Old       CPU New
-------------------------------------------------------------------------------------------------------------------------
tuple_tuple_compare_mean                  +0.0158         +0.0158            27            27            27            27
tuple_tuple_compare_median                +0.0153         +0.0152            27            27            27            27
tuple_tuple_compare_stddev                +0.0042         +0.0039             1             1             1             1
tuple_tuple_compare_cv                    -0.0114         -0.0117             0             0             0             0
NO_WRAP

Closes #9611

NO_DOC=bugfix
NO_CHANGELOG=unreleased

This patch finalizes the refactoring of Tarantool package.loaders
required for Tarantool integrity protection.

Finally, to make it possible to instrument the searchers with Tarantool
integrity protection, two steps are left.
* Implement mapping from searchers table to <package.loaders> using
  <gen_legacy_loader> helper.
* Obtain the searcher from the table directly, so the runtime
  instrumentation takes the effect.

As a result, <gen_legacy_loader> is reimplemented to use the table with
searchers as an upvalue instead of the particular searcher routine, and
<package.loaders> table is filled by the function generated with the
updated helper. Furthermore, the aforementioned table with searchers is
added to 'internal.loaders' module exports to make the instrumentation
more convenient for users.

Needed for tarantool/tarantool-ee#585

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

This patch continues the refactoring of Tarantool package.loaders
required for Tarantool integrity protection.

Though all Lua 5.1 loaders machinery will be superseded by Lua 5.4
searchers machinery, Tarantool still should follow Lua 5.1 semantics, so
the "legacy" loader is required. This patch implements the common helper
<gen_legacy_loader> to wrap all loaders and make the new machinery
conform the Lua 5.1 semantics.

Moreover, since all tweaks and helpers are implemented for searching
machinery, <chain_loaders> is not required anymore.

Needed for tarantool/tarantool-ee#585

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

This patch continues the refactoring of Tarantool package.loaders
required for Tarantool integrity protection.

Though croot loader is not the popular way to work with Lua C libraries,
some modules are organised using this mechanism, so they have to be
verified via Tarantool integrity protection system, hence, croot loader
has to be reimplemented to the new searching machinery.

The corresponding croot searcher is implemented in this patch and added
to the indexed part of the <package.searchers> table.

It's worth to mention two peculiarities related to the croot loader and
searcher:

* Since Tarantool uses the original croot loader from LuaJIT, it ignores
  all paths except ones specified in <package.cpath>. However, there
  might be some croot-like modules in the Tarantool-specific cpaths (cwd
  or application root). It looks like croot machinery is not widely
  used, if this bug bothers nobody, but it still needs to be fixed.
* A similar situation relates to the overriding mechanism that was
  recently introduced: there is no "prefixed" croot loader and all
  libraries using croot loading machinery are ignored for the overridden
  paths as a result.

Needed for tarantool/tarantool-ee#585

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

This patch continues the refactoring of Tarantool package.loaders
required for Tarantool integrity protection.

Within this patch, the mechanism for enabling/disabling loaders
introduced in the commit 87b4da31 ("lua:
enable/disable override loader") is moved to the "searching" phase of
the new loading machinery.

Needed for tarantool/tarantool-ee#585

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring