Seed corpus based on test data used in regression tests.
Dictionary was created using fuzzing test after 10^6 test executions.

Follows up #6731
Part of #8488

NO_CHANGELOG=fuzzing corpus
NO_DOC=fuzzing corpus
NO_TEST=fuzzing corpus

metrics.cfg is a module configuration entrypoint [1]. After this patch,
box.cfg{metrics = cfg} triggers metrics.cfg. Parameter is dynamic.

This patch also enables all metrics by default. To disable them, use
box.cfg{metrics = {include = 'none'}}. Since callbacks are triggered
manually (when user calls `invoke_callbacks()` or
`collect{invoke_callbacks=true}`), it should not affect
the performance.

1. https://www.tarantool.io/en/doc/latest/book/monitoring/api_reference/#lua-function.metrics.cfg

Closes #7725

@TarantoolBot document
Title: embedded metrics configuration

This patch add new box.cfg dynamic parameter: metrics. It is a table
with the same effect as in metrics.cfg. metrics.cfg has its own
documentation, see [1].

1. https://www.tarantool.io/en/doc/latest/book/monitoring/api_reference/#lua-function.metrics.cfg

tarantool/metrics [1] is a lua module (distributed as a separate rock)
for metrics aggregation and export. After this patch, it will be a part
of the tarantool binary.

1. https://github.com/tarantool/metrics

Part of #7725

@TarantoolBot document
Title: embedded metrics

Now tarantool has metrics module on its board. metrics is a lua module
previously distributed as a separate rock which is widely used by
tarantool applications. Metrics has its own documentation section:
https://www.tarantool.io/en/doc/latest/book/monitoring/ , but it doesn't
yet mention anything about the embedding.

Patch replaces "sql_value" and "struct sql_value" with "struct Mem" and
renames sqlValueFree to mem_delete to be consistent with functions names
for "struct Mem".

NO_CHANGELOG=refactoring
NO_DOC=refactoring
NO_TEST=refactoring

Function argument uses a name that is a reserved word in a C++ [1].
Patch renames it to avoid clash when building with C++ compiler.

1. https://en.cppreference.com/w/cpp/keyword

NO_CHANGELOG=internal
NO_DOC=internal
NO_TEST=internal

We need matras statistics for reporting read view memory usage, see
https://github.com/tarantool/tarantool-ee/issues/143. Let's also use
them for reporting index memory usage in box.info.memory() for
consistency and add a test.

While we are at it, drop unused memtx_index_extent_pool declaration,
which is a leftover from commit 3d138884 ("memtx: move all global
variables to engine").

NO_DOC=refactoring
NO_CHANGELOG=refactoring

Simply forward the new matras_stats argument to matras_create().
Currently, it's only used in tests.

While we are at it:
 - Drop custom alloc/free func types in favor of matras types since
   now container constructors depend on matras_stats anyway.
 - Rearrange arguments order to group allocator-related arguments
   together.
 - Drop rtree_init return value (it's always 0).

Needed for https://github.com/tarantool/tarantool-ee/issues/143

NO_DOC=refactoring
NO_CHANGELOG=refactoring

This commit pulls matras statistics.

Needed for https://github.com/tarantool/tarantool-ee/issues/143

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Bump test-run to new version with the following improvements:

- Introduce flaky tests statistics [1]

[1] tarantool/test-run@7729a4c

Part of tarantool/tarantool-qa#299

NO_DOC=testing stuff
NO_TEST=testing stuff
NO_CHANGELOG=testing stuff

There was a problem with the leader's relay continuing to ping the
remote followers even when the leader's tx thread is hung. This tricked
the followers into thinking the leader is alive and well, even though it
couldn't serve any new requests.

The problem was partially fixed by commit 56571d83 ("raft: make
followers notice leader hang"): that commit made relay thread stop
sending heartbeats in case tx thread is unresponsive.

Up to now we didn't differentiate between heartbeats and data rows: the
receipt of both was considered a sign the master is alive. So if some
replicas are not up to date with the master, they will continue thinking
it's alive until they are fully synced and notice there are no more
heartbeats from it.

In order to fix this, stop treating all data as heartbeats and start
sending heartbeats on top of an active replication stream.

Closes #7515

NO_DOC=bugfix

`box.stat.memtx` is a table that contains the 'tx' function. This is
confusing because other stat entries are callable: `box.stat.net()`,
`box.stat.vinyl()`, `box.stat.sql()`.

Let's make `box.stat.memtx` callable for consistency. The function
returns a table with the only field 'tx'. Note, we can't drop
`box.stat.memtx.tx()` without breaking backward compatibility so we now
return `box.stat.memtx().tx` when it's called.

Also, let's use `info_handler` instead of pushing statistics directly to
Lua for better encapsulation.

Needed for https://github.com/tarantool/tarantool-ee/issues/143
Closes #8448

NO_DOC=updated https://github.com/tarantool/doc/issues/2801

Corpus based on PUC Rio Lua tests imported from LuaJIT repository [1].

1. https://github.com/tarantool/luajit/tree/tarantool/test/PUC-Rio-Lua-5.1-tests

Follows up #4823

NO_CHANGELOG=corpus
NO_DOC=corpus
NO_TEST=corpus

Patch adds a LuaJIT fuzzer based on libprotobuf-mutator and LibFuzzer.
Grammar is described via messages in protobuf format, serializer is
applied to convert .proto format to string.

For displaying generated code on the screen during fuzzing set
the environment variable 'LPM_DUMP_NATIVE_INPUT'.

For displaying error messages from lua functions set
the environment variable 'LUA_FUZZER_VERBOSE'.

Note: UndefinedBehaviourSanitizer is unsupported by LuaJIT (see #8473),
so fuzzing test is disabled when CMake option ENABLE_UB_SANITIZER is
passed.

Closes #4823

NO_DOC=<fuzzing testing of LuaJIT>
NO_TEST=<fuzzing testing of LuaJIT>

Added options for fuzzing and for getting more information
on debugging.

NO_CHANGELOG=<fuzzing options>
NO_DOC=<fuzzing options>
NO_TEST=<fuzzing options>

Added Google's 'libprotobuf-mutator' and 'protobuf' libraries
for developing grammar-based LuaJIT and SQL fuzzers based on
LibFuzzer.

It is needed to build protobuf module from source because
by default, the system-installed version of protobuf is used
by libprotobuf-mutator, and this version can be too old.

Part of #4823

NO_CHANGELOG=<dependencies>
NO_DOC=<dependencies>
NO_TEST=<dependencies>

Now we use _space:max() instead of max_id for space id generation, so
it is not used anymore. The patch removes max_id from bootstrap snapshot,
adds upgrade and downgrade scripts.

Closes #5997

@TarantoolBot document
Title: Update the description of _schema
Root document: https://www.tarantool.io/en/doc/latest/reference/reference_lua/box_space/_schema/

Since tarantool 2.11.1, there is no max_id field in space _schema.

Currently, _schema.max_id is used to generate sequentially growing
space ids. The main drawback of this approach is that generated space id
can be not unique if one mixes implicit and explicit space ids. Let's use
actual maximal space id to generate a new one, and scan for free id if
overflow happened.

Closes #8036

NO_DOC=bugfix

Disable to downgrade in this case.

Closes #8457

NO_DOC=bugfix

JIT has been disabled for these 4 tests on arm64 to avoid failing due to
side-effects of constant rematerialization:
* <app-tap/datetime.test.lua>
* <box-luatest/gh_6539_log_user_space_empty_or_nil_select_test.lua>
* <box-luatest/pagination_netbox_test.lua>
* <engine-luatest/pagination_test.lua>

The problem was solved via the commit
15e62a67 ("luajit: bump new version").
So, enable JIT compilation for these tests back.

Closes #6599
Closes #7739
Closes #8011

NO_CHANGELOG=tests
NO_DOC=tests

Currently, we use space to get primary key definition for extracting
iterator position in functional index. It was not a good solution - we
are not allowed to use space in read_view pagination, because space can
be dropped. Let's use primary key definition from index_def - it was
recently added there.

Part of tarantool/tarantool-ee#285

NO_TEST=no behavior changes
NO_CHANGELOG=no behavior changes
NO_DOC=no behavior changes

When working with a non-unique secondary index, one may need a primary
key as well. We have cmp_def, which is key_def merged with pk_def, but
it's difficult to work with primary key in such form. For example, we have
to get primary key definition right from primary index of space to extract
position of tuple in functional index. Let's add primary key definition to
index_def to simplify use of primary key parts and avoid unnecessary
dependencies on space and its primary index.

Part of tarantool/tarantool-ee#285

NO_TEST=no behavior changes
NO_CHANGELOG=no behavior changes
NO_DOC=no behavior changes

Follow-up #8463

NO_DOC=style fix
NO_CHANGELOG=style fix
NO_TEST=style fix

The `lua_add_key_u64()` function pushes an `uint64_t` value using
`lua_pushinteger()`, which accepts `int64_t` argument. A value >= 2^63
will be interpreted as a negative value on all architectures we're
supporting. However, technically it is implementation defined behavior
(see n1256, 6.3.1.3.3).

It is not a problem, in fact, because the function is used only to
report `http_client:stat()` statistics and because values beyond 2^63-1
are unreachable in practice.

OTOH, it is easy to eliminate the undefined behavior by replacing
`lua_pushinteger()` with our own helper function, which accepts
`uint64_t`: `luaL_pushuint64()`.

The values above 10^14 - 1 are now pushed as `cdata<uint64_t>`. Lower
values are pushed as `number` just like before the commit.

Reported-in: https://github.com/tarantool/security/issues/103

NO_DOC=The type of values in the statistics is not specified explicitly
       in the documentation (not obligated to be `number`) and it is
       quite common for Tarantool to return a value of `cdata<int64_t>`
       or `cdata<uint64_t>` type for an integer with a large absolute
       value.
NO_CHANGELOG=see NO_DOC
NO_TEST=It is hard to reach so large values externally (send 2^63
        requests) and it doesn't look worthful to introduce an error
        injection/a internal API to test it. `luaL_pushuint64()` is
        covered by the module API test.

Bump test-run to new version with the following improvements:

- Bump luatest to 0.5.7-29-geef05dd [1]

[1] tarantool/test-run@cc3c38e

NO_DOC=testing stuff
NO_TEST=testing stuff
NO_CHANGELOG=testing stuff

The index directory is created on demand since commit c00ba8e7
("xlog: make log directory if needed") and removed when it becomes
empty. There's no need to create it when an index is created anymore.

Follow-up #8441

NO_DOC=bugfix

When vinyl space is dropped, its files are left on the file system
until GC removes them. At the moment GC removes only run files,
but not the root directory. These empty directories are never
removed and occupy 4KB on ext-family file systems each. In a case
of many dropped vinyl spaces it can become a serious disk space
and inode leak. Current commit makes gc always remove root directory
if there are no runs in it.

Closes #8441

NO_DOC=bugfix

`func_opts_reg` definition misses a `OPT_END` termintator item. This
leads to UB on iterating it. Particularly when `func_opts_reg` is used
as argument to `opts_decode` in `func_def_new_from_tuple`.

Closes #8463

NO_DOC=bugfix

The function is unused and still triggers some static analysis
warnings. Let's drop it.

itoa() became unused with removal of fdprintf(), so let's drop it as
well.

Closes tarantool/security#113

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Casting a uint64_t greater than INT64_MAX to int64_t is
implementation-defined behaviour, according to the C standard. Let's
avoid that.

In both cases fixed `len` is uint32_t and `ibuf_used(in)` returns a
size_t (aka uint64_t on every platform that we care about).
Hence the result of the subtraction is uint64_t and better use
it directly. Besides, `coio_breadn_timeout` also takes a size_t.

While I'm at it, let's actually change `len` to be uint64_t:
`mp_decode_uint()` returns that anyway.

Closes tarantool/security#108
Closes tarantool/security#109

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Due to a typo some big numbers were coded as MP_(U)INT.

Since msgpackffi is used in selectffi, which is used for memtx,
that could lead to strange select results with big number keys.

Closes #6119

NO_DOC=bugfix

This patch fixes incorrect conversion of an integer greater than
INT64_MAX or less than 0 to decimal during SQL arithmetic operations.

Closes #8460

NO_DOC=bugfix

This commit fixes the following assertion failure that happens on
a client in case a remote schema contains an unknown field type:

  src/box/lua/misc.cc:395: int lbox_tuple_format_new(lua_State*):
  Assertion `fields[i].type != field_type_MAX' failed.

To fix the bug we remove the code that tries to set field types from
box.internal.new_tuple_format. Actually, the format is used solely for
providing field names so types are ignored anyway.

Closes #4632

NO_DOC=bug fix

Prior to this patch, the return value of region_alloc() in
lbox_tuple_format_new() was not checked. This patch fixes this by
replacing region_alloc() with xregion_alloc(). Also, this patch
replaces region_alloc_array() to xregion_alloc_array() in the same
function.

Closes tarantool/security#116

NO_DOC=bugfix
NO_TEST=hard to reproduce the bug
NO_CHANGELOG=bugfix for unlikely bug

The `merger.new()` call has the following code in the
`luaT_merger_new_parse_sources()` function:

 | uint32_t source_count = lua_objlen(L, idx);
 | for (uint32_t i = 0; i < source_count; ++i) {
 |     <...>
 | }
 | lua_pop(L, source_count);

It is possible that zero amount of sources are passed:

 | merger.new(kd, {})

In this case the `source_count` variable is zero.

`lua_pop()` is a macro defined this way:

 | #define lua_pop(L,n)		lua_settop(L, -(n)-1)

It means that `n` in the `-(n)-1` expression is an unsigned 32 bit zero.
Unsigned overflow is okay: it has defined behavior by the C standard and
has the result 2^32-1 in the given case.

The `lua_settop()` function is defined as follows:

 | LUA_API void  (lua_settop) (lua_State *L, int idx);

We pass the `-(n)-1` value as `int idx` argument to `lua_settop()`. The
value has uint32_t type and it is out of the `int` range ([-2^31,
2^31]). Casting it to `int` has implementation defined behavior
according to the standard (n1256,
6.3.1.3.3).

In practice, we're building Tarantool only for architectures with two's
complement integers. The result of the cast is -1 and everything works
as expected: the stack top remains unchanged.

However, it is easy to eliminate the signed integer overflow, so it is
worthful to do. We can just save the stack top value and use
`lua_settop()` to restore it, which is quite common idiom.

The problem can be found by clang's undefined behavior sanitizer.

Apply the following patch:

NO_WRAP
 | --- a/cmake/compiler.cmake
 | +++ b/cmake/compiler.cmake
 | @@ -238,6 +238,7 @@ macro(enable_tnt_compile_flags)
 |                  alignment bool bounds builtin enum float-cast-overflow
 |                  float-divide-by-zero function integer-divide-by-zero return
 |                  shift unreachable vla-bound
 | +                implicit-integer-sign-change
 |              )
 |
 |              # Exclude "object-size".
 | @@ -272,7 +273,7 @@ macro(enable_tnt_compile_flags)
 |              # the typeof(*obj) when obj is NULL, even though there is nothing
 |              # related to return.
 |
 | -            set(SANITIZE_FLAGS "-fsanitize=${SANITIZE_FLAGS} -fno-sanitize-recover=${SANITIZE_FLAGS}")
 | +            set(SANITIZE_FLAGS "-fsanitize=${SANITIZE_FLAGS}")
 |
 |              add_compile_flags("C;CXX" "${SANITIZE_FLAGS}")
 |          endif()
NO_WRAP

Build Tarantool with the sanitizer:

 | CC=clang-15 CXX=clang++-15 cmake . \
 |     -DCMAKE_BUILD_TYPE=Debug       \
 |     -DENABLE_BACKTRACE=ON          \
 |     -DENABLE_DIST=ON               \
 |     -DENABLE_FEEDBACK_DAEMON=OFF   \
 |     -DENABLE_BUNDLED_LIBCURL=OFF   \
 |     -DENABLE_BUNDLED_LIBUNWIND=OFF \
 |     -DENABLE_UB_SANITIZER=ON && make -j

Run the interactive console and create a merger with zero sources:

 | tarantool> key_def = require('key_def')
 | tarantool> merger = require('merger')
 | tarantool> kd = key_def.new({{field = 1, type = 'number'}})
 | tarantool> m = merger.new(kd, {})

Observe the 2^32-1 cast to 32 bit signed integer:

 | <...>/src/box/lua/merger.c:334:2: runtime error: implicit conversion
 |     from type 'unsigned int' of value 4294967295 (32-bit, unsigned)
 |     to type 'int' changed the value to -1 (32-bit, signed)
 | SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior
 |     <...>/src/box/lua/merger.c:334:2 in

The commit eliminates this report from the clang's sanitizer.

I've added a test case, which goes over the relevant code path. It
succeeds as before the commit as well as after it. If we'll enable a
relevant dynamic analysis in a future (such as clang's
`-fsanitize=implicit-integer-sign-change`), the test case may reveal
problems on the given code path.

Reported-in: https://github.com/tarantool/security/issues/103

NO_DOC=no user-visible behavior changes
NO_CHANGELOG=no user-visible behavior changes

If a read view is created while space upgrade is in progress, tuples
fetched from the read view may be either upgraded or not. We need to
be able to differentiate those tuples so that we can use the appropriate
tuple format for them. To achieve that this commit adds the following
function stubs:

 - memtx_space_upgrade_track_tuple and memtx_space_upgrade_untrack_tuple
   will be used to maintain a set of all upgraded tuples.
 - memtx_read_view_tuple_needs_upgrade will do a lookup in the set of
   all upgraded tuples to check if a tuple needs upgrade.

The stubs will be implemented in the EE repository.

Note that we have to call memtx_space_upgrade_untrack_tuple from
memtx_engine_rollback_statement. The problem is that the space may be
deleted while a transaction is inprogress, in which case we must not
access space->upgrade in memtx_engine_rollback_statement. Fortunately,
we call memtx_tx_on_space_delete when a memtx space is altered to
rollback memtx transactions. So to handle this situation we set
txn_stmt->engine_savepoint to NULL from memtx_tx_history_remove_stmt
called from memtx_tx_on_space_delete. This makes the rollback function
return early.

Needed for tarantool/tarantool-ee#236

NO_DOC=ee
NO_TEST=ee
NO_CHANGELOG=ee

This reverts commit e771d06d.

Not needed anymore.

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Currently, we pass only disable_decompression flag, but to handle tuples
in case the read view was created while space upgrade was in progress,
we'll need extra information stored in the read view struct. Let's pass
index_read_view to memtx_prepeare_read_view_tuple instead of the flag.
To do that we need to store the flag in struct read_view.

Needed for tarantool/tarantool-ee#236

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

This commit introduces read_view_tuple struct which is used for
returning raw tuple data and size from a read view. In the following
commits we'll add a flag indicating if the tuple was upgraded or not to
this struct (relevant if the read view was created while space upgrade
was running).

Needed for tarantool/tarantool-ee#236

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

This commit adds the new callback space_vtab::prepare_upgrade. It is
invoked after preparing to alter a space and passed the old and new
space objects, like prepare_alter. The generic callback implementation
raises an error saying that the engine doesn't support space upgrade.
The memtx implementation raises an error saying that space upgrade isn't
available in the community edition. It'll be overridden in the Tarantool
EE repository.

The new callback replaces calls to space_upgrade_check_alter and
space_upgrade_new. Their job is now supposed to be done by the callback
implementation in Tarantool EE. This change makes it easier to extend
space upgrade implementation in Tarantool EE. In particular, we can now
make it engine-dependent, which is required to fix the issue with tuple
formats when a read view is created while space upgrade is in progress.

Needed for tarantool/tarantool-ee#236

NO_DOC=refactoring
NO_CHANGELOG=refactoring

```
<...>/src/box/sql/vdbe.c:378:11: error: variable 'nVmStep' set but not
    used [-Werror,-Wunused-but-set-variable]
        unsigned nVmStep = 0;      /* Number of virtual machine steps */
                 ^
```

The usage of the variable was removed in commit dbad19ef ("sql: drop
unused functions").

See also #8110.

NO_DOC=no user visible behavior changes
NO_TEST=see NO_DOC
NO_CHANGELOG=dbad19ef is not released yet