Fuzzing test for LuaJIT generates random Lua programs and executes them.
We want to build a fuzzing test that will produce Lua programs that will
not contain semantic errors and will trigger as much as possible
components in LuaJIT.

This proposed patch introduces metrics that gathered after running the
test. LuaJIT metrics gathered using LuaJIT getmetrics module [1]. All
gathered metrics test will output after running with a finite number of
runs or finite duration of time (options `-runs` and `-max_total_time`)
or after sending SIGUSR1 to a test process.

```
$ ./build/test/fuzz/luaL_loadbuffer/luaL_loadbuffer_fuzzer -runs=1000

<snipped>

Done 1000 runs in 1 second(s)
Total number of samples: 1000
Total number of samples with errors: 438 (43%)
Total number of samples with recorded traces: 87 (8%)
Total number of samples with snap restores: 30 (3%)
Total number of samples with abort traces: 55 (5%)
```

1. https://www.tarantool.io/en/doc/latest/reference/tooling/luajit_getmetrics/#getmetrics-c-api

NO_CHANGELOG=testing
NO_DOC=testing

The function can't be called on an unconfigured instance because it
needs IPROTO threads up and running. Let's raise an error to avoid
a crash.

Since we have two other places where we need to raise the same error
(box.session.su and box.__index), let's introduce the new code
ER_UNCONFIGURED for this error.

Closes #8975

NO_DOC=bug fix

The `popen_new()` function may be called with inherit_fds = NULL,
nr_inherit_fds = 0. The lua/popen.c code actually does this.

It leads to the following memcpy() call.

```
memcpy(dst, NULL, 0);
```

According to the C11 standard (n1256, 7.21.1), the pointer argument
should have a valid value, which means pointing to some area in the
program's address space, not NULL. The standard doesn't make an
exception for a zero size array.

Personally I doubt that any memcpy() implementation attempts to
dereference the source pointer in case of a zero size, but it is my
assumption, while the standard is the standard.

The problem is found by Coverity.

Follows up #8926

NO_DOC=it is a bug
NO_CHANGELOG=this code is not released yet
NO_TEST=verified by existing test cases, which call popen.new() without
        the inherit_fds option

Closes #8802

@TarantoolBot document
Title: Deprecate box.session.push

The `box.session.push` Lua function and `box_session_push` C API
function are deprecated starting from Tarantool 3.0. Calling any
of these functions for the first time results in printing a warning
message to the log.

The new compat module option `box_session_push_deprecation` was
introduced to control whether the functions are still available.
With the old behavior, which is the default in Tarantool 3.0,
`box.session.push` is still available. With the new behavior,
any attempt to use it raises an exception.

(Please create https://tarantool.io/compat/box_session_push_deprecation)

We are planning to switch the compat option to the new behavior
starting from Tarantool 4.0 with the ability to revert to the
old behavior. Starting from Tarantool 5.0 we are planning to
drop `box.session.push` completely.

It lists the environment variables that are taken into account by the
config module.

Part of #8862

NO_DOC=reflected in https://github.com/tarantool/doc/issues/3544

The next commit needs to output a lot of environment variables, their
types in the config schema, defaults and so on. It looks much more
readable when formatted in the tabular form.

This commits adds a simple tabular formatter for this purpose.

Part of #8862

NO_DOC=the new module is internal
NO_CHANGELOG=see NO_DOC

Sometimes we need to know somethings about ancestor nodes during
processing of a current node.

Now, the `<schema node>.computed.annotations` table represents all the
annotations from the root of the schema down to the given schema node
merged into a flat table. If the same named annotation met several times
on the path, the descendant value is preferred.

This computed annotations table is available everywhere, where the
schema node is returned or passed as a callback's argument. For example,
in values of the `<schema>:pairs()` iterator and in the `validate()`
schema node's callback.

It will be used in one of the next commits this way:

```lua
for _, w in instance_config:pairs() do
    if w.schema.computed.annotations.enterprise_edition then
        <...>
    else
        <...>
    end
end
```

Part of #8862

NO_DOC=the schema module is internal, at least now
NO_CHANGELOG=see NO_DOC

Pin in cache the collation identifiers that are referenced by space format
and/or indexes, so that they can't be deleted.

Closes #4544

NO_DOC=bugfix

It was possible to delete a collation, which is in use by a key_def.

Part of #4544

NO_DOC=bugfix
NO_CHANGELOG=next commit

Since number type was introduced we can not assume if tuples are
equal by comparison then their sizes are equal too. So the place
the assumption is used is fixed.

Closes #8899

NO_DOC=bugfix

Getting rid of C++ exception is our long term goal. This commit removes
the exceptions from the IPROTO code as follows:

- Make iproto_set_msg_max return -1 instead of raising an exception
  on error. Update box_set_net_msg_max accordingly.
- Make box_process_auth return -1 instead of raising an exception on
  error. Move it along with box_process_vote (which never fails) to
  the extern "C" namespace.
- Remove try/catch from iproto_connection_on_input, tx_process_misc,
  tx_process_connect. Use a label instead.
- Panic instead of raising an exception on an error starting an IPROTO
  thread in iproto_init. It should be fine because the function is
  called only at startup and shouldn't normally fail.
- Remove try/catch from iproto_do_cfg_f. It turns out that this function
  never fails so we don't even need to use an error label.
- Drop iproto_do_cfg_crit and move the assertion to iproto_do_cfg
  because this function should never fail and so we don't need the
  wrapper.

After this commit, the only exception-handling piece of code left in
IPROTO is tx_process_replication. Dropping it would entail patching
replication, relay, and recovery code so it was postponed.

NO_DOC=code cleanup
NO_TEST=code cleanup
NO_CHANGELOG=code cleanup

The buffer is unlimited so allocations from it never fail. Even if
decide to add a hard limit some day, it's better to do it somewhere in
one place to simplify testing so the dropped code isn't going to be
needed.

NO_DOC=code cleanup
NO_TEST=code cleanup
NO_CHANGELOG=code cleanup

The mempools are unlimited so allocations from them never fail. Even if
we decide to add a hard limit some day, it's better to do it somewhere
in one place to simplify testing so the dropped code isn't going to be
needed.

NO_DOC=code cleanup
NO_TEST=code cleanup
NO_CHANGELOG=code cleanup

malloc isn't supposed to fail so let's use xmalloc helper in rmean_new
and get rid of allocation failure paths.

NO_DOC=code cleanup
NO_TEST=code cleanup
NO_CHANGELOG=code cleanup

The new function is a wrapper around the socketpair system call.
It takes the same arguments as the socket constructor and returns
two socket objects representing the two ends of the newly created
socket pair on success.

It may be useful for establishing a communication channel between
related processes.

Closes #8927

@TarantoolBot document
Title: Document new socket functions

Two socket module functions and one socket object method were added to
Tarantool 3.0:

- `socket.from_fd(fd)`: constructs a socket object from the file
  descriptor number. Returns the new socket object on success. Never
  fails. Note, the function doesn't perform any checks on the given
  file descriptor so technically it's possible to pass a closed file
  descriptor or a file descriptor that refers to a file, in which case
  the new socket object methods may not work as expected.

- `socket:detach()`: like `socket:close()` but doesn't close the socket
  file descriptor, only switches the socket object to the closed state.
  Returns nothing. If the socket was already detached or closed, raises
  an exception.

  Along with `socket.from_fd`, this method may be used for transferring
  file descriptor ownership from one socket to another:

  ```Lua
  local socket = require('socket')
  local s1 = socket('AF_INET', 'SOCK_STREAM', 'tcp')
  local s2 = socket.from_fd(s1:fd())
  s1:detach()
  ```

- `socket.socketpair(domain, type, proto)`: a wrapper around the
  [`socketpair`][1] system call. Returns two socket objects representing
  the two ends of the new socket pair on success. On failure, returns
  nil and sets [`errno`][2].

  Example:

  ```Lua
  local errno = require('errno')
  local socket = require('socket')
  local s1, s1 = socket.socketpair('AF_UNIX', 'SOCK_STREAM', 0)
  if not s1 then
      error('socketpair: ' .. errno.strerror())
  end
  s1:send('foo')
  assert(s2:recv() == 'foo')
  s1:close()
  s2:close()
  ```

[1]: https://man7.org/linux/man-pages/man2/socketpair.2.html
[2]: https://www.tarantool.io/en/doc/latest/reference/reference_lua/errno/

The new socket object method is equivalent to socket:close except it
doesn't close the underlying socket file descriptor - it just switches
the socket object to the closed state.

It may be useful for transferring fd ownership from a socket object, to
for example, a net.box connection.

Part of #8927

NO_DOC=later
NO_CHANGELOG=later

The new function constructs a socket object from a file descriptor
number. No checks are performed on the given fd so the user may pass
a closed fd or a fd that refers to a file, in which case the socket
object methods will fail and/or work not as expected.

It may be useful for creating a socket object from a file descriptor
inherited from a parent process.

The only tricky part here is setting the socket itype, which is used
for setting the correct receive buffer size for datagram sockets, see
commit 11fb3ab9 ("socket: evaluate buffer size in recv / recvfrom").
On error the itype is silently set to 0.

Part of #8927

NO_DOC=later
NO_CHANGELOG=later

Small library currently depends on Tarantool core through 'exception.h'.
This is not the way to go. Let's drop this dependency and instead of
moving _xc functions to Tarantool repo we can just stop using them. Our
current policy is to panic on OOM in case of runtime allocation.

Part of #7327

NO_DOC=<OOM behaviour is not documented>
NO_CHANGELOG=<no OOM expectations>
NO_TEST=<no test harness for checking OOM>

The test can be used for regression testing. It is advisable to tune
the machine: check the NUMA configuration, fix the pstate or similar
CPU autotune. Although, running dozen times gives more-less stable
result for the peak performance, that should be enough for regression
identification.

NO_DOC=adding an internal test
NO_CHANGELOG=ditto
NO_TEST=ditto

There must be a couple of rules:
* foreign key from non-temporary space to temporary space must be
  forbidden since after restart all existing links will be broken.
* foreign key from non-local space to local space must be forbidden
  on any replica all existing can be broken.

This patch implements the rules.

Closes #8936

NO_DOC=bugfix

The test fails on osx_debug quite frequently with the following error:

NO_WRAP
> box/before_replace.test.lua                                     [ fail ]
>
> Test failed! Result content mismatch:
> --- box/before_replace.result	Mon Aug  7 10:36:06 2023
> +++ /tmp/t/rejects/box/before_replace.reject	Mon Aug  7 10:42:03 2023
> @@ -899,6 +899,7 @@
>  ...
>  for i = 1,17 do gen_inserts() end
>  ---
> +- error: fiber slice is exceeded
>  ...
>  test_run:cmd('restart server test')
>  s = box.space.test_on_schema_init
> @@ -906,7 +907,7 @@
>  ...
>  s:count()
>  ---
> -- 1
> +- 0
>  ...
>  -- For this test the number of invocations of the before_replace
>  -- trigger during recovery multiplied by the amount of return values
> @@ -921,7 +922,8 @@
>  -- the value is only increased by 1, and this change is visible only in memory.
>  s:get{1}[2] == 1 + 16999 + 17000 + 1 or s:get{1}[2]
>  ---
> -- true
> +- error: '[string "return s:get{1}[2] == 1 + 16999 + 17000 + 1 o..."]:1: attempt to
> +    index a nil value'
>  ...
>  test_run:cmd('switch default')
>  ---
NO_WRAP

Let's try to increase the max fiber slice up to 15 seconds to avoid
that, like we do in other potentially long tests. (The default value
is 1 second.)

NO_DOC=test fix
NO_CHANGELOG=test fix

This should suppress the following coverity issues:

https://scan7.scan.coverity.com/reports.htm#v39198/p13437/fileInstanceId=146712118&defectInstanceId=18978766&mergedDefectId=1563095
https://scan7.scan.coverity.com/reports.htm#v39198/p13437/fileInstanceId=146712113&defectInstanceId=18978750&mergedDefectId=1563094

While we are at it, let's use space_by_id instead of space_cache_find
because read view creation is a rare operation affecting all spaces so
caching the last space by id doesn't make any sense.

NO_DOC=code health
NO_TEST=code health
NO_CHANGELOG=code health

In #7309 a truncation of a space that was referenced by foreign
key from some other space was prohibited.

It appeared that this solution is too bothering since a user can't
truncate a space even if he truncated referring space before that.

Fix it by allowing space truncate if referring spaces are empty.
Also allow drop of the primary index in the same case with the
same reason: logically the index along with all space data is
not needed for consistency if there's no referring data.

Note that by design space truncate is implemented quite similar
to space drop. Both delete all indexes, from secondary to primary.
Since this patch allows deletion of the primary index (which is
the action that actually deletes all data from the space), this
patch changes the result of space drop too: the space remains
alive with no indexes, while before this patch it remained alive
with no secondary indexes but with present primary. In both cases
the behaviour is quite strange and must be fixed in #4348. To
make tests pass I had to perform drop in box.atomic manually.

Closes #8946

NO_DOC=bugfix

Closes #8926

@TarantoolBot document
Title: Document `inherit_fds` option of `popen.new`

The new option takes an array of file descriptor numbers that should be
left open in the child process if the `close_fds` flag is set. If the
`close_fds` flag isn't set, the option has no effect because in this
case none of the parent's file descriptors are closed anyway.

The option may be useful for establishing a communication channel
between a Tarantool instance and a process spawned by it with
`popen.new`, for example, using `socket.socketpair`.

Allocation from malloc and the fiber region never fail so we can use x*
variants to simplify code and improve coverage.

NO_TEST=code cleanup
NO_CHANGELOG=code cleanup

@TarantoolBot document
Title: Remove `OutOfMemory` error from `popen.new` and `popen.read`

These functions don't raise the `OutOfMemory` exception anymore so all
mentions of it should be removed from the documentation:

 - https://www.tarantool.io/en/doc/latest/reference/reference_lua/popen/#popen-new
 - https://www.tarantool.io/en/doc/latest/reference/reference_lua/popen/#popen-read

Remove hard-coded compiler version for the `test-release-asan` target
in the .test.mk file.

NO_DOC=make
NO_TEST=make
NO_CHANGELOG=make

This patch introduces all metrics configuration.

Part of #8861

NO_DOC=tarantool/doc#3544 links the most actual schema,
       no need to update the issue.

Bump the metrics submodule to 1.0.0-3-4865675c

NO_DOC=metrics submodule bump
NO_TEST=metrics submodule bump
NO_CHANGELOG=metrics submodule bump

All LuaJIT related LSan warnings were suppressed in the scope of the
commit 985548e4 ("asan: suppress all
LSAN warnings related to LuaJIT"), since all compiler flags tweaks were
enclosed in LuaJIT CMake machinery. As a result of the commit in LuaJIT
submodule tarantool/luajit@a86e376 ("build: introduce LUAJIT_USE_ASAN
option") ASan and LSan support has been finally added to LuaJIT runtime,
so it was decided to remove LSan suppressions for LuaJIT functions.
Unfortunately, it was not so easy as it looked like.

At first, Lua global state is not closed properly at Tarantool instance
exit (see <tarantool_free> in src/main.cc and <tarantool_lua_free> in
src/lua/init.c for more info), so LSan false-positive leaks are detected
(for more info, see #3071). Hence, the original LSan suppression for
lj_BC_FUNCC is returned back (temporarily) until the aforementioned
issue is not resolved.

Furthermore, the internal LuaJIT memory allocator is not instrumented
yet, so to find any memory faults it's worth building LuaJIT with system
provided memory allocator (i.e. enable LUAJIT_USE_SYSMALLOC option).
However, again, since Tarantool doesn't finalize Lua universe the right
way, so running Tarantool testing routine with LUAJIT_USE_SYSMALLOC
enabled generates false-positive LSan leaks. Return back here to enable
LUAJIT_USE_SYSMALLOC, when #3071 is resolved.

Last but not least, the default value of fiber stack size is 512Kb, but
several tests in test/PUC-Rio-Lua-5.1-test suite in LuaJIT submodule
(e.g. some cases with deep recursion in errors.lua or pm.lua) have
already been tweaked according to the limitations mentioned in #5782,
but the crashes still occur while running LuaJIT tests with ASan support
enabled. Experiments once again confirm the notorious quote that "640 Kb
ought to be enough for anybody".

Anyway, LuaJIT tests are added to <test-release-asan> target in .test.mk
and LUAJIT_TEST_ENV is extended with required ASan and LSan options.

Follows up #5878

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

It is convenient to access configuration using `config:get()` from the
application script (`app.file` or `app.module`).

However, before this commit, it was not possible, because the
configuration was not considered as applied before the application
script is loaded.

Now, the script loading is moved into the post-apply phase.

The resulting sequence of steps on startup/reload is the following.

* collect configuration information (from all the sources)
* <if the previous step is failed, set `check_errors` status and break>
* apply the configuration (call all the appliers)
* <if the previous step is failed, set `check_errors` status and break>
* <set the new successful status: `ready` or `check_warnings`>
* call post-apply hooks (including application script loading)
* <if the previous step is failed, set `check_errors` status and break>
* <set the new successful status: `ready` or `check_warnings`>

I would like to briefly comment the changes in the tests.

* `app_test.lua`: added the check for the new behavior (call
  config:get() from the app script)
* `appliers_test.lua`: fixed the applier call (`.apply` ->
  `.post_apply`)
* `config_test.lua`: fixed status observed in the app script
  (`*_in_progress` -> `ready`)

Part of #8862

NO_DOC=reflected in https://github.com/tarantool/doc/issues/3544

The declarative config has the `app` section with following three
parameters:

* `app.file` -- the path to the script file
* `app.module` -- the same, but the script is searched using the Lua
  search paths (and loaded using `require`)
* `app.cfg` -- a user provided configuration (arbitrary map)

This commit adds a few success/failure cases. The goal is to have simple
test examples to add more ones easier in a future.

Implemented several test helpers for typical scenarios that can be used
outside of the application script test.

Part of #8862

NO_DOC=testing improvements
NO_CHANGELOG=see NO_DOC

The usual environment configuration source is useful for parametrized
run:

```
TT_MEMTX_MEMORY=<...> tarantool --name <...> --config <...>
```

However, sometimes a user may need to set a default value, which doesn't
rewrite one that is provided in the configuration. Now, it is possible
to do using the environment variables with the `_DEFAULT` suffix.

```
TT_MEMTX_MEMORY_DEFAULT=<...> tarantool --name <...> --config <...>
```

This feature may be especially useful for wrappers that run tarantool
internally with some default paths for data directories, socket files,
pid file. We likely will use it in the `tt` tool.

Part of #8862

NO_DOC=added into https://github.com/tarantool/doc/issues/3544 manually

It simplifies code reusage in the following commit, which adds the
second environment config source.

Part of #8862

NO_DOC=no user-visible changes, just code refactoring
NO_CHANGELOG=see NO_DOC
NO_TEST=see NO_DOC

Bump test-run to new version with the following improvements:

- luatest: fix detect tarantool crash at exit [1]
- Fix bug when lua script name truncated by dot [2]
- Raise an error and log it if test timeouts are set incorrectly [3]
- Pin PyYAML version to 5.3.1 [4]
- Add ability to set path to executable file [5]
- Migrate tarantoolctl from tarantool repository [6]
- Fix test-run crash when default server is crashed [7]
- Disable reproduce content printing [8]

[1] tarantool/test-run@be693d1
[2] tarantool/test-run@a6405f1
[3] tarantool/test-run@d34ecb0
[4] tarantool/test-run@704420e
[5] tarantool/test-run@0a70001
[6] tarantool/test-run@ad43d8f
[7] tarantool/test-run@b31329e
[8] tarantool/test-run@31f0ced

NO_DOC=test
NO_TEST=test
NO_CHANGELOG=test

* ci: introduce testing workflow with sanitizers
* build: introduce LUAJIT_USE_ASAN option
* test: introduce test:done TAP helper
* memprof: remove invalid assertions
* ci: clean up workflow for exotic builds

Closes #5878

NO_DOC=LuaJIT submodule bump
NO_TEST=LuaJIT submodule bump
NO_CHANGELOG=LuaJIT submodule bump

The is_old compat option method is more reliable than comparing
the current option value to 'old' directly because it may be set to
'default', in which case we also have to check the default value.

Follow-up #8807

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

The is_new and is_old methods are the same for all compat options so
they should be defined in a metatable. A good thing about this change
is that it removes is_new and is_old from serialization:

* Before:

NO_WRAP
  tarantool> require('compat').yaml_pretty_multiline
  ---
  - is_new: 'function: 0x4175d6e8'
    is_old: 'function: 0x4175d790'
    brief: |
      Whether to encode in block scalar style all multiline strings or ones
      containing "\n\n" substring. The new behavior makes all multiline string output
      as single text block which is handier for the reader, but may be incompatible
      with some existing applications that rely on the old style.

      https://tarantool.io/compat/yaml_pretty_multiline
    current: default
    default: new
  ...
NO_WRAP

* After:

NO_WRAP
  tarantool> require('compat').yaml_pretty_multiline
  ---
  - current: default
    brief: |
      Whether to encode in block scalar style all multiline strings or ones
      containing "\n\n" substring. The new behavior makes all multiline string output
      as single text block which is handier for the reader, but may be incompatible
      with some existing applications that rely on the old style.

      https://tarantool.io/compat/yaml_pretty_multiline
    default: new
  ...
NO_WRAP

To achieve that, we have to remove the option name from the usage error
message but it seems to be okay because such errors shouldn't happen in
practice and the error message is clear enough to figure out what went
wrong.

Follow-up #8807

NO_DOC=refactoring
NO_CHANGELOG=refactoring

Run release build LTO testing inside a Docker container created from the
`tarantool/testing:ubuntu-jammy-clang16` image with Clang 16 installed.

Closes #318

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

Run release build testing inside a Docker container created from the
`tarantool/testing:ubuntu-jammy-clang16` image with Clang 16 installed.

Closes #317

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

NO_DOC=build
NO_TEST=build
NO_CHANGELOG=build