Previously, sysbench benchmark was running inside Docker on CI.
According to the performance testing best practices it is a quite
doubtful approach. So let's switch to a more traditional practice:
run on hardware.

Closes tarantool/tarantool-qa#158

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

When the master is just starting up it's possible for replica's JOIN
request to arrive right in time to bypass ER_LOADING check (after master
is fully recovered) but still fail due to ER_READONLY: box.cfg.read_only
is only read and set after box_cfg() (its C part) returns.

In this case the joining replica simply exits with an error and doesn't
retry JOIN.

Let's fix that. Make ER_READONLY a recoverable error and let replica
retry joining after receiving ER_READONLY.

Anonymous nodes relied on ER_READONLY to forbid replication from
anonymous to normal replicas. That check doesn't work anymore.
So introduce explicit checks banning replication from anonymous nodes.

Note, there were some alternatives to this fix.

First of all, theoretically, we could stop firing ER_LOADING later,
after box_cfg() is complete. This solution wouldn't work because it
would lead to deadlocks: the nodes would be stuck in replicaset_sync(),
because each of them rejects replication with ER_LOADING.

Another solution would be to read the real box.cfg.read_only value
earlier, in order to allow replication right after the node finishes
recovery.
This would also be bad, because we should never let a node become
writeable before box.cfg is finished. Even after local_recovery is
complete, the node should stay read-only until it synchronizes with
other replicas.

That said, neither of the two alternatives fit, so the solution with
retrying JOIN on ER_READONLY was chosen.

Since the bug is fixed, re-enable the test in which it was discovered:
replication-py/init_storage.test.py

Also, remove replication/ddl.test.lua from fragile list, since this bug
was the only reason for its fragility.

Closes #5337
Closes #6966

NO_DOC=minor bugfix

The source file will be added to the Enterprise repository.

NO_DOC=cmake
NO_TEST=cmake
NO_CHANGELOG=cmake

If Tarantool is built as a subproject, and there's a Lua source file
stored in the superproject directory, its path won't start with
PROJECT_SOURCE_DIR and hence won't be replaced with PROJECT_BINARY_DIR
by the lua_source() function. As a result, the lua.c file generated by
lua_source() will be put in the source directory instead of the binary
directory. Fix this by using CMAKE dirs instead of PROJECT dirs.

Ironically, CMAKE dirs were replaced with PROJECT dirs everywhere in
the first place to allow building Tarantool as a subproject, see commit
d8097325 ("cmake: align folders dependencies"). This patch basically
reverts that change.

NO_DOC=cmake
NO_TEST=cmake
NO_CHANGELOG=cmake

Disable the init_storage.test.py test due to the bug #6966.
After a bug fix the test should be enabled again.

NO_DOC=testing stuff
NO_CHANGELOG=testing stuff
NO_TEST=testing stuff

fiber_new_ex() used to ignore fiber_attr flags when the fiber was
taken from the cache, not created anew.

It didn't matter much though for the public API, because the only
public flag in fiber_attr was FIBER_CUSTOM_STACK (which can be
set via fiber_attr_setstacksize()).

Anyway that was a bug for internal API and would lead to issues in
the future when more public flags are added. The patch fixes it.

NO_DOC=Bugfix
NO_CHANGELOG=No reproducer via public API

There was a user who complained about this code crashing:

    f = fiber_new_ex(...);
    fiber_start(f);
    fiber_cancel(f);

The crash was at cancel. It happened because the fiber finished
immediately. It was already recycled after fiber_start() return.

Recycled fiber didn't have any flags, so fiber_cancel() didn't
see the fiber was already dead and tried to wake it up. It crashed
when the fiber tried to call its 'fiber->f' function which was
NULL.

In debug build the process fails earlier with an assertion on
'fiber->fid != 0'.

It can't be really fixed because the problem is the same as with
use-after-free. The fiber could be not recycled but already freed
completely, returned back to the mempool.

This patch tries to help the users by a panic with a message
saying that it wasn't just a crash, it is a bug in user's code.

There is an alternative - make fibers never return to the mempool.
Then fiber_cancel() could ignore recycled fibers. But it would
lead to another problem that if the fiber is already reused, then
fiber_cancel() would hit a totally irrelevant fiber who was
unlucky to reuse that fiber pointer. It seems worse than panic.

Same problem exists for `fiber_wakeup()`, but I couldn't figure
out how to add a panic there and not add an `if` on the normal
execution path (which includes 'ready' and 'running' fibers).

Closes #6837

NO_CHANGELOG=The same crash remains, but happens a bit earlier and
  with a message.

@TarantoolBot document
Title: `fiber_cancel()` C API clarification

The documentation must warn that the fiber passed to
`fiber_cancel()` must not be already dead unless it was set to be
joinable. Same for `fiber_wakeup()` and all the other fiber
functions. A dead non-joinable fiber could already be freed or
reused.

Fibers with custom stack couldn't be reused via cord->dead list,
but neither were ever deleted via mempool_free(). They just leaked
until the cord was destroyed. Their custom stack also leaked.

It happened for all non-joinable custom-stack fibers. That was
because fiber_destroy() simply skipped the destruction if the
fiber is the current one.

It didn't affect joinable fibers because their fiber_destroy() is
done in another fiber. Their stack was deleted, but the fiber
itself still leaked.

The fix makes so fiber_destroy() is never called for the current
fiber. Instead, cord uses an approach like in pthread library -
the fiber who wants to be deleted is saved into cord->garbage
member. When some other fiber will want to be deleted in the
future, it will firstly cleanup the previous one and put self into
its place. And so on - fibers cleanup each other.

The process is optimized for the case when the fiber to delete is
not the current one - can delete it right away then.

NO_DOC=Bugfix

As a result of recording <crc32:update> method or <digest.crc32>
function wrong semantics is compiled (strictly saying, the resulting
trace produces the different result from the one yielded by
interpreter). The easiest solution is disabling JIT for particular
functions, however, such approach drops the overall platform
performance. Hence, the mentioned functions are rewritten line by line
via Lua C API to avoid JIT misbehaviour.

NO_DOC=no visible changes
NO_CHANGELOG=no visible changes

Use a new storage bucket, made specifically for open-source
third-party software distributions.

NO_DOC=testing
NO_TEST=testing
NO_CHANGELOG=testing

During the context switch required for backtracing a suspended fiber,
unwinders go crazy, as the unwind information they had gets implicitly
invalidated: provide an annotation for a dummy frame for
`coro_unwcontext`, as if it were at the bottom of the call-chain — that
way unwinders can normally proceed further.

We need to know the exact location of the stack pointer: replace the
16-byte stack alignment instruction on x86_64 macOS by adding the
`force_align_arg_pointer` attribute to `unw_getcontext_f`.

Needed for #4002

NO_DOC=bug fix
NO_CHANGELOG=bug fix
NO_TEST=unwind information annotation in inline assembly

Fiber call-chains end at `coro_{init, startup}`, but unwinders don't
stop there, trying to use `coro_{init, startup}` stack frame's return
address (which points to some garbage) and, in turn, failing. A similar
issue was experienced by seastar and julia (see JuliaLang/julia#23074
and scylladb/scylla#1909).

In order to make unwinding stop at `coro_{init, startup}`'s stack frame
we need to annotate it with CFI assembly: previously, annotation was
provided only for GCC on x86_64 — also provide it if ENABLE_BACKTRACE is
set during configuration.

Zero out rbp on x86_64 (to conform to x86_64 ABI): this requires setting
"-fomit-frame-pointer" compile flag for coro.c.

Backtrace collection from inactive fiber based on pseudo context-switch
relied on the stack frame structure: remove redundant
"-fno-omit-frame-pointer" and "-fno-stack-protector"
compile flags for other Tarantool sources.

For some reason unwinders ignore platform ABIs regarding ending of
call-chains: explicitly invalidate the topmost (`coro_{init, startup}`)
current frame information (CFI) for both x86_64 and AARCH64.

References:
1. glibc:
 * clone: https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/unix/sysv/linux/x86_64/clone.S;h=31ac12da0cc08a934d514fed1de9eba1cb3e8ec5;hb=ebbb8c9f64c3486603ef4ccee4dd2a5574e41039
 * start: https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/x86_64/start.S;h=9edd17b60cd54ec9eef11c76ab02322dcb5d057a;hb=5b736bc9b55115e67129e77db4de6cf193054cd2
2. seastar:
 * thread_context::main(): https://github.com/scylladb/seastar/blob/d27bf8b5a14e5b9e9c9df18fd1306489b651aa42/src/core/thread.cc#L278-L293
3. julia:
 * https://github.com/JuliaLang/julia/blob/2e2b1d2ad50fe12999cbded0b5acd3f0a36ec8c5/src/julia_internal.h#L90-L106
4. android:
 * https://cs.android.com/android/platform/superproject/+/master:bionic/libc/platform/bionic/macros.h;l=52-60;drc=2528dab7419a63f57fe20027886ba7dd3857aba8

Needed for #4002

NO_DOC=internal bug fix
NO_CHANGELOG=internal bug fix
NO_TEST=unwind information annotation in inline assembly

NO_DOC=no behavior changes
NO_TEST=no behavior changes
NO_CHANGELOG=no behavior changes

Added predefined system events: box.status, box.id, box.election and
box.schema.

Closes #6260

@TarantoolBot document
Title: Built-in events for pub/sub

Built-in events are needed, first of all, in order to learn who is the
master, unless it is defined in an application specific way. Knowing who
is the master is necessary to send changes to a correct instance, and
probably make reads of the most actual data if it is important. Also
defined more built-in events for other mutable properties like leader
state change, his election role and election term, schema version change
and instance state.

Built-in events have a special naming schema - their name always starts
with box.. The prefix is reserved for built-in events. Creating new events
with this prefix is banned. Below is a list of all the events + their names
and values:

1. box.id
Description - identification of the instance. Changes are extra rare. Some
values never change or change only once. For example, instance UUID never
changes after the first box.cfg. But is not known before box.cfg is called.
Replicaset UUID is unknown until the instance joins to a replicaset or
bootsa new one, but the events are supposed to start working before that -
right at listen launch. Instance numeric ID is known only after
registration. On anonymous replicas is 0 until they are registered
officially.
Value - {
    MP_STR “id”: MP_UINT; box.info.id,
    MP_STR “instance_uuid”: MP_UUID; box.info.uuid,
    MP_STR “replicaset_uuid”: MP_UUID box.info.cluster.uuid,
}

2. box.status
Description - generic blob about instance status. Its most commonly used
and not frequently changed config options and box.info fields.
Value - {
    MP_STR “is_ro”: MP_BOOL box.info.ro,
    MP_STR “is_ro_cfg”: MP_BOOL box.cfg.read_only,
    MP_STR “status”: MP_STR box.info.status,
}

3. box.election
Description - all the needed parts of box.info.election needed to find who
is the most recent writable leader.
Value - {
    MP_STR “term”: MP_UINT box.info.election.term,
    MP_STR “role”: MP_STR box.info.election.state,
    MP_STR “is_ro”: MP_BOOL box.info.ro,
    MP_STR “leader”: MP_UINT box.info.election.leader,
}

4. box.schema
Description - schema-related data. Currently it is only version.
Value - {
    MP_STR “version”: MP_UINT schema_version,
}

Built-in events can't be override. Meaning, users can't be able to call
box.broadcast(‘box.id’, any_data) etc.

The events are available from the very beginning as not MP_NIL. It's
necessary for supported local subscriptions. Otherwise no way to detect
whether an event is even supported at all by this Tarantool version. If
events are broadcast before box.cfg{}, then the following values will
available:
    box.id = {}
    box.schema = {}
    box.status = {}
    box.election = {}

This way the users will be able to distinguish an event being not supported
at all from box.cfg{} being not called yet. Otherwise they would need to
parse _TARANTOOL version string locally and peer_version in netbox.

Example usage:

 * Client:
   ```lua
   conn = net.box.connect(URI)
   -- Subscribe to updates of key 'box.id'
   w = conn:watch('box.id', function(key, value)
       assert(key == 'box.id')
       -- do something with value
   end)
   -- or to updates of key 'box.status'
   w = conn:watch('box.status', function(key, value)
       assert(key == 'box.status')
       -- do something with value
   end)
   -- or to updates of key 'box.election'
   w = conn:watch('box.election', function(key, value)
       assert(key == 'box.election')
       -- do something with value
   end)
   -- or to updates of key 'box.schema'
   w = conn:watch('box.schema', function(key, value)
       assert(key == 'box.schema')
       -- do something with value
   end)
   -- Unregister the watcher when it's no longer needed.
   w:unregister()
   ```

We use a special, less efficient index vtab if a space can store
compressed tuples. The problem is it isn't enough to look at a space
definition to figure out if there are compressed tuples in the space:
there may be compressed tuples left from before the alter operation that
disabled compression, since we don't rebuild tuples on alter. To update
an index vtab dynamically, we implement some complicated logic, but
it's buggy (results in a test failure in EE). Fixing it requires some
non-trivial effort, because a vtab may be changed after index creation
(when a space format is updated).

Let's drop this optimization altogether for now and use the same vtab
for both compressed and uncompressed indexes. We might return to this
issue in future, but first we need to run some benchmarks to check if
this optimization is worth the complexity. Possible ways how we could
resurrect this optimization:
 - Call get_raw from get directly (without function pointer), inline
   memtx_prepare_result_tuple, and move is_compressed flag to struct
   tuple for better cache locality.
 - Rebuild all tuples on space alter and use a different vtab for
   compressed indexes.

NO_DOC=bug fix
NO_TEST=enterprise
NO_CHANGELOG=unrelased

If tuple compression is enabled, txn_stmt::new_tuple points to
a temporary tuple created by uncompressing a compressed tuple stored
in an index. We must use txn_stmt::rollback_info::new_tuple instead.

NO_DOC=bug fix
NO_TEST=enterprise
NO_CHANGELOG=unreleased

This change adds execution of the `git clean -xffd` command to the
.github/actions/environment action to clean workspace more thoroughly.

NO_DOC=ci
NO_CHANGELOG=ci
NO_TEST=ci

This change adds the server start timeout to the PRESERVE_ENVVARS
environment variable to deliver it to 'packpack' docker containers
while running packaging workflows.

NO_DOC=ci
NO_CHANGELOG=ci
NO_TEST=ci

It looks like the step for storing the artifact with logs after workflow
failures was missed. Now it is added.

NO_DOC=ci
NO_CHANGELOG=ci
NO_TEST=ci

This change adds facility to store artifacts with logs after workflow
failures to FreeBSD workflows. The ChristopherHX/github-act-runner known
limitation

    You won't be able to run steps after a failure without using
    continue-on-error: true

is not relevant anymore [1]. So we can add steps with storing artifacts.

[1] https://github.com/ChristopherHX/github-act-runner/tree/58ae37abc6c2244d91822b8ba536aa0a8b829632#known-limitations

NO_DOC=ci
NO_CHANGELOG=ci
NO_TEST=ci

This change fixes hardcoded artifact paths by using ${{ env.VARDIR }}
expression.

At this moment VARDIR=/tmp/tnt, so affected workflows are:
- debug_aarch64.yml
- memtx_allocator_based_on_malloc.yml
- osx_11_aarch64.yml
- osx_11_aarch64_debug.yml

NO_DOC=ci
NO_CHANGELOG=ci
NO_TEST=ci

This change updates the 'test_odroid_arm64_no_deps' make target in the
.travis.mk file with the following construction instead of `make test`:

    make LuaJIT-test
    cd test && ./test-run.py --vardir ${VARDIR} --force $(TEST_RUN_EXTRA_PARAMS)

This is needed for consistency with other targets and saving artifacts
with failure logs to ${VARDIR}.

NO_DOC=ci
NO_CHANGELOG=ci
NO_TEST=ci

This change tries to fix the following sporadic test error:

    [007] not ok 4	http_client.sock_family:"AF_INET".test_cancel_and_errinj
    [007] #   ...arantool/tarantool/test/app-luatest/http_client_test.lua:221: Timeout check - status
    [007] #   expected: 408, actual: 200

Fixes tarantool/tarantool-qa#154

NO_DOC=testing
NO_TEST=testing
NO_CHANGELOG=testing

zlib.net is unavailable, so we have to download zlib distributions
from a backup storage on VKCS S3.

NO_DOC=testing
NO_TEST=testing
NO_CHANGELOG=testing

We need it for audit log.

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

The newly released 0.26.0 emits a warning on the `_box` variable.
From luacheck v.0.26.0 release notes:

"Function arguments that start with a single underscore
get an "unused hint". Leaving them unused doesn't result
in a warning. Using them, on the other hand, is a
new warning (№ 214)."

NO_DOC=testing
NO_TEST=testing
NO_CHANGELOG=testing

Reverting changes from 3cad9398,
originally introduced in 0b46154f

NO_DOC=testing
NO_TEST=testing
NO_CHANGELOG=testing

@TarantoolBot document
Title: Document constraints and foreign keys

See details here:
https://docs.google.com/document/d/1EtqfFSIMi6fDqj2y-9WgpunkHsszU65p6LfNly5E9Ag/

NO_TEST=no code changes

Implement complext foreign keys addition to field foreign keys.
They are quite similar to field foreign keys, the difference is:
* The are set in space options instead of format field definition.
* Several fields may be specified in relation.
* By design field foreign keys are more optimal.

One can set up foreign keys in space options:
box.schema.space.create(.. {.., foreign_key=<foreign_key>})
where foreign_key can be of one of the following forms:
 foreign_key={space=..,field=..}
 foreign_key={<name1>={space=..,field=..}, ..}
where field must be a table with local -> foreing fields mapping:
 field={local_field1=foreign_field1, ..}

NO_DOC=see later commits
NO_CHANGELOG=see later commits

Foreign key is a special type of constraint that makes a relation
between spaces. When declared in some space, each tuple in that
space refers to some tuple in another, foreign space. Reference is
defined in foreign key definition as a correspondence of field of
that spaces, local and remote.

Foreign key preserves that reference between tuples and consists
of two checks:
1. When a tuple is added to space with foreign space constraint,
it must be checked that there is a corresponding tuple in foreign
space, with the same values in fields according to foreign key
definitiion.
2. When a tuple is deleted from space that is a foreign space for
some other space, it must be checked that no tuple references the
deleted one.

This commit introduces field foreign keys that link spaces by
just one field. They are declared in tuple format in one of the
following forms:
 space:format{..,{name=.., foreign_key=<fkey>},..}
 space:format{..,{name=.., foreign_key={<name>=<fkey>}},..}
Where fkey has a form of a table:
 {space=<foreign space id/name>, field=<foreign field id/name>}

NO_DOC=see later commits
NO_CHANGELOG=see later commits

There are cases when we need to be sure that a space by given
id or name is not deleted; and if it is replaced (in space cache),
there's need to track pointer to new space. Like ib constraints:
they must hold a pointer to struct space while it's very hard to
determine whether there'a constraint that points to given space.

Implement space pin/unpin for this purpose. You can pin a space to
declare that the space is require to exist. To have to unpin it
when the space is not needed anymore.

NO_DOC=refactoring
NO_CHANGELOG=refactoring
NO_TEST=refactoring

I'm going to extend space cache API so it should be separated.
One function went to space.h/c.
No logical changes.

NO_DOC=refactoring
NO_CHANGELOG=refactoring
NO_TEST=refactoring

The previous commit adds tuple constraint lua functions that check
format of entire tuple. The problem was that in those functions
tuple could be accessed only by field indexes.

Add an ability to use field names too.

NO_DOC=see later commits
NO_CHANGELOG=see later commits

Implement whole tuple constraints in addition to field constraints.
They are quite similar to field constraints, the difference is:
 * The are set in space options instead of format field definition.
 * Entire tuple is passed to check function.
 * By design field constraints are a bit more optimal.

One can set up constraint in space options, with one or several
functions that must be present in _func space:
box.schema.space.create(.. {.. constraint='func1'})
box.schema.space.create(.. {.. constraint={name1='func1'})
box.schema.space.create(.. {.. constraint={name1='f1', name2='f2'})

NO_DOC=see later commits
NO_CHANGELOG=see later commits

tuple_format_new has lots of arguments, all of them necessary
indeed. But a small analysss showed that almost always there are
only two kinds of usage of that function: with lots of zeros as
arguments and lots of values taken from space_def.

Make two versions of tuple_format_new:
simple_tuple_format_new, with all those zeros omitted, and
space_tuple_format_new, that takes space_def as an argument.

NO_DOC=refactoring
NO_CHANGELOG=refactoring

Introduce field constraints - limitaions for particular fields.
Each constraint must refer to a function in _func space. For the
first step we expect lua functions with body there.

Field constraint checks are very close to field type checks, so
it's natural to implement them in tuple formats. On the other hand
tuple formats belong to tuple library, that does not include
functions (func.h/c etc), so constraints are split into two parts:
- a part in tuple library that implements arbitrary constraints
 with pointers to functions that actually check constraints.
- a part in box library which uses the part above, sets particular
 check functions and handles alter etc.

There are two not-so-obvious problems that are solved here:
 - Functions in _func space must be preserved while used by such
 constraints. Func pinning is used for this purpose.
 - During initial recovery constraits are created before _func
 space recovery, so we have to pospone constraint initialization.

One can set up constraint for any field in tuple format with one
or several functions that must be present in _func space:
space:format{name='a', constraint='func1'}
space:format{name='a', constraint={name1='func1'}}
space:format{name='a', constraint={name1='func1', name2='func2'}}

So constraint(s) can be set by one function name or by lua table
with function name values. Each consraints has a name that can be
specified directly (with string key in table) or imlicitly set to
the name of function.

The check function receives two arguments: the checking value and
the name of the constraint. Also the name of the failed constraint
is present in raised exception.

The only way to pass the constraint is to return true from its
function. All other values and exception are treated as failure
(exeptions are also logged).

NO_DOC=see later commits
NO_CHANGELOG=see later commits

Now C port allows to add a tuple or raw msgpack to it.

Add another function that encodes and appends given string.

NO_DOC=refactoring
NO_CHANGELOG=refactoring
NO_TEST=refactoring

gpr_alloc is a small library that is designed for simplification
of allocation of several objects in one memory block. It could be
anything, but special attention is given to string objects, that
are arrays of chars.

Typical usage consist of two phases: gathering total needed size
of memory block and creation of objects in given block.

NO_DOC=refactoring
NO_CHANGELOG=refactoring

There's several important stages of recovery of database: loading
from snapshot, then loading from WAL(s) and then normal operation.

Introduce a global recovery state that shows this stage.

Note that there's already a recovery state in memtx engine which is
very close but still different to the new introduced state. That
state in memtx is a private property of memtx that internally shows
initialization status of memtx spaces and indexes. Memtx can set the
value for convenience' sake, for example it can jump directly to
MEMTX_OK before snapshot loading in case of force recovert.

NO_DOC=refactoring
NO_CHANGELOG=refactoring
NO_TEST=refactoring

opt_def is an option parser from msgpack by given scheme.
The scheme consists of set of predefined types that the parser
can handle (something like int, enum, str). The problen is that
those predefined types must be generic, but there are cases when
a specific unusual must be parsed.

This patch introduces OPT_CUSTOM type that uses arbitrary function
callback and can be used for any non-generic option value.

NO_DOC=refactoring
NO_CHANGELOG=refactoring
NO_TEST=refactoring