** Implementation details **

We disable DDL by patching the existing on_replace_dd_system_space
trigger callback installed for each system space so that now it raises
an error in case the current schema version is less than the most
recent one known to this build. Since to perform a schema upgrade
we need to execute DDL, we suppress the error for the fiber that is
currently running a schema upgrade. To achieve that, the upgrade script
calls box_schema_upgrade_begin and box_schema_upgrade_end before
starting and after completing a schema upgrade. The functions keep track
of the fiber that is currently running a schema upgrade so that we can
allow all DDL operations for it. We also allow DDL during recovery so
that we can replay DDL statements written to the WAL.

Since there may be a bug in the `box.schema.upgrade` implementation,
we export `box.internal.run_schema_upgrade`, which runs the given
function as a schema upgrade script (allowing DDL). The user may use
this function to recover after a schema upgrade failure.

** Note about the tests **

A test server instance started by luatest grants permissions to the
guest user so that luatest can execute commands on it. It means that if
a test uses a generated snap file committed to the repository for a test
server instance, it will fail because granting permissions is a DDL
operation. To prevent this, we have to regenerate snap files so that
they contain all required permissions. This works because a test server
instance grants permissions with the `if_not_exists` flag.

The problem is that it isn't easy to regenerate the snap files for the
following tests because there's no generator script:
 - `test/box-luatest/gh_6794_recover_nonmatching_xlogs_test.lua`
 - `test/box-luatest/gh_7974_force_recovery_bugs_test.lua`

So we temporarily disable these tests and file tickets to fix them.

Other notes:
 - We drop `test/box-luatest/upgrade/2.9.1` and make the test using it
   use `test/box-luatest/upgrade/2.10.0` instead. We do this because
   2.9.1 was never released and the earliest Tarantool version using the
   2.9.1 schema version is 2.10.0. This shouldn't affect the test
   anyhow.
 - We drop the part of the `user_auth_history_last_modified_upgrade`
   test that checks that creating users/roles with an old schema works
   fine because this is forbidden now.
 - We wrap the code that creates a space with an old schema in the
   downgrade test in `box.internal.run_schema_upgrade`. Even though it's
   unsupported now, we still need to check that space creation works
   after a downgrade.

Closes #7149

@TarantoolBot document
Title: Document that DDL is disabled with an old system schema

Executing DDL operations with an old (not upgraded) system schema is
dangerous and might result in unexpected breakages. So we decided to
explicitly forbid all DDL operations with an old system schema until
`box.schema.upgrade()` is called.  Note, one can still call `box.schema`
functions with an old schema provided they do nothing, for example, if
an object is created with the `if_not_exists` flag and the object with
same id already exists:

```lua
box.schema.create_space('test', {if_not_exists = true})
```

Otherwise an attempt to create a space with an old schema will raise
an error like shown below:

```yaml
tarantool> box.schema.space.create('test')
---
- error: Your schema version is 1.6.8 while Tarantool
    3.0.0-entrypoint-262-g3eaba1cef686 requires a more recent
    schema version. Please, consider using box.schema.upgrade().
...
```

The patch adds a new check to the _space on_replace trigger failing
on attempt to drop a system table.

Closes #5279

NO_DOC=bugfix

NO_DOC=changelog update
NO_TEST=changelog update

Bump the metrics submodule to 1.0.0 version.

NO_DOC=submodule bump
NO_TEST=submodule bump
NO_CHANGELOG=submodule bump

Bump test-run to new version with the following improvements:

- lib: propagate test status 'skip' [1]
- Show overall progress while running [2]
- Follow test timeout for luatest [3]
- Run luatest test by pattern [4]
- Refactor command to run luatest test [5]
- Bump luatest to 0.5.7-39-g89da427 [6]
- consistent mode: fix worker's vardir calculation [7]

[1] tarantool/test-run@6fbb7fd
[2] tarantool/test-run@c5fa909
[3] tarantool/test-run@f67d523
[4] tarantool/test-run@264af05
[5] tarantool/test-run@e19bb11
[6] tarantool/test-run@3e74192
[7] tarantool/test-run@aac77f5

NO_DOC=testing stuff
NO_TEST=testing stuff
NO_CHANGELOG=testing stuff

Currently on tuple encoding failure we raise Lua error. In many placess
the error is not handled in Lua C code and we get misc leaks. Let's
instead pass error as return value.

Note that generally speaking encoding code can raise an error on OOM.
Which will lead to leak again. Hopefully application will be killed by
OOM killer instead. Other then that we expect no more errors in the
code. If code calls a user defined callback then pcall is used (see
lua_field_inspect_ucdata for example). So the turn from raising errors
to returning error code seems the right direction.

Closes #7939

NO_DOC=bugfix

This will bring new ibuf_truncate method.

Part of #7939

NO_TEST=internal
NO_CHANGELOG=internal
NO_DOC=internal

* Fix IR_RENAME snapshot number. Follow-up fix for a32aeadc.
* OSX: Disable unreliable assertion for external frame unwinding.
* Disable unreliable assertion for external frame unwinding.
* Handle on-trace OOM errors from helper functions.
* LJ_GC64: Make ASMREF_L references 64 bit.
* lldb: introduce luajit-lldb
* x64/LJ_GC64: Fix emit_rma().
* Limit path length passed to C library loader.

Closes #7745
Part of #4808
Part of #8069
Part of #8516

NO_DOC=LuaJIT submodule bump
NO_TEST=LuaJIT submodule bump

Currently is_nullable property of a functional index part disables the
unique property of the index. The bug is in func_index_compare(), which
compares functional keys first, and if they are equal it compares the
primary keys. This behaviour is correct only when some part of the key
is NULL (and for non-unique indexes), but for now the primary keys are
compared unconditionally. Fix this by checking for NULL key parts.

Closes #8587

NO_DOC=bugfix

This patch adds a check that sqlXPrintf() does not fail in the built-in
SQL function printf(). There are two possible problems: the result might
get too large, or there might be an integer overflow because internally
int values are converted to size_t.

Closes #tarantool/security#122

NO_DOC=bugfix

This patch fixes problems with INSERT INTO ... SELECT FROM optimization.
These problems appeared after 6b8acd8f, where the check became redundant,
but was not updated. Two problems arose:
1) an assertion or segmentation fault when optimization was used and the
source space does not have an index;
2) optimization can be used even if the indexes are incompatible.

The second problem does not result in changes that are user-visible, so
there is no test.

Closes #8661

NO_DOC=bugfix

Hostname is a useful piece of information in state reports. So it was
decided to add it to box.info.

Hostname is obtained on requested and is not cached.

Closes #8605

@TarantoolBot document
Title: Add hostname to box.info

This patch adds hostname to box.info, it can be useful e.g. to supplement
various instance state reports. It is not cached and is requested on
each call.

The error of invalid negation occurred because of invalid check,
which did not cover the case when value is equal to INT64_MIN, so the
negation of INT64_MIN equal to itself. This must be fixed, because
negation of INT64_MIN is undefined behavior.

It is fixed by the explicit check for the value variable.

NO_DOC=refactoring
NO_CHANGELOG=refactoring
NO_TEST=refactoring

Add support for accepting IPROTO requests with space or index name instead
of identifier (name is preferred over identifier to disambiguate missing
identifiers from zero identifiers): mark space identifier request
key as present upon encountering space name, and delay resolution of
identifier until request gets to transaction thread.

Add support for sending DML requests from net.box connection objects with
disabled schema fetching by manually specifying space or index name or
identifier: when schema fetching is disabled, the space and index tables of
connections return wrapper tables that store necessary context (space or
index name or identifier, determined by type, connection object and space
for indexes) for performing requests. The space and index tables cache the
wrapper table they return.

Closes #8146

@TarantoolBot document
Title: Space and index name in IPROTO requests

Refer to design document for details:
https://www.notion.so/tarantool/Schemafull-IPROTO-cc315ad6bdd641dea66ad854992d8cbf?pvs=4#f4d4b3fa2b3646f1949319866428b6c0

Change original `space_by_name` to `space_by_name0` and
`space_index_by_name` to `space_index_by_name0`, since they accept
NULL-terminated names, and add `space_by_name` and `space_index_by_name`
for arbitrary strings.

Needed for #8146

NO_CHANGELOG=refactoring
NO_DOC=refactoring
NO_TEST=refactoring

Previously it wasn't allowed to change instance UUID in _cluster.
When needed, it had to be done manually by deleting the instance
from _cluster and inserting it back with a new UUID. Or not to be
done at all.

Re-UUID (like re-name) was reported to be used when people didn't
want to register new replica IDs. They wanted to rejoin lost
replicas from scratch but keep the numeric ID. With UUID they
could deal by either setting it explicitly to the old value on a
new instance, or by doing the manual re-UUID like described above.

This commit is supposed to make things simpler. If a replica has a
name, then its re-join with another UUID is not an error. Its
record in _cluster is automatically updated to store the new UUID.

That is only possible if the old-UUID-instance is not connected
anymore and is not listed in replication cfg.

Closes #5029

@TarantoolBot document
Title: Instance rebootstrap with new UUID but same ID and name
If an instance has a non-empty instance name
(`box.cfg.instance_name`), then at rebootstrap it can keep the
name and its old numeric ID (space `_cluster['id']` field).

This might be needed if one doesn't want to pollute `_cluster`
with new rows, and somewhy doesn't want to or can't just drop the
rows belonging to the dead replicas.

In order for this to work 1) the rebootstrapping replica must keep
its old non-empty instance name, 2) the other instances should not
have any alive connections to the old dead replica. Ideally, the
old replica should be just deleted from `box.cfg.replication`
everywhere.

When that works, the old row in `_cluster` is automatically
updated with the new instance UUID.

The instance name is carried with instance UUID everywhere in the
replication protocols. It is visible in all other instances via
_cluster and is displayed in monitoring.

Part of #5029

@TarantoolBot document
Title: `box.cfg.instance_name` and `box.info.name`
The new option `box.cfg.instance_name` allows to assign the
instance name to a human-readable text value to be displayed in
the new info key - `box.info.name`. Instances can see names of
their peers in `box.info.replication[id].name`.

The name is broadcasted in "box.id" built-in event as
"instance_name" key. It is string when set and nil when not set.

When set, it has to be unique in the instance's replicaset.

If a name wasn't set on cluster bootstrap (was forgotten or the
cluster is upgraded from a version < 3.0), then it can be set
on an already running instance via `box.cfg.instance_name`.

To change or drop an already installed name one has to use
`box.cfg.force_recovery == true` in all instances of the cluster.
After the name is updated and all the instances synced, the
`force_recovery` can be set back to `false`.

The name can be <= 63 symbols long, can consist only of chars
['0'-'9'], '-' and 'a'-'z'. It must start with a letter. When
upper-case letters are used in `box.cfg`, they are automatically
converted to lower-case. The names are host- and DNS-friendly.

The replicaset name is carried with replicaset UUID wherever any
sanity validations are needed like whether 2 instances belong to
the same replicaset.

Part of #5029

@TarantoolBot document
Title: `box.cfg.replicaset_name` and `box.info.replicaset.name`
The new option `box.cfg.replicaset_name` allows to assign the
replicaset name to a human-readable text value to be displayed in
the new info key - `box.info.replicaset.name` - and to be
validated when the instances in the replicaset connect to each
other.

The name is broadcasted in "box.id" built-in event as
"replicaset_name" key. It is string when set and nil when not set.

When set, it has to match in all instances of the entire
replicaset.

If a name wasn't set on cluster bootstrap (was forgotten or the
cluster is upgraded from a version < 3.0), then it can be set
on an already running instance via `box.cfg.replicaset_name`.

To change or drop an already installed name one has to use
`box.cfg.force_recovery == true` in all instances of the cluster.
After the name is updated and all the instances synced, the
`force_recovery` can be set back to `false`.

The name can be <= 63 symbols long, can consist only of chars
['0'-'9'], '-' and 'a'-'z'. It must start with a letter. When
upper-case letters are used in `box.cfg`, they are automatically
converted to lower-case. The names are host- and DNS-friendly.

The patch adds 2 new entities to replication: the concept of a
cluster which has multiple replicasets and a name for this
cluster.

The name so far doesn't participate in any replication protocols.
It is just stored in _schema and is validated against the config.

The old mentions of 'cluster' (in logs, in some protocol keys like
in the feedback daemon) everywhere are now considered obsolete and
probably will be eventually replaced with 'replicaset'.

Part of #5029

@TarantoolBot document
Title: `box.cfg.cluster_name` and `box.info.cluster.name`
The new option `box.cfg.cluster_name` allows to assign the cluster
name to a human-readable text value to be displayed in the new
info key - `box.info.cluster.name` - and to be validated when the
instances in the cluster connect to each other.

The name is broadcasted in "box.id" built-in event as
"cluster_name" key. It is string when set and nil when not set.

When set, it has to match in all instances of the entire cluster
in all its replicasets.

If a name wasn't set on cluster bootstrap (was forgotten or the
cluster is upgraded from a version < 3.0), then it can be set
on an already running instance via `box.cfg.cluster_name`.

To change or drop an already installed name one has to use
`box.cfg.force_recovery == true` in all instances of the cluster.
After the name is updated and all the instances synced, the
`force_recovery` can be set back to `false`.

The name can be <= 63 symbols long, can consist only of chars
'0'-'9', '-' and 'a'-'z'. It must start with a letter. When
upper-case letters are used in `box.cfg`, they are automatically
converted to lower-case. The names are host- and DNS-friendly.

The new function check_global_ids_integrity() checks that the
replicaset UUID specified in the config and found in the data
match. Instance UUID is created at bootstrap and validated at the
beginning of recovery, not in the end. Hence not checked here.

For now this function is not very useful, but soon there will be
more global IDs stored in WAL which will need validation.

Needed for #5029

NO_DOC=refactoring
NO_CHANGELOG=refactoring
NO_TEST=already covered

Node name stores a DNS- and host- friendly string name. It will be
used in the next patches for some new global names: cluster,
replicaset, and instance.

Part of #5029

NO_DOC=internal
NO_CHANGELOG=internal

It was named 'cluster', but really was just about the replicaset.
This is going to be even more confusing soon, because there will
be introduced an actual concept of cluster as multiple
replicasets.

The patch renames it to 'replicaset'. `box.info.cluster` now means
the whole cluster and is empty so far. Next patches will add here
the cluster name.

Part of #5029

@TarantoolBot document
Title: `box.info.cluster` is renamed to `box.info.replicaset`

Done since 3.0.0. The old behaviour can be reverted back via the
`compat` option `box_info_cluster_meaning`.

`box.info.cluster` key is still here, but now means a totally
different thing - the entire cluster with all its replicasets.

<h2>Compat documentation</h2>

`box.info.cluster` default meaning is the whole cluster with all
its replicasets. To get info about only the current replicaset
`box.info.replicaset` should be used.

In old versions (< 3.0.0) `box.info.cluster` meant the current
replicaset and `box.info.replicaset` didn't exist.

<h3>Old and new behaviour</h3>

New behaviour:
```
tarantool> box.info.cluster
---
- <some cluster keys>
...

tarantool> box.info.replicaset
---
- uuid: <replicaset uuid>
- <... other attributes of the replicaset>
...
```
Old behaviour:
```
tarantool> box.info.cluster
---
- uuid: <replicaset uuid>
- <... other attributes of the replicaset>
...

tarantool> box.info.replicaset (= nil on < 3.0.0)
---
- uuid: <replicaset uuid>
- <... other attributes of the replicaset>
...
```

<h3>Known compatibility issues</h3>

VShard versions < 0.1.24 do not support the new behaviour.

<h3>Detecting issues in you codebase</h3>

Look for all usages of `box.info.cluster`, `info.cluster`, and
even just `.cluster`, `['cluster']`, `["cluster"]`. For the new
behaviour to work all of them have to use 'replicaset' key.

Replicaset UUID was stored in _schema['cluster'] tuple. This is
going to be confusing soon, because there will be introduced an
actual concept of cluster as multiple replicasets.

The patch renames it to 'replicaset_uuid'.

Part of #5029

@TarantoolBot document
Title: Update '_schema' with new 'replicaset_uuid' key

Currently _schema system space is documented to have 'cluster' key
with replicaset UUID value. Now this key is deleted (since 3.0)
and the UUID is stored in 'replicaset_uuid' key.

box_on_join() was called not only on IPROTO_JOIN but also on
IPROTO_REGISTER. The name was a bit misleading. It is now called
box_register_replica().

The old box_register_replica() is renamed to
box_insert_replica_record(). It says "insert record", because this
is what it does - inserts a new tuple into _cluster space.

It also skips the check whether the instance is read-only. It
allows to use the function on the bootstrap master so as it could
register itself.

Needed for #5029

NO_DOC=refactoring
NO_CHANGELOG=refactoring
NO_TEST=already covered

_cluster on_replace trigger in alter.cc was a huge multi-screen
function with many indentation levels. It was not too bad in its
old state. But soon it is going to get more complicated as
_cluster will get a new field - 'name'. Its update will require
own on commit and rollback triggers, own checks, errors.

Trying to fit name processing into the old monstrous function
didn't look too tempting, so the trigger now is split into
multiple functions serving update, insert, and delete separately.
At least it helps to reduce the indentation.

Needed for #5029

NO_DOC=refactoring
NO_CHANGELOG=refactoring
NO_TEST=already covered

To tell whether the instance is anon there used to be just one
flag in C code: replication_anon.

Having one flag both for cfg and for the actual state is bad
because if cfg is updated, then there is a moment when that flag
can't be safely used to check the actual state.

For example, when replication_anon had been true and was set to
false, it took time to register the instance. In the meantime the
C flag replication_anon was already false, although the instance
is still anon (not present in _cluster).

In the existing code it could lead to insignificant errors like
when an anon instance was being registered, it could already
accept IPROTO_REGISTER requests. It would fail on ER_READONLY
instead of ER_UNSUPPORTED. It wasn't a critical problem, but still
it wasn't correct to use cfg flag for checking the actual state.

Now there is a separate cfg flag and a function for checking the
real state.

This patch is done because soon there will be a new option which
also takes time to change: instance name. This commit sets a
pattern how to deal with such options.

In scope of #5029

NO_DOC=refactoring
NO_CHANGELOG=refactoring
NO_TEST=already covered

If attempt to set `box.cfg{replication_anon = false}` failed, the
instance's ballot event had is_anon = false nonetheless. This was
because on reconfig failure the option's scope guard did revert
the option itself in C++ code, but didn't update the ballot.

NO_DOC=bugfix

There were a few places where instance and replicaset UUIDs from
box.cfg where passed as arguments in box.cc functions. It was fine
although sometimes could cause struggling like "where along the
callstack replicaset UUID is created when it was nil in cfg".

But soon the situation will get more complicated. There will be up
to 3 new arguments - cluster, replicaset, and instance names.

Passing all these identifiers as parameters would be cumbersome.
The patch makes the UUIDs fetched from the config by the functions
which need them. The same will be done with the names where they
are relevant.

Needed for #5029

NO_DOC=refactoring
NO_CHANGELOG=refactoring
NO_TEST=already covered

box.cfg.force_recovery used to be needed only during box.cfg() in
a few places, but its usage is going to extend.

In future commits about cluster/replicaset/instance names it will
be needed to allow rename. It won't be entirely legal (hence can't
be done without any flags), but won't be fully illegal either.

The "valid" rename will be after upgrading, when an
old cluster updated to a new version and wants to start using the
names. Then it will have to set force_recovery, set the names,
sync the instances, drop force_recovery. One-time action to allow
old installations use the new feature - the names.

Part of #5029

NO_DOC=refactoring
NO_CHANGELOG=refactoring
NO_TEST=already covered

_schema on_replace trigger used to treat replace as commit. No
support for rollback at all and all changes are immediately
visible.

That is fine most of the time but still incorrect. The patch makes
the space properly respect transactions.

This is done as a preparation for adding several new _schema keys
which will be transactional from the start. Would be strange to
leave certain keys ignoring transactions. Hence this fix is done.

In scope of #5029

NO_DOC=bugfix
NO_CHANGELOG=couldn't happen with legal usage of public APIs

Deletion of the own entry from _cluster space is allowed during
the join stage, because the remote master could have already had
the joining instance UUID in _cluster space but then deleted it.

Then for the joining instance it looks like deletion of self from
_cluster. But that is fine - in the end of join the master will
register the replica again.

The case is handled, but not covered with a test. The patch adds
one.

NO_DOC=test
NO_CHANGELOG=test

The function replica_check_id() is called on any change in
_cluster: insert, delete, update. It was supposed to check if the
replica ID is valid - not nil, not out of range (VCLOCK_MAX).

But it was also raising an error when the ID matched this
instance's ID unless the instance was joining. That happened even
if a _cluster tuple was updated without changing the ID at all.
For example, if one would just do
_cluster:replace(_cluster:get(box.info.id)).

That was a surprising side effect of the ID checker which blocked
next patches. The next commits are going to introduce a new field
in _cluster (replica name) which will be mutable. Such behaviour
of replica_check_id() wouldn't allow to update even that new
field.

Better do the check in the only place where the mutation can
happen - on deletion. Since replica ID is a primary key in
_cluster, it can't be updated there. Only inserted or deleted.

Needed for #5029

NO_DOC=bugfix and refactoring
NO_CHANGELOG=couldn't happen unless user touched _cluster in a
    weird way
NO_TEST=covered by next commits, too insignificant for an own test

It wasn't allowed to drop it, but was allowed to update. The patch
bans it. Firstly, it was not supposed to work. Secondly, a future
patch will introduce a new tuple in _schema, which would store
replicaset UUID too. It won't allow UUID update.

Would be strange, if the update would be let through one _schema
tuple and wouldn't work via another.

Needed for #5029

NO_DOC=bugfix

The next major release of Tarantool (3.0.0) will not support Cartridge.
So tet's be ready in advance. This patch removes the cartridge and crud
integration test runs to make the integration tests pass for #8289.

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

This directory belongs to the EE repository.

Also, delete test/enterprise-tap because we don't have any TAP tests in
the EE repository.

Follow-up commit d8faca69 ("lua: implement stubs for etcd client").

NO_DOC=gitignore
NO_TEST=gitignore
NO_CHANGELOG=gitignore

Patch made following changes:
- add a CMake function lua_multi_source that unpacks symbol names and
  Lua source paths in passed list and compiles Lua source into *.lua.c
  sources using lua_source() function. Function is needed for embedding
  Lua modules consisting from a number of files to Tarantool EE.
- add using lua_multi_source instead of lua_source for etcd client module.
- remove function prototypes and declarations related to extra features.
  All this code is moved to Tarantool EE source code.

Part of #8620
Follows up https://github.com/tarantool/tarantool-ee/issues/443

NO_CHANGELOG=refactoring
NO_DOC=refactoring
NO_TEST=refactoring

Fefactor lua name resolution, simplify and comment.
Add an ability to specify path with brackets, for example in
'box.space[512]:get' or 'box.space["test"]:get'.
Only literals (strings and numbers) are supported.

Closes #8604

@TarantoolBot document
Title: square brackets in procedure resolution for Lua calls

Square brackets are now supported in Lua call procedure resolution. This is
applicable to `net.box` connection objects `call` method as well as
`box.schema.func.call`.

Examples of function calls with square brackets can be found in the test to
this patch.

Default token for actions provided by GitHub lacks
read and write permissions for private projects. This
commit changes the token used for the stale PRs workflow
to the dedicated token with needed permissions.

NO_DOC=Workflow fix
NO_TEST=Workflow fix
NO_CHANGELOG=Workflow fix

Sometimes we need negative timestamps to work with dates before
1970. But seems such cases were even covered in tests. So there
wasn't any handling of negative timestamps with fraction part.
Such datetime objects had incorrect string representation (e.g.
"1963-11-22T12:30:02.-999"). This patch fixes it.

Closes #8570

NO_DOC=bugfix

Seems that problem code part was ported from Lua as is. But there
is some difference between modulo operator in C and in Lua. Lua
always returns positive value but in C result could be negavive.
This difference led to the case when after subtraction nsec part
of datetime object become negative that yielded weird result on
attempt to get string representation (e.g."2008-02-03T03:36:43.-100Z").
This patch fixes it.

Part of #8570

NO_DOC=bugfix
NO_CHANGELOG=see next commit