In our SQL implementation temporary spaces are used. They come to
MVCC engine in two variants - NULL or ephemeral. In both cases
MVCC engine must not do anything, there are several checks for
that in different parts of code.

Normalize these checks and make them similar to each other.

Part of #8648
Part of #8654

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

The only place where this static function is used is more general
static function - memtx_tx_track_read_story. This is a bit
confusing - usually slow variant stand for public inline 'fast'
method.

So merge both functions in one.

Part of #8648
Part of #8654

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

By a mistake in 8a565144 a shortcut was added to procedure
that handles gap write: it was considered that if the writing
transaction is the same as reading - there is no actual conflict
that must be stored further. That was a wrong decision: if such
a transaction yields and another transaction comes and commits
a value with the same key - the first one must go to conflicted
state since it has read no more possible state.

Another similar mistake was made in e6f5090c, where writing
after full scan of the same transaction was not tracked as read.
Obviously that was wrong: if some other transaction overwrites
the key and commits - this transaction must go to read view since
it did not see anything by this key which is not so anymore.

Fix it, reverting the first commit and an modifying the second and
add a test.

Closes #8326

NO_DOC=bugfix

There's a special 'point hole' mechanism in mvcc transactional
manager that manages point gap reads by full key when no raw
tuple was found in the index. For instance, it's the only way
to collect gap reads for non-tree indexes.

Once a new tuple is inserted to the index, the read records are
transferred to the normal read set in the corresponding story.
Actually after that the 'point hole' record in no more needed.

So let's remove it.

While we are here, drop unused point_holes_size, improve names
and comments.

Part of #8648
Part of #8654

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Before this patch there were several different places in the code
that deal with referencing tuple in space, setting in_index member
and marking the story as retained or not. But logically all above
is about the same - about placing a story to the top of a chain,
i.e. the first story in version list to which index points.

This commit refactors these things a bit. This mostly relates to
two functions - memtx_tx_story_new and memtx_tx_story_link_top.

Changes in memtx_tx_story_new are based on the fact that if a story
is created by tuple, it is or immediately will be at the top of
chain. Considering this we can omit argument `is_referenced_to_pk`
and always create a story ready to be in top of chain. If a story
is already in the top - nothing else is needed; if it is to become
the top - memtx_tx_story_link_top must be called after.

Further, linking to top of chain is needed exactly in two cases:
* if a story just created by memtx_tx_story_new must become a top
* if a chain is reordered involving the top story (the top and the
  next stories are swapped)
These two cases are logically very close but still different.
Even more, previously there were two functions for that:
memtx_tx_story_link_top_light and memtx_tx_story_link_top
correspondingly. This commit introduces one function for that
(although with one more argument) that also incapsulates
activities about referencing tuples and marking stories as
retained.

After this patch the rules are logical and simple:
* if a tuple is inserted - call _story_new and _link_top(.. true).
* if a story of existing clean tuple is needed - call _story_new.
* if a chain is reordered involving top story - _link_top(.. false).

Part of #8648
Part of #8654

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Remove runtime allocation error handling and use panic-on-fail
versions of allocation functions. Reasons for that:
* Memory error handling was never tested an probably doesn't work.
* Some return codes was ignored so the code had obvious flaws.
* Rollback in case of memory error made some code overcomplicated.

Part of #8648
Part of #8654

NO_DOC=no new functionality added
NO_TEST=no new functionality added
NO_CHANGELOG=no new functionalily added

Conflict trackers are used to store information that if some
transaction is committed then some another transaction must be
aborted. This happens when the first transaction writes some
key while the other reads the same key. On the other hand there
are another trackers - read trackers - that are designed to
handle exactly the same situation. That's why conflict trackers
can be simply replaced with read trackers.

That would allow to remove conflict trackers as not needed
anymore.

Part of #8648
Part of #8654

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

If addition of a tuple is rolled back while the corresponding
story is needed for something else (for example it stores a read
set of another transaction) - the story cannot be deleted.
Now there's a special flag `rollbacked` that is set to true
for such stories, and the flag must be considered in places
where history chains are scanned. That approach also requires
psn to be set for rolled-back transactions, which surprisingly
not as simple as it to say. All that makes the code complicated
and hard to maintain.

There's another approach for managing rolled back stories: simply
set their del_psn to a low enough value (lower than any existing
transaction's PSN) and (if necessary) push them to the end of
history chain. Such a story would be invisible to any transaction
due to already existing mechanisms, that's what is needed.

In order to provide "low enough" del_psn it will be natural to
assign real PSN starting from some predefined value, so any value
below that predefined value will be less that any existing PSN and
thus "low enough".

Implement this more simple approach.

Part of #8648
Part of #8654

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

That's strange, but in this test in a group of simple test cases
there are test cases that checks replaces, updates and deletes,
but occasionally there's no test case that checks inserts.

Fix it and add simple test cases for inserts.

No logical changes.

Part of #8648
Part of #8654

NO_DOC=new test case
NO_CHANGELOG=new test case

It is useful for options like `log_level` (with values like `5` or
`'info'`) or `synchro_quorum` (with values like `5` or `'N / 2 + 1'`).

This is a temporary solution. I want to introduce an explicit union
schema node later.

Fixes #8725

NO_DOC=the module is for internal use from the config code
NO_CHANGELOG=see NO_DOC

This is a shortcut for an array of unique string values from the given
list of allowed values. Useful for schema nodes declarations that
describes permissions such as {'read'}, {'read', 'write', 'execute'}
and so on.

Part of #8725

NO_DOC=the module is for internal use from the config code
NO_CHANGELOG=see NO_DOC

This is a shortcut for a string scalar with the given allowed values.
Suitable for declarations of options like `wal_mode`: `none`, `write`,
`fsync`.

Part of #8725

NO_DOC=the module is for internal use from the config code
NO_CHANGELOG=see NO_DOC

The function interprets typeless data from an environment variable
as a value of the given type. Raises an error if it is not possible.

Part of #8725

NO_DOC=the module is for internal use from the config code
NO_CHANGELOG=see NO_DOC

The method allows to walk over the schema and process scalar, map and
array nodes somehow.

Part of #8725

NO_DOC=the module is for internal use from the config code
NO_CHANGELOG=see NO_DOC

This method is useful to merge several configuration data from different
sources. For example, a config formed from environment variables, a
config read from a local file, a config from etcd.

Part of #8725

NO_DOC=the module is for internal use from the config code
NO_CHANGELOG=see NO_DOC

This method adds default values from the `default` annotation for missed
fields. The annotation is supported for scalar values. The default can
be conditional, it is controlled by the `apply_default_if` annotation.

Part of #8725

NO_DOC=the module is for internal use from the config code
NO_CHANGELOG=see NO_DOC

The method allows to perform hierarchical data transformations using
schema information. For example, apply a function to all string scalars.
Or replace all missed values with their defaults from the schema.

Part of #8725

NO_DOC=the module is for internal use from the config code
NO_CHANGELOG=see NO_DOC

It allows to walk over the hierarchical data and process it with
knowledge about corresponding schema nodes.

In conjunction with user-provided annotations (stored in the schema
nodes) it offers a powerful way to slice the data in different ways.

Part of #8725

NO_DOC=the module is for internal use from the config code
NO_CHANGELOG=see NO_DOC

It allows to construct a hierarchical data step-by-step or modify it
easily. Just like a field assignment operation for a flat data.

The function validates the value that is to be assigned, so data
constructed using a sequence of `:set()` calls is valid.

It performs the same path validation as `<schema object>:get()` to prevent
indexing of a scalar value or assigning an unknown record's field.

Part of #8725

NO_DOC=the module is for internal use from the config code
NO_CHANGELOG=see NO_DOC

It allows to acquire a nested data with several convenience properties.

* Optional chaining semantic: `<schema object>:get(data, 'foo.bar')`
  works similarly to TypeScript's `data?.foo?.bar` operator.
* Path validation against the schema: it prevents attempt to get an
  unknown field and attempt to index a scalar value.

Part of #8725

NO_DOC=the module is for internal use from the config code
NO_CHANGELOG=see NO_DOC

It allows to add a specific validator for a particular schema node.

Part of #8725

NO_DOC=the module is for internal use from the config code
NO_CHANGELOG=see NO_DOC

The annotation allows to restrict values accepted by a schema node. It
effectively implements a enumeration type.

Part of #8725

NO_DOC=the module is for internal use from the config code
NO_CHANGELOG=see NO_DOC

The config module should ensure that a received configuration data
corresponds to a given schema before start to process it.

This commit implements a simple validation procedure.

There are possible optimization opportunities, such as generation a
validation code by a given schema.

No optimization tricks are used here, because the configuration
validation occurs at the configuration update. It is not a frequent
operation.

Part of #8725

NO_DOC=the module is for internal use from the config code
NO_CHANGELOG=see NO_DOC

It is the initial commit for the schema-aware data manipulation module.

Just add constructors here. Methods will be added in further commits.

The future config module will validate and manipulate hierarchical
configuration data. The schema module is a set of utilities for the
config module.

Part of #8725

NO_DOC=the module is for internal use from the config code
NO_CHANGELOG=see NO_DOC

It was disabled because ligomp.a on some AArch64 systems (e.g. CentOS)
is compiled without PIC support. Now Tarantool doesn't depend on ligomp,
so it's safe to turn on PIC for AArch64 systems.

Follow-up #7536

NO_DOC=build
NO_TEST=build

It was disabled because dlsym tests failed when PIC is on (#7640).
However, later it turned out that the issue is not related to PIC, and
LuaJIT was turned off for such tests by commit 67e79b15 ("test: turn
LuaJIT off in tests with dlsym"). Now it's safe to turn on PIC for FreeBSD.

Follow-up #7536

NO_DOC=build
NO_TEST=build

OpenMP is no longer used since commit 4f617b70 ("box: introduce
memtx_sort_threads config parameter"). All dependencies on libgomp
should be removed.

Follow-up #7689

NO_DOC=build
NO_TEST=build

The 2.11.0 release is out now.

It reverts PR #8367.

NO_DOC=no code changes
NO_CHANGELOG=see NO_DOC
NO_TEST=see NO_DOC

Historically, we encode strings that contain invalid or non-printable
utf-8 sequences in YAML as binary base64 blobs. We do that because of
limitations/bugs of the YAML encoder, which refuses to encode invalid
utf-8 strings. To work around this issue, we introduced the helper
utf8_check_printable, which is basically a copy of yaml_check_utf8,
and treat strings for which it fails as binary data (MP_BIN).

This commit updates the YAML submodule to the version where all known
issues with encoding invalid/unprintable utf-8 strings are fixed and
removes special treatment of such strings (drops utf8_check_printable).
Now unprintable or invalid utf-8 sequences are emitted as code points,
e.g. '\xFF' or '\uFFFF'. This change is a pre-requisite for introducing
the new varbinary type to Lua. Without it plain strings would be
implicitly converted to varbinary after decoding/encoding them in YAML,
which would be confusing.

Closes #8756

NO_DOC=bug fix

The fselect space/index method returns select results in a pretty format
that looks like a table in the console, which encodes the results in the
YAML format. Before the yaml_pretty_multiline compat option was added,
multi-line strings emitted by the console looked mangled so the fselect
method returns a table of strings instead of a multi-line string for the
output to look good. Moreover, it also prepends the strings with the
special zero-width space character (U+200B) to employ a bug in the YAML
formatter: this character isn't printed but it makes the YAML formatter
emit the string without quotes. This hack won't work when #8756 is fixed
because we'll update the YAML formatter to output all non-printable
character as code points.

Now, that we have yaml_pretty_multiline compat option available and
enabled by default, we don't need these hacks anymore. Let's patch
fselect to return the result as a multi-line string and drop the
'use_nbsp' option, which was used to enable or disable the zero-width
space character hack. Let's also drop the 'print' option because one can
now print the fselect results directly.

Needed for #8756

NO_DOC=fselect is undocumented

The future config module works with hierarchical data a lot and needs a
way to show a table as a part a log message. The config module issues
log messages at different log levels and it is undesirable to perform
JSON encoding for messages that will be filtered out by the current log
level.

It is complicated to achieve the goal using the `log` module. Let's add
a wrapper as the temporary solution. The wrapper can be eliminated after
solving the problems in the `log` module that are linked in the
wrapper's code.

The wrapper also simplifies debugging of the config module itself.

NO_DOC=the module is for internal use from the config code
NO_CHANGELOG=see NO_DOC

I'm going to add the `src/box/lua/config/utils/schema.lua` file and
other *.lua files into the `src/box/lua/config` directory.

Let's adjust gitignore for generated *.lua.c file.

NO_DOC=gitignore
NO_TEST=gitignore
NO_CHANGELOG=gitignore

It was decided to include the `osx_debug.yml` and `osx_release.yml`
workflows to the default tests run (without the `full-ci` label).
Now we can get test results for macOS faster and without an extra
load on CI.

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

The `space` table of `net.box` stream objects shouldn't have its own
fields, since it used to lookup spaces and those would interfere its
`__index` metamethod, but currently, it erroneously has `_stream` and
`_stream_space_cache` — replace them with upvalues.

Also fixed the same problem for the `index` table of `space` wrapper
objects returned by `__index` metamethod of `net.box` objects `space`
table.

Closes #8598

NO_DOC=bugfix

There are two new options introduced in Tarantool CLI:
* --name (-n) to specify instance name to be started. The option can be
  omitted in case TT_INSTANCE_NAME environment variable is set.
* --config (-c) to specify the path to the config file. If the option
  is not set the value of TT_CONFIG environment variable is considered.

Closes #8613

Co-authored-by: Sergey Bronnikov <sergeyb@tarantool.org>

@TarantoolBot document
Title: introduce new CLI options for conf module

There are two new options introduced in Tarantool CLI:
* --name (-n) to specify instance name to be started. The option can be
  omitted in case TT_INSTANCE_NAME environment variable is set.
* --config (-c) to specify the path to the config file. If the option
  is not set the value of TT_CONFIG environment variable is considered.

Previously Tarantool binary entered interactive mode starting Lua REPL
by default. As a result of the patch Tarantool binary yields the message
with usage and doesn't run Lua interpreter.

Part of #8613

@TarantoolBot document
Title: disable Lua REPL by default

Previously Tarantool binary entered interactive mode starting Lua REPL
by default. As a result of the patch Tarantool binary yields the message
with usage and doesn't run Lua interpreter.

The help message is printed to the stdout unconditionally, however, the
stream should be chosen properly considering the reason the usage should
be displayed. As a result of this patch, the help message can be printed
to the desired stream being passed as an argument.

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Commit/rollback triggers are run asynchronously, upon receiving the
write status from WAL. We can't run them in the original fiber that
submitted the WAL request, because it would open a time window between
writing a transaction to WAL and committing it in tx, which could lead
to violating the cascading rolback principles. As a result,
commit/rollback triggers run with admin privileges.

The issue was already solved for confirming async transaction, but
session and user are still not correct, when the transaction is
confirmed by the limbo. Let's fix this issue by temporarily setting
session and credentials to the original fiberfor running
commit/rollback triggers.

Closes #8742

NO_DOC=bugfix

Currently some transactions on synchronous space fail to complete with
the `ER_CURSOR_NO_TRANSACTION` error, when on_rollback/on_commit triggers
are set.

This is caused due to the fact, that some rollback/commit triggers
require in_txn fiber variable to be set but it's not done when a
transaction is completed from the limbo. Callbacks, which are used to
work with iterators (`lbox_txn_pairs` and `lbox_txn_iterator_next`),
acquire tnx statements from the current transactions, but they cannot
do that, when this transaction is not assigned to the current fiber, so
`ER_CURSOR_NO_TRANSACTION` is thrown.

Let's assign in_txn variable when we complete transaction from the limbo.
Moreover, let's add assertions, which check whether in_txn() is correct,
in order to be sure, that `txn_complete_success/fail` always run with
in_txn set.

Closes #8505

NO_DOC=bugfix

See the docbot request for details.

Closes #7932

@TarantoolBot document
Title: correct recovery of mixed transactions

In this patch implemented correct recovery of mixed transactions. To do
this, set  `box.cfg.force_recovery` to `true`. If you need to revert to
the old behavior, don't set the `force_recovery` option.

What to do when one node feeds the other a xlog with mixed transactions?
Let there be two nodes (`node#1` and `node#2`). And let the data be
replicated from `node#1` to `node#2`. Suppose that at some point in time,
`node#1` is restoring data from an xlog containing mixed transactions. To
replicate data from `node#1` to `node#2`, do the following:
1. Stop `node#2` and delete all xlog files from it
2. Restart `node#1` by setting `force_recovery` to `true`
3. Make `node#2` rejoin to `node#1` again.