This section describes the instance process configuration.

Part of #8778

NO_DOC=will be added later
NO_CHANGELOG=will be added later

This section describes general instance configuration options.

Part of #8778

NO_DOC=will be added later
NO_CHANGELOG=will be added later

This patch introduces the instance_config module. This module will
contain the instance configuration schema.

Part of #8778

NO_DOC=will be added later
NO_CHANGELOG=will be added later

The msgpack module can't lack `msgpack.NULL`, because it leads to the
'Please call box.cfg{} first' error at access to `box.NULL`.

An upcoming patch regarding schemas for tarantool's new hierarchical
configuration uses `box.NULL` during tarantool's initialization.

The override feature is tested on a set of modules, which are easy to
replace with an arbitrary table without losing an ability to pass
tarantool's initialization. The msgpack module will not meet the
criteria after the mentioned patch.

The test doesn't attempt to hold a precise list of modules available for
overriding. It rather checks the overall mechanics. So it is OK to
adjust the list.

NO_DOC=testing code change
NO_CHANGELOG=see NO_DOC

There are sporadic segfaults in libunwind during backtrace_collect().
Reproducible with the latest version, and there are open issues with
similar stacks:

https://github.com/libunwind/libunwind/issues/150
https://github.com/libunwind/libunwind/issues/260
https://github.com/libunwind/libunwind/issues/473

Let's disable ENABLE_BACKTRACE for AArch64 Linux until these issues are
resolved in libunwind.

Closes #8572
Part of #8791

NO_DOC=bugfix

The ability to support backtraces is checked in cmake/compiler.cmake,
it makes no sense to duplicate the check in rpm/tarantool.spec. Also do
not enable backtraces unconditionally in apk/APKBUILD and static-build.

Part of #6998

NO_DOC=build
NO_TEST=build
NO_CHANGELOG=build

Instead of relying on fixed xlog / snap pair in gh_6794_data, let's
better generate the desired files right in the test. This way we won't
face the problem of the files getting out of date. For example, when
schema version changes.

Remove the now unneeded gh_6794_data files and re-enable the
gh_6794_recover_nonmatching_xlogs test.

Closes #8701

NO_DOC=test
NO_CHANGELOG=test

This patch fixes the crontab expression in the
Lango team stale PRs workflow, so it runs not every minute,
but every hour.

Also, python version is updated to 3.11, since the script is
reliant onto `datetime.fromisoformat' changes, that were
introduced in that version.

NO_DOC=Workflow fix
NO_TEST=Workflow fix
NO_CHANGELOG=Workflow fix

The description of CURLOPT_UPLOAD [1] is confusing:

    If you use PUT to an HTTP 1.1 server, you can upload data without
    knowing the size before starting the transfer if you use chunked
    encoding. You enable this by adding a header like
    "Transfer-Encoding: chunked" with CURLOPT_HTTPHEADER.

In fact, libcurl adds this header itself. Actually we need to avoid
adding this header by libcurl with CURLOPT_INFILESIZE [2].

The CURLOPT_UPLOAD documentation has been updated [3].

1. https://github.com/curl/curl/blob/555bacd6d522bcca497573765056354f4d5d7b33/docs/libcurl/opts/CURLOPT_UPLOAD.3
2. https://curl.se/libcurl/c/httpput.html
3. https://github.com/curl/curl/pull/11300

Closes #8744

NO_DOC=bugfix

There's case when a transaction is rolled back from prepared
state. This happens when WAL fails, synchronized replication
confirmation failure or perhaps in other similar cases. By design
other RW transactions and transactions with READ_COMMITTED
isolation level are allowed to read prepared state. All these
transactions must be aborted in case of rollback of prepared
transaction since they definitely have read no more possible
database state.

This patch implements this abortion.

Closed #8654

NO_DOC=bugfix

Rollback is rather complicated part if MVCC implementation that
is meant to handle two kinds of rollback:
* rollback from in-progress state, if box.rollback() is called.
* rollback from prepared state, when WAL fails.
Unfortunately the last one was not properly tested and surely
has at least one flaw. When an inserting transaction becomes
prepared its stories could be linked as deleted (via del_stmt
pointer) by other in-progress transactions in order to maintain
correct visibility. The problem is that in case of rollback of
such prepared transaction those links remained. Broken links
breaks the chain structure, and some older changes (that were
linked as deleted before that hapless preparation) can become
visible to other transaction.

Refactor, simplify a bit that part of code and fix the issue
described above; cover with tests.

Closes #8648

NO_DOC=bugfix

Almost completely rewrite, simplify and comment this part of code.

Part of #8648
Part of #8654

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Almost completely rewrite, simplify and comment this part of code.

Part of #8648
Part of #8654

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

The latter flag is a bit wider: it reveals not only inserting
statements after deleting by the same transaction, but also
replacing and deleting statements after all kinds of previois
changes but the same transaction. This extended behavior will
be used in further commits.

Part of #8648
Part of #8654

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

It was an ugly solution when MVCC engine requires outside engine
to set this flag which is not convenient.

Remove it and use mode arguments to set up proper read trackers.

Part of #8648
Part of #8654

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

The function memtx_tx_story_delete is expected to delete fully
unlinked stories and thus should not try to unlink something by
itself. So remove unlink and add asserts instead.

Part of #8648
Part of #8654

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Now there are three kinds of very close trackers:
* The transaction have read some tuple that is not committed and
  thus not visible. This kind is now stored as gap_item with
  is_nearby = false.
* The transaction made a select or range scan, reading a key or
  range between two adjacent tuples of the index. This kind is
  stored as gap_iteam with is_nearby = true.
* A transaction completed a full scan of unordered index. This
  kind is stored as full_scan_item.

All these trackers serve for the same thing: to record a fact
that a transaction read something but didn't see anything.

There are some problems with the current solution:
* gap_item with is_nearby = false has several unused members that
  just consume space.
* bool is_nearby flag for type descriptin is an ugly solution.
* full_scan_item is separated from logically close items.

This commit joins all these trackers under one base (that is
struct gap_item_base) and solves problems above.

Part of #8648
Part of #8654

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Now read trackers are used both for cases when a transaction has
read an existing value and it has read nothing (read by key but
there was no visible tuple in this place). For latter case an
additional index_mask was used to identify from which index the
read was done. Along with that there was per-index interval gap
trackers.

This patch divides area of responsibility between read trackers
and gap tracker in the following way:
* Reads of existing visible values are stored in read trackers.
* Reads of non-existing or non-visible values are store in gap
  trackers.

This new approach allows to provide new invariants: gap trackers
are stored only at top of chain, and read trackers are stored
only at topmost committed story in chain.

Part of #8648
Part of #8654

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

In further commit this list will be used for tracking all read
gaps, not only 'nearby'. Since this rename has rather huge
diff, let's make it in separate commit.

No logical changes.

Part of #8648
Part of #8654

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

In our SQL implementation temporary spaces are used. They come to
MVCC engine in two variants - NULL or ephemeral. In both cases
MVCC engine must not do anything, there are several checks for
that in different parts of code.

Normalize these checks and make them similar to each other.

Part of #8648
Part of #8654

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

The only place where this static function is used is more general
static function - memtx_tx_track_read_story. This is a bit
confusing - usually slow variant stand for public inline 'fast'
method.

So merge both functions in one.

Part of #8648
Part of #8654

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

By a mistake in 8a565144 a shortcut was added to procedure
that handles gap write: it was considered that if the writing
transaction is the same as reading - there is no actual conflict
that must be stored further. That was a wrong decision: if such
a transaction yields and another transaction comes and commits
a value with the same key - the first one must go to conflicted
state since it has read no more possible state.

Another similar mistake was made in e6f5090c, where writing
after full scan of the same transaction was not tracked as read.
Obviously that was wrong: if some other transaction overwrites
the key and commits - this transaction must go to read view since
it did not see anything by this key which is not so anymore.

Fix it, reverting the first commit and an modifying the second and
add a test.

Closes #8326

NO_DOC=bugfix

There's a special 'point hole' mechanism in mvcc transactional
manager that manages point gap reads by full key when no raw
tuple was found in the index. For instance, it's the only way
to collect gap reads for non-tree indexes.

Once a new tuple is inserted to the index, the read records are
transferred to the normal read set in the corresponding story.
Actually after that the 'point hole' record in no more needed.

So let's remove it.

While we are here, drop unused point_holes_size, improve names
and comments.

Part of #8648
Part of #8654

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Before this patch there were several different places in the code
that deal with referencing tuple in space, setting in_index member
and marking the story as retained or not. But logically all above
is about the same - about placing a story to the top of a chain,
i.e. the first story in version list to which index points.

This commit refactors these things a bit. This mostly relates to
two functions - memtx_tx_story_new and memtx_tx_story_link_top.

Changes in memtx_tx_story_new are based on the fact that if a story
is created by tuple, it is or immediately will be at the top of
chain. Considering this we can omit argument `is_referenced_to_pk`
and always create a story ready to be in top of chain. If a story
is already in the top - nothing else is needed; if it is to become
the top - memtx_tx_story_link_top must be called after.

Further, linking to top of chain is needed exactly in two cases:
* if a story just created by memtx_tx_story_new must become a top
* if a chain is reordered involving the top story (the top and the
  next stories are swapped)
These two cases are logically very close but still different.
Even more, previously there were two functions for that:
memtx_tx_story_link_top_light and memtx_tx_story_link_top
correspondingly. This commit introduces one function for that
(although with one more argument) that also incapsulates
activities about referencing tuples and marking stories as
retained.

After this patch the rules are logical and simple:
* if a tuple is inserted - call _story_new and _link_top(.. true).
* if a story of existing clean tuple is needed - call _story_new.
* if a chain is reordered involving top story - _link_top(.. false).

Part of #8648
Part of #8654

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Remove runtime allocation error handling and use panic-on-fail
versions of allocation functions. Reasons for that:
* Memory error handling was never tested an probably doesn't work.
* Some return codes was ignored so the code had obvious flaws.
* Rollback in case of memory error made some code overcomplicated.

Part of #8648
Part of #8654

NO_DOC=no new functionality added
NO_TEST=no new functionality added
NO_CHANGELOG=no new functionalily added

Conflict trackers are used to store information that if some
transaction is committed then some another transaction must be
aborted. This happens when the first transaction writes some
key while the other reads the same key. On the other hand there
are another trackers - read trackers - that are designed to
handle exactly the same situation. That's why conflict trackers
can be simply replaced with read trackers.

That would allow to remove conflict trackers as not needed
anymore.

Part of #8648
Part of #8654

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

If addition of a tuple is rolled back while the corresponding
story is needed for something else (for example it stores a read
set of another transaction) - the story cannot be deleted.
Now there's a special flag `rollbacked` that is set to true
for such stories, and the flag must be considered in places
where history chains are scanned. That approach also requires
psn to be set for rolled-back transactions, which surprisingly
not as simple as it to say. All that makes the code complicated
and hard to maintain.

There's another approach for managing rolled back stories: simply
set their del_psn to a low enough value (lower than any existing
transaction's PSN) and (if necessary) push them to the end of
history chain. Such a story would be invisible to any transaction
due to already existing mechanisms, that's what is needed.

In order to provide "low enough" del_psn it will be natural to
assign real PSN starting from some predefined value, so any value
below that predefined value will be less that any existing PSN and
thus "low enough".

Implement this more simple approach.

Part of #8648
Part of #8654

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

That's strange, but in this test in a group of simple test cases
there are test cases that checks replaces, updates and deletes,
but occasionally there's no test case that checks inserts.

Fix it and add simple test cases for inserts.

No logical changes.

Part of #8648
Part of #8654

NO_DOC=new test case
NO_CHANGELOG=new test case

It is useful for options like `log_level` (with values like `5` or
`'info'`) or `synchro_quorum` (with values like `5` or `'N / 2 + 1'`).

This is a temporary solution. I want to introduce an explicit union
schema node later.

Fixes #8725

NO_DOC=the module is for internal use from the config code
NO_CHANGELOG=see NO_DOC

This is a shortcut for an array of unique string values from the given
list of allowed values. Useful for schema nodes declarations that
describes permissions such as {'read'}, {'read', 'write', 'execute'}
and so on.

Part of #8725

NO_DOC=the module is for internal use from the config code
NO_CHANGELOG=see NO_DOC

This is a shortcut for a string scalar with the given allowed values.
Suitable for declarations of options like `wal_mode`: `none`, `write`,
`fsync`.

Part of #8725

NO_DOC=the module is for internal use from the config code
NO_CHANGELOG=see NO_DOC

The function interprets typeless data from an environment variable
as a value of the given type. Raises an error if it is not possible.

Part of #8725

NO_DOC=the module is for internal use from the config code
NO_CHANGELOG=see NO_DOC

The method allows to walk over the schema and process scalar, map and
array nodes somehow.

Part of #8725

NO_DOC=the module is for internal use from the config code
NO_CHANGELOG=see NO_DOC

This method is useful to merge several configuration data from different
sources. For example, a config formed from environment variables, a
config read from a local file, a config from etcd.

Part of #8725

NO_DOC=the module is for internal use from the config code
NO_CHANGELOG=see NO_DOC

This method adds default values from the `default` annotation for missed
fields. The annotation is supported for scalar values. The default can
be conditional, it is controlled by the `apply_default_if` annotation.

Part of #8725

NO_DOC=the module is for internal use from the config code
NO_CHANGELOG=see NO_DOC

The method allows to perform hierarchical data transformations using
schema information. For example, apply a function to all string scalars.
Or replace all missed values with their defaults from the schema.

Part of #8725

NO_DOC=the module is for internal use from the config code
NO_CHANGELOG=see NO_DOC

It allows to walk over the hierarchical data and process it with
knowledge about corresponding schema nodes.

In conjunction with user-provided annotations (stored in the schema
nodes) it offers a powerful way to slice the data in different ways.

Part of #8725

NO_DOC=the module is for internal use from the config code
NO_CHANGELOG=see NO_DOC

It allows to construct a hierarchical data step-by-step or modify it
easily. Just like a field assignment operation for a flat data.

The function validates the value that is to be assigned, so data
constructed using a sequence of `:set()` calls is valid.

It performs the same path validation as `<schema object>:get()` to prevent
indexing of a scalar value or assigning an unknown record's field.

Part of #8725

NO_DOC=the module is for internal use from the config code
NO_CHANGELOG=see NO_DOC

It allows to acquire a nested data with several convenience properties.

* Optional chaining semantic: `<schema object>:get(data, 'foo.bar')`
  works similarly to TypeScript's `data?.foo?.bar` operator.
* Path validation against the schema: it prevents attempt to get an
  unknown field and attempt to index a scalar value.

Part of #8725

NO_DOC=the module is for internal use from the config code
NO_CHANGELOG=see NO_DOC

It allows to add a specific validator for a particular schema node.

Part of #8725

NO_DOC=the module is for internal use from the config code
NO_CHANGELOG=see NO_DOC