Previously, reusable tuple formats were only used for ephemeral spaces and
by `net.box` (which only used 'name' definitions), so non-space tuple
format definitions were ignored in tuple format comparison and hash
functions, but in scope of #4693 reusable tuple formats will be
interchangeable with space formats, so now we need to account for these
too.

Needed for #4693

NO_CHANGELOG=internal
NO_DOC=internal

In scope of #4693 we need to save the MsgPack encoding of the original
(i.e, user-provided) format clause in the tuple format structure for
serialization to Lua and IPROTO (#8147, #8633): since tuple formats for
spaces are created from space definitions, we need to also save the MsgPack
encoding in the latter.

Since we need to pass the format clause MsgPack encoding to the runtime
tuple format constructor, refactor it to perform field definition decoding
in-place. We cannot use default field definition array decoding for
external formats (67578d1f), so introduce a new field name decoder for this
case.

Needed for #4693

NO_CHANGELOG=refactoring
NO_DOC=refactoring
NO_TEST=refactoring

userdata is the preferred type for Lua objects wrapping C structures, but
tuple formats were made cdata for no good reason: change them to userdata.

Needed for #4693

NO_CHANGELOG=refactoring
NO_DOC=refactoring
NO_TEST=refactoring

Since we are going to make public Lua tuple format objects, introduce a new
`tuple_format` submodule and move the code related to these objects there.

Needed for #4693

NO_DOC=refactoring
NO_CHANGELOG=refactoring
NO_TEST=refactoring

Test runs an external process with tarantool that writes to a log file.
Then test reads that log file and searches a string with required
message in it (see function check_err_msg). Test was flaky on macOS and
I suspect it was happening due to a high log level - timeout was not
enough to wait message in the log file.

Patch decreases a log level to a default value and replaces io
functions with the similar alternatives in a fio module. Using
fio functions allows to not block fibers.

NO_CHANGELOG=test fix
NO_DOC=test fix

The xlog reader Lua module uses the xlog_cursor_next_row, which decodes
the row header with xrow_header_decode. The latter silently ignores any
unknown fields, which complicates catching bugs when garbage is written
to a row header by mistake, for example, see #8783.

Let's parse a row header without using xrow_header_decode in the xlog
reader module, like we parse a row body, and output all unknown/invalid
keys as is.

To do that, we have to extend the xlog cursor API with the new method
xlog_cursor_next_row_raw that returns a pointer to the position in the
tx buffer where the next xrow is stored without advancing it. To avoid
a memory leak in case the caller fails to parse an xrow returned by this
function, we also have to move the call to xlog_tx_cursor_destroy from
xlog_tx_cursor_next_row to xlog_cursor_next_tx.

While we are at it,
 - Don't raise an error if a key type encountered in a row body is
   invalid (not an integer). Instead, silently ignore such keys.
 - Remove the useless body MsgPack validness check because we already
   check it after decoding the header.
 - Add error injection based tests to check all the corner cases.

NO_DOC=bug fix

To avoid garbage written to xlog.

Closes #8783

NO_DOC=bug fix
NO_TEST=next commit

These are misc tests that can be related to the issue. Not sure all of
them do not work before the patch set for the issue. It is nice to have
them and be sure everything works fine.

Closes #8658

NO_DOC=minor

We can replace the last `xrow_update_err_double` occurrence with
`xrow_update_err_no_such_field`. The last is correct one for this place
as `xrow_update_op_do_field_##op_type` is called only for non terminal
paths. Thus if XUPDATE_SCALAR is encountered then the field referenced
does not exist.

Part of #8658

NO_DOC=minor
NO_CHANGELOG=minor

This will help to do further updates into into/inside affected field.
It does not make much difference from the performance POV but allows to
reuse existing code.

Part of #8658

NO_CHANGELOG=later
NO_DOC=minor

This way after branching the operation will be presented by XUPDATE_SCALAR
in the parent array/map so that we can apply multiple operations to the
same field.

Part of #8658

NO_CHANGELOG=later
NO_DOC=minor

Current representation allows only single splice. Rope in turn allows
for arbitrary number of updates and have good asymptotic. Using only
rope gives 4% degradation of single update of the string field (full
time of update operation is measured). Thus rope is started to use
from the second update into the same string field.

Part of #8658

NO_DOC=internal
NO_CHANGELOG=internal

Now when groundwork is done it is time to unblock multiple updates to
the same field.

Note that along the way we fix one more case. For example 'bar' update
on path '[2].a.b.c.d' and then setting on the path '[2].a.b.c'.
Currently this leads to cryptic 'can not update map by non-string key'
error. The issue is if new path is shorter than old path of the bar
update which is to be branched then we exit path comparison loop when
new_toke.type == JSON_TOKEN_END.

Note that current code for handling double update of the same field for
the bar case is not optimal. It will produce route node (optional),
array or map node and node for field itself in the update tree. Instead
we can have just same bar node which apply 2 scalar updates like in case
of terminal update in map or array. This optimization is work to be
done.

Part of #8658

NO_DOC=minor
NO_CHANGELOG=later

Now result is stored in operation itself. It is inconvenient if we are
going to apply multiple operations on the same field. To keep the said
result let's introduce scalar type that can hold all possible scalar
including strings.

By the way it is good time to get rid of `new_field_len` field. It is
excessive now.

Part of #8658

NO_TEST=refactoring
NO_DOC=refactoring
NO_CHANGELOG=refactoring

It is actually not a scalar as we can set arrays/maps which further
holds arrays/maps and so on and so on. Let's make it XUPDATE_NOP
instead. XUPDATE_NOP is basically a state of array/map item without
changes so we can easily make further field updates after '=' update.

For this purpose we need to untie tail data from updated field data
by introducing tail data pointer.

We can do updates to/into newly inserted field before this patch. Yet let's
require it explicitly here by tests for insert operation.

Part of #8658

NO_DOC=minor
NO_CHANGELOG=later

Panic if we fail to allocate internal temporary objects on region. We do
not test allocation failures and this should normally happen also
 (see #3534).

Part of #8658

NO_DOC=code cleanup
NO_TEST=code cleanup
NO_CHANGELOG=code cleanup

Apple's libunwind for AArch64 returns the Instruction Pointer with the
Pointer Authentication Codes (bits 47-63) even though Tarantool is compiled
for arm64 (not arm64e) architecture, so we have to strip them out [1].
Although there is the ptrauth_strip macro for this purpose, it works only
if compilation target is arm64e (not arm64) [2].

1. https://developer.apple.com/documentation/security/preparing_your_app_to_work_with_pointer_authentication#3042105
2. https://github.com/dotnet/runtime/issues/42955#issuecomment-886910180

Closes #8074
Closes tarantool/tarantool-qa#308
Closes tarantool/tarantool-qa#309

NO_DOC=bugfix

Implementation notes:
 - The varbinary type is implemented as VLS cdata so we can't use
   the existing luaL_pushcdata and luaL_checkcdatas helpers for
   pushing an object of this type to Lua stack. Instead, we copied
   the implementation from the Lua JIT internals.
 - We already have the code handling `MP_BIN` fields in all built-in
   serializers. We just need to patch it to convert the data to/from
   a varbinary object instead of a plain string.
 - We updated the tuple.tostring method to set the NOWRAP base64
   encoder flag when dumping binary blobs. The flag was apparently
   omitted by mistake because we mask all other new line characters
   while converting a tuple to a string.
 - The box/varbinary_type test was rewritten using the luatest
   framework with all the FFI code needed to insert binary data
   replaced with the new varbinary object.
 - We have to update quite a few SQL tests involving varbinary type
   because binary blobs are now returned as varbinary objects, not
   as plain strings, as they used to be.

Closes #1629

@TarantoolBot document
Title: Document the varbinary type

The new module `varbinary` was introduced. The module implements the
following functions:
 - `varbinary.new` - constructs a varbinary object from a plain string
   or cdata pointer and size (to be used with the `buffer` module).
 - `varbinary.is` - returns true if the argument is a varbinary object.

```Lua
local bin = varbinary.new('data')
assert(varbinary.is(bin))
assert(not varbinary.is('data'))
```

Like a plain string, a varbinary object stores arbitrary data. Unlike
a plain string, it's encoded as a binary blob by the built-in encoders
that support the binary type (MsgPack, YAML). (Actually, encoding binary
blobs with the proper type is the main goal of the new type.)

```
tarantool> '\xFF\xFE'
---
- "\xFF\xFE"
...

tarantool> varbinary.new('\xFF\xFE')
---
- !!binary //4=
...

tarantool> msgpack.encode('\xFF\xFE')
---
- "\xA2\xFF\xFE"
...

tarantool> msgpack.encode(varbinary.new('\xFF\xFE'))
---
- "\xC4\x02\xFF\xFE"
...
```

Note, the JSON format doesn't support the binary type so a varbinary
object is still encoded as a plain string:

```
tarantool> json.encode('\xFF\xFE')
---
- "\"\xFF\xFE\""
...

tarantool> json.encode(varbinary.new('\xFF\xFE'))
---
- "\"\xFF\xFE\""
...
```

The built-in decoders now decode binary data fields (fields with the
'binary' tag in YAML; the `MP_BIN` type in MsgPack) to a varbinary
object by default:

```
tarantool> varbinary.is(msgpack.decode('\xC4\x02\xFF\xFE'))
---
- true
...

tarantool> varbinary.is(yaml.decode('!!binary //4='))
---
- true
...
```

This also implies that the data stored in the database under the
'varbinary' field type is now returned to Lua not as a plain string,
but as a varbinary object. It's possible to revert to the old behavior
by toggling the new compat option `binary_data_decoding` because this
change may break backward compatibility:

```
tarantool> compat.binary_data_decoding = 'old'
---
...

tarantool> varbinary.is(msgpack.decode('\xC4\x02\xFF\xFE'))
---
- false
...

tarantool> varbinary.is(yaml.decode('!!binary //4='))
---
- false
...
```

Please create a documentation page for the new compat option:
https://tarantool.io/compat/binary_data_decoding

A varbinary object implements the following meta-methods:
- `__len` - returns the length of the binary data, in bytes.
- `__tostring` - returns the data in a plain string.
- `__eq` - returns true if the varbinary object contains
  the same data as another varbinary object or a string.

```Lua
local bin = varbinary.new('foo')
assert(#bin == 3)
assert(tostring(bin) == 'foo')
assert(bin == 'foo')
assert(bin ~= 'bar')
assert(bin == varbinary.new('foo'))
assert(bin ~= varbinary.new('bar'))
```

There are no string manipulation methods, like `string.sub` or
`string.match`. If you need to match a substring in a varbinary
object, you have to convert it to a string first.

For more details, see the [design document][1].

[1]: https://www.notion.so/tarantool/varbinary-in-Lua-a0ce453dcf5a46e3bc421bf80d4cc276

Let's drop yaml/b64 in favor of the base64 encoder used everywhere else
in the Tarantool source code.

yaml/b64 is also used by serialize_lua to print MP_BIN values. Let's
print MP_BIN values as MP_STR there. This doesn't have any user-visible
changes because since commit 890a821c ("yaml: don't encode
unprintable strings as binary blobs") luaL_tofield never creates MP_BIN
values. However, when we introduce the varbinary type to Lua, we will
use the MP_BIN value type for it, and printing it in the Lua format as a
string with unprintable characters escaped is going to be less confusing
than encoding it in base64 without any tags or markers.

While we're at it, let's use the luaL_field.sval.data in the encoders
instead of extracting the string from the Lua stack again.

Needed for #1629

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

There's base64_bufsize for calculating the buffer size needed for
base64_encode but there's no such function for base64_decode - one's
supposed to pass a buffer > 3/4 of the input size. This is confusing.
Let's rename base64_bufsize to base64_encode_bufsize and introduce
base64_decode_bufsize.

While we're at it, move base64_encode_bufsize body from base64.h to
base64.c because otherwise the linker fails if this function, which is
currently declared as extern inline, is used in lyaml.cc.

Needed for #1629

NO_DOC=refactoring
NO_CHANGELOG=refactoring

This patch allows additional config features to be added to EE.

Closes #8789

NO_DOC=will be added later
NO_TEST=will be added in EE code
NO_CHANGELOG=will be added later

This patch integrates config module with the Tarantool --config option.

Part of #8789

NO_DOC=will be added later
NO_CHANGELOG=will be added later

Our macOS runners have such a TMPDIR value. It breaks
`config-luatest/basic_test.lua`, because net.box seems unable to connect
a Unix domain socket using an URI with a double slash in the middle.

See the comment in the code for details.

NO_DOC=testing helper change
NO_CHANGELOG=see NO_DOC
NO_TEST=see NO_DOC

This patch introduces a config test helper that simplifies the creation
of a replica set.

NO_DOC=testing helper
NO_CHANGELOG=see NO_DOC

NO_DOC=testing helper
NO_CHANGELOG=see NO_DOC

It makes the helper a bit more convenient to use it tests.

NO_DOC=testing helper
NO_TEST=see NO_DOC
NO_CHANGELOG=see NO_DOC

This patch implements broadcastion of the current statusi.

NO_DOC=will be added later
NO_CHANGELOG=will be added later

This patch introduces the "config" module. This module allows you to
access the configuration, apply it, and reload.

Part of #8789

NO_DOC=will be added later
NO_CHANGELOG=will be added later

This patch introduces the env configuration source. This is the instance
source that can be used to describe configuration options for a single
instance.

Part of #8789

NO_DOC=will be added later
NO_CHANGELOG=will be added later

This patch introduces the file configuration source. This is the cluster
source that can be used to describe all configuration options for
cluster instances.

Part of #8789

NO_DOC=will be added later
NO_CHANGELOG=will be added later

This patch introduces the application applier. This applier is used to
set user application options.

Part of #8789

NO_DOC=will be added later
NO_CHANGELOG=will be added later

This patch introduces the fiber applier. This applier is used to set
fiber options.

Part of #8789

NO_DOC=will be added later
NO_CHANGELOG=will be added later

This patch introduces the console applier. This applier is used to set
console options.

Part of #8789

NO_DOC=will be added later
NO_CHANGELOG=will be added later

This patch introduces the credentials applier. This applier is used to
create roles and users, set roles to users and roles, set password to
users, set any privileges for users and roles for universe.

Part of #8789

NO_DOC=will be added later
NO_CHANGELOG=will be added later

This patch introduces the box_cfg applier. This applier is used to
process box options. This appliers is called after mkdir applier because
we need all necessary directories to exist by the time the box options
are processed.

Part of #8789

NO_DOC=will be added later
NO_CHANGELOG=will be added later

This patch introduces the mkdir applier. This appliers is used to create
all required directories. This applier is called before all other
appliers.

Part of #8789

NO_DOC=will be added later
NO_CHANGELOG=will be added later

This patch introduces the configdata module, which prepares data from
configuration sources before it is provided to appliers.

Part of #8789

NO_DOC=will be added later
NO_CHANGELOG=will be added later

<luaT_pushtuple> function created a new GCfunc object for a tuple __gc
handler (i.e. <lbox_tuple_gc>), having no upvalues, on each call when
tuple object for Lua world is created.

The change introduces the reference to the single GCfunc object created
on Tarantool startup and stored to Lua registry. Using this approach in
scope of <luaT_pushtuple> is aimed to reduce Lua GC memory usage.

Part of #5201

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Reviewed-by: Vladimir Davydov <vdavydov@tarantool.org>
Reviewed-by: Sergey Kaplun <skaplun@tarantool.org>
Signed-off-by: Igor Munkin <imun@tarantool.org>

We are going to introduce universal triggers registry and use it for
tarantool triggers (e.g. space.on_replace). However, when trigger is set
to the registry, we do not know which arguments will be passed there
from C. That is why current implementation of triggers in tarantool is not
suitable for use in universal registry.

We decided to introduce func_adapter interface which allows to call
functions from any programming language. Now we support only one
language, which is Lua, so for the most part this interface is needed
to invert dependencies.

The patch introduces func_adapter interface and its implementation for
Lua functions.

Part of #8657

NO_CHANGELOG=internal
NO_DOC=internal

`cord_slab_cache` was not designed to offer a backward compatible API, and
leaving it exposed inevitably leads to errors like those highlighted in
#7124 and tarantool/memcached#96: hence, remove it from the public API
export.

Closes #7124

@TarantoolBot document
Title: `cord_slab_cache` was removed from public API export

The `cord_slab_cache` needs to be removed from the C API reference of the
fiber module.