A user is unlikely interesting in intermediate changes. Let's just say
that the new declarative configuration is supported now.

NO_DOC=changelog changes
NO_TEST=see NO_DOC

There may be some confusion, so let's start with a background
information.

There are `TT_*` environment variables introduced in commit 1b330121
("box: set box.cfg options via environment variables"). They're
interpreted by the `box.cfg()` call.

There are `TT_*` environment variables introduced in commit 82b0cff9
("config: introduce env source"). They're interpreted by the declarative
configuration logic, when tarantool starts with the `--name <...>` CLI
option.

box.cfg's env variables have names deduced from box.cfg option names,
while config's env variable names are deduced from the config schema.

Some options have the same names here and there, for example
`TT_REPLICATION_ANON` (from `box.cfg.replication_anon` and
`replication.anon`). However, there are ones that have different names,
for example `TT_LISTEN` and `TT_IPROTO_LISTEN`.

Moreover, the declarative configuration has its own restrictions on the
configuration data. For example, `TT_IPROTO_LISTEN` is always a list of
URIs (like `[{"uri": <...>, "params": {<...>}}]`), not a single URI, not
a string, not a number. The declarative configuration has a certain
shape and doesn't allow polymorphic values.

Next, handling of box.cfg's variables by the old code in `load_cfg.lua`
doesn't work well with the declarative configuration flow.

The main reason is that the new configuration flow calls `box.cfg()`
with all the `box.cfg` values set, including default ones. If a user
removes an option from its config, it applies its default. On the same
time it instructs `box.cfg()` to don't read the corresponding
environment variables.

This commit offers a partial solution: it adds support of the most of
the box.cfg environment variables. The values are added into the
configuration data with the lowest priority: if the same value is set
in, for example, a file configuration, the file's value is preferred.

The following box.cfg's environment variables are not handled in this
commit.

* `TT_LOG`
* `TT_METRICS`
* `TT_INSTANCE_NAME`
* `TT_REPLICASET_NAME`
* `TT_CLUSTER_NAME`
* `TT_FORCE_RECOVERY`,
* `TT_READ_ONLY`
* `TT_BOOTSTRAP_LEADER`
* `TT_REPLICATION`
* `TT_REPLICATION_CONNECT_QUORUM`

Fixes #9485

NO_DOC=looks more like a bug fix or a kind of compatibility layer

This function makes it easier to run a code that can't be run directly
for some reason: for example, it needs the initialized database.

It is a wrapper around treegen and justrun.

Part of #9485

NO_DOC=testing helper change
NO_CHANGELOG=see NO_DOC
NO_TEST=see NO_DOC

`TT_LISTEN` and `TT_REPLICATION` environment variables were interpreted
by `box.cfg()` in a confusing way if query parameters with values are
present. For example, `localhost:3301?transport=plain` was interpreted
as the following map: `{['localhost:3301?transport'] = 'plain'}`. Later,
`box.cfg()` looks into this map for known URI fields like `login`,
`password`, `uri`, `host`, `service` and so on. It found nothing and
doesn't start a listening socket.

The reason of such a behaviour is that the environment value is
interpreted as a mapping in the `key=value,key=value` format, because
there is `=` in it.

The patch changes this behavior for an `key=value,key=value` environment
variable that contains `?` in a key: now such a value is not interpreted
as a mapping.

Note: Everything said above is also applicable to the so called
multilisten case: when several URIs are defined in the environment
variable. The following URI list is interpreted correctly now.

NOWRAP
```sh
export TT_LISTEN=localhost:3301?transport=plain,localhost:3302?transport=plain
```
NOWRAP

Note 2: Examples are given with the `plain` transport, which is default,
but the query parameters are the way to define TLS options. They're
supported in Tarantool Enterprise Edition, see [1].

Fixes #9539

NO_DOC=bugfix

[1]: https://www.tarantool.io/en/doc/latest/enterprise/security/#traffic-encryption

`httpc` module has two GC-finalizers: the first one for a Lua http
client (C function `luaT_httpc_cleanup`) and the second one for a Lua
http chunked requests (C function `luaT_httpc_io_cleanup`) introduced in
commit 417c6cb7 ("httpc: introduce stream input/output interface").
In a C implementation HTTP requests depends on structures of HTTP client
and there is a problem with destroying Lua objects in `httpc` module -
these GC-finalizers are not synchronized. This could lead to at least
two problems:

There is a race with GC-finalization that leads to use-after-free errors
when HTTP client is collected before collecting HTTP request. In a
stacktrace the problem looks as below:

```
0x55ca7d47652e in crash_collect+256
0x55ca7d476f6a in crash_signal_cb+100
0x7fb876c42520 in __sigaction+80
0x55ca7d641e51 in curl_slist_free_all+35
0x55ca7d441498 in httpc_request_delete+45
0x55ca7d4653f1 in httpc_io_destroy+27
0x55ca7d4674bc in luaT_httpc_io_cleanup+36
0x55ca7d4e00c7 in lj_BC_FUNCC+70
0x55ca7d4f8364 in gc_call_finalizer+668
0x55ca7d4f8946 in gc_finalize+1387
0x55ca7d4f91e2 in gc_onestep+864
0x55ca7d4f9716 in lj_gc_fullgc+276
...
```

Lua object `http.client` could be GC-collected when chunked HTTP request
is alive. This will lead to an error "IllegalParams: io: request must be
io" because we call a method when Lua object is already a `nil`.

```lua
local url = 'https://bronevichok.ru/'
local c = require('http.client').new()
local r = c:get(url, {chunked = true})
c = nil
collectgarbage()
collectgarbage()
r:read(1) -- IllegalParams: io: request must be io
```

The patch introduces two functions: `httpc_env_finish` and
`curl_env_finish`, that prepares curl and httpc environments for
destruction. HTTP client's GC finalizer now calls `httpc_env_finish`
instead of `httpc_env_destroy`, this prevents from destroying memory
that could be in use by HTTP requests. Additionally `httpc_env_finish`
sets a flag `cleanup`. HTTP environment destroying is called when flag
`cleanup` is set and a there are no active HTTP requests. The main idea
of the patch is a synchronization of destructors for HTTP client and
HTTP chunked requests. Unfortunately, GC will eventually collect HTTP
client object after calling its `__gc`. To prevent this we put a
reference to a Curl's userdata in Lua objects with HTTP chunked requests
and HTTP default client.

Fixes #9346
Fixes #9453

NO_DOC=bugfix

Bump curl version to 8.4.0 triggers a crash in Tarantool due to commit
"h2: testcase and fix for pausing h2 streams" [1]. The original
reproducer involves etcd and an etcd-client Lua module, running
etcd-client tests as a part of Tarantool integration testing is planned
to do in scope of [1].

However, the problem could be reproduced with a Lua code below:

```
local url = 'https://google.com/'

local c = require('http.client').new()

r1 = c:get(url, {chunked = true})
r1:read(1)
r2 = c:get(url, {chunked = true})
r2:read(1)
r3 = c:get(url, {chunked = true})
r3:read(1)
r4 = c:get(url, {chunked = true})
r4:read(1)

c = nil
collectgarbage()
collectgarbage()

r1:read(1)
r2:read(1)
r3:read(1)
r4:read(1)

collectgarbage()
collectgarbage()
```

According to Curl documentation, `curl_multi_cleanup` [1] must be called
before any easy handles are cleaned up. The patch adds a cleanup of easy
handles on running `curl_env_destroy`, right before calling
`curl_multi_cleanup`. The patch uses a function 'curl_multi_get_handles'
that returns all added easy handles introduced in Curl 8.4.0. Therefore
bump to 8.4.0 is required.

1. https://github.com/curl/curl/commit/6b9a591bf7d82031f463373706d7de1cba0adee6
2. https://curl.se/libcurl/c/curl_multi_cleanup.html

Fixes #9283

1. https://github.com/tarantool/tarantool/issues/9093

NO_DOC=bugfix
NO_TEST=no simple reproducer, covered by tests in etcd-client

FreeBSD instances in Tarantool CI have installed libcurl package (as a
dependency of Zabbix monitoring agent). Curl 8.4.0 introduces a new
function `curl_multi_get_handles` that is used in the following commit
in `src/curl.c`, but libcurl system package has no such symbol in
headers. On building on FreeBSD in Tarantool CI C compiler produces a
warning about implicit declaration of function, because it looks at
system headers by default and due to enabled CMake option
`-DENABLE_WERROR=ON` building has failed:

```
[ 63%] Building C object src/CMakeFiles/server.dir/title.c.o
/.cache/act/55d136250dd94303/hostexecutor/src/curl.c:266:17: error: implicit declaration of function 'curl_multi_get_handles' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
                CURL **list = curl_multi_get_handles(env->multi);
                              ^
/.cache/act/55d136250dd94303/hostexecutor/src/curl.c:266:17: note: did you mean 'curl_multi_add_handle'?
/usr/local/include/curl/multi.h:140:23: note: 'curl_multi_add_handle' declared here
CURL_EXTERN CURLMcode curl_multi_add_handle(CURLM *multi_handle,
                      ^
/.cache/act/55d136250dd94303/hostexecutor/src/curl.c:266:10: error: incompatible integer to pointer conversion initializing 'CURL **' (aka 'void **') with an expression of type 'int' [-Werror,-Wint-conversion]
                CURL **list = curl_multi_get_handles(env->multi);
                       ^      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2 errors generated.
```

The patch fixes that by reordering headers passed to compiler, see [1].

1. https://cmake.org/cmake/help/latest/command/include_directories.html

Needed for #9283

NO_CHANGELOG=build
NO_DOC=build
NO_TEST=build

The patch updates curl module to the version 8.4.0 [1] that brings a
number of functional fixes and security fix of SOCKS5 heap buffer
overflow (CVE-2023-38545), see description in [2] and commit
fb4415d8aee6 ("socks: return error if hostname too long for remote
resolve") in [3].

1. https://curl.se/changes.html#8_4_0
2. https://curl.se/docs/CVE-2023-38545.html
3. https://github.com/curl/curl/commit/fb4415d8aee6c1045be932a34fe6107c2f5ed147

NO_DOC=libcurl submodule bump
NO_TEST=libcurl submodule bump

Message below is printed every time on shutdown `httpd.py` when
`test/app-luatest/http_client_test.lua` is running by luatest without
capturing stdout:

```
BrokenPipeError: [Errno 32] Broken pipe exception handling
```

The patch suppress this exception by adding a handler for a signal
`SIGPIPE`.

NO_CHANGELOG=testing
NO_DOC=testing
NO_TEST=testing

NO_CHANGELOG=fixed typos
NO_DOC=fixed typos
NO_TEST=fixed typos

The patch propagates debug mode to building of third party components:
c-ares, libcurl, libeio, nghttp2, zstd. Other components enables debug
mode automatically once it is enabled in Tarantool build.

Curl has two similar options that enables debug mode, however they are
different: `ENABLE_CURLDEBUG` enable memory debugging and `ENABLE_DEBUG`
restricts code which is only compiled for debug enabled builds [1].

1. https://everything.curl.dev/internals/memory-debugging

NO_CHANGELOG=build
NO_DOC=build
NO_TEST=build

Fixed a bug when the assertion in box_promote_qsync would fail.
The assertion is that at the moment when box_promote_qsync is
executed, no other promote is executed. It turned out that this
assertion is basically incorrect. Now after this patch the newly
elected leader is trying to repeat box_promote_qsync in
box_raft_update_synchro_queue until it fails due to the fact
that some other promotion is currently being executed.

Closes #9263

NO_DOC=bugfix

This commit changes the order of configuration appliers and move
`console` before `box_cfg`. The `console` applier is adjusted to
correctly interpret the configured socket file path before the first
box.cfg().

The main reason to change the order is to allow a user to call
`box.ctl.make_bootstrap_leader()` for a replicaset that starts in the
`replication.boostrap_strategy: supervised` mode.

Part of #8862

NO_DOC=The documentation already assumes that a control command may be
       issued on the console. No changes are required.

It is important to listen on a Unix domain socket, because its path
length is limited to 107 on Linux and 103 on Mac OS (in our
implementation -- in theory could be 108 and 104).

Part of #8862

NO_DOC=no public API changes
NO_CHANGELOG=see NO_DOC
NO_TEST=negligible change

These functions are to be re-used in the console applier code in the
next commit. Exporting them allows to eliminate a duplication of this
code.

Part of #8862

NO_DOC=no public API changes
NO_CHANGELOG=see NO_DOC
NO_TEST=no new behavior, just refactoring

It is needed to verify the `supervised` bootstrap strategy. We should
start the instances, issue `box.ctl.make_bootstrap_leader` and only than
wait for `config:info().status == 'ready'`.

Let's support this flow in the replicaset management helper.

Part of #8862

NO_DOC=testing helper change
NO_CHANGELOG=see NO_DOC

Sometimes a testing code needs to interpret parts of the configuration
and it is convenient to look at the global scope only without diving
into group, replicaset, instance options.

Let's allow to set a global scope option in the configuration builder
helper.

Part of #8862

NO_DOC=testing helper change
NO_CHANGELOG=see NO_DOC

Before this patch a replica adds an alert regarding inability to write
new credentials from config on startup. In most cases these new
credentials are applied on a master and the warning becomes obsolete.
The problem is that it is not removed at this point. A manual
`config:reload()` is needed to flush it.

It makes the warning more confusing than helpful. Let's remove it for
now.

We can return the alert back in a future, but we should do it together
with a logic that drops the alert if the needed data is received from a
master.

Part of #8862

NO_DOC=bugfix

This patch sets requirements for vshard version that is supported by
config module.

Part of #8862

NO_TEST=internal

@TarantoolBot document
Title: config: supported vshard version

The vshard version supported by the config module cannot be less than
0.1.25. Note, that vshard will only be loaded if any of instances have
sharding storage role or sharding router role.

* FFI: Fix dangling reference to CType in carith_checkarg().
* FFI: Fix dangling reference to CType. Improve checks.
* FFI: Fix dangling reference to CType.
* FFI: Ensure returned string is alive in ffi.typeinfo().
* FFI: Fix missing cts->L initialization in argv2ctype().
* Abstract out on-demand loading of FFI library.
* test: fix flaky finalizer error handler tests
* test: adjust lua-Harness test error assertion
* Fix snapshot PC when linking to BC_JLOOP that was a BC_RET*.
* snap: check J->pc is within its proto bytecode
* Fix HREFK forwarding vs. table.clear().
* Fix FOLD rule for BUFHDR append.
* Prevent CSE of a REF_BASE operand across IR_RETF.
* test: rewrite sysprof test using managed execution
* test: disable buffering for the C test engine

Part of #9145

NO_DOC=LuaJIT submodule bump
NO_TEST=LuaJIT submodule bump

To fix #9507 before release. We need more time to prepare a good patch
dealing with the issue.

This reverts commit 26acba83.

NO_TEST=fixing ASAN run of existing test
NO_CHANGELOG=issue is only in unreleased version
NO_DOC=bugfix

A recent TLS options support patchset (PR #9385) has changed a way to
work with URIs: they're now tables, not strings. So, a modification may
influence the original data.

Prevent it by copying a table before a modification.

Part of #8862

NO_DOC=bugfix
NO_CHANGELOG=the fixed code was not released yet
NO_TEST=no known failure scenarios

I accidentally discarded some changes made by @ImeevMA in PR #9385. The
previous version of the patchset goes to the main branch as result. I'm
very sorry.

This commit returns the code of the last version of the patchset.

The changes are described in [1]. Quoted here:

> 1. Reduce code duplication in the validation logic. It also
>    eliminates remaining (dead) code that interprets user@ and
>    user:pass@ syntax. And give better error messages in come cases
>    (comma separates URIs for example).
> 2. Forbid {} as an URI to simplify logic in instance_uri.
> 3. Refresh the comments.

[1]: https://github.com/tarantool/tarantool/pull/9385#discussion_r1430935585

Part of #8862

NO_DOC=fixup
NO_CHANGELOG=see NO_DOC

This patch adds missing field in instance_config_schema_test.lua.

Follow-up #9506

NO_DOC=fix for test
NO_CHANGELOG=fix for test

This patch updates example for vshard usage in config module.

NO_DOC=example update
NO_TEST=example update
NO_CHANGELOG=example update

Fixes #9506

@TarantoolBot document
Title: config.context: define cluster config parts in env/file

Example:

```yaml
config:
  context:
    replicator_password:
      from: file
      file: secrets/replicator_password.txt
      rstrip: true
    client_password:
      from: file
      file: secrets/client_password.txt
      rstrip: true

credentials:
  users:
    replicator:
      password: '{{ context.replicator_password }}'
      roles: [replication]
    client:
      password: '{{ context.client_password }}'
      roles: [super]
```

The new `config.context` section allows a user to define its own
variables, whose content resides in an environment variable or a file.

The primary usage is to extract passwords from the configuration.

All the variables are defined in `config.context` section as keys. Each
key is a record with the following fields.

* `from`: either `env` or `file`
* `file`: a file path
* `env`: an environment variable name
* `rstrip`: whether to strip whitespace characters from the end of the
   data (it strips newlines too)

A relative file path is interpreted as relative to `process.work_dir`.

`from` is mandatory, `file` is mandatory when `from` is `file`, `env` is
mandatory when `from` is `env`, `rstrip` is optional.

If there is no given file or no given environment variable, an error of
a configuration applying is reported.

The plan is to re-use it in a next commit.

Part of #9506

NO_DOC=no public API changes
NO_CHANGELOG=see NO_DOC
NO_TEST=see NO_DOC

NO_DOC=testing helper change
NO_CHANGELOG=see NO_DOC
NO_TEST=see NO_DOC

This patch enables vshard manual schema management mode.

Part of #8862

NO_DOC=internal
NO_CHANGELOG=internal

This patch introduces the sharding user role. This role must be set to
the vshard storage user, if user is in 'credentials.user'.

Part of #8862

@TarantoolBot document
Title: config: credential `sharding` role

The credential `sharding` role is a new credential role. This is the
default credential role, but it is different from other default
credential roles because it is created by the config module. Other
default credential roles are already described in the bootstrap.snap
file.

This role has different privileges depending on the replicaset sharding
role. For replicasets with the sharding `storage` role, the credential
`sharding` role will have rights to execute necessary `vshard.storage.*`
functions and the credential `replicaset` role. If the replicaset does
not have the sharding `storage` role, the credential `sharding` role
does not have any privileges.

A sharding storage user must have the credential `sharding` role among
their credential roles, if the user is in `credentials.users`. If the
user is not in `credentials.users` we do not check its privileges.

This patch adds missing tests to validate configs from doc/examples.

NO_DOC=tests for example configs
NO_CHANGELOG=tests for example configs

This patch moves example for config of config-storage to doc/examples
from enterprise doc/.

NO_DOC=added an example
NO_TEST=validation will be added later
NO_CHANGELOG=added an example

This patch fixes iproto.listen and iproto.advertise.peer in local.yaml.

Part of #8862

NO_DOC=changes in example
NO_TEST=will be tested in EE
NO_CHANGELOG=changes in example

When integrity protection is enabled, the Lua chunk given via -e option
is not run to avoid any effects from such a kind of platform behaviour
tweak. Fortunately, -e itself and its argument are stripped from the Lua
<arg> table, so the patch just omits the execution of the Lua chunk, and
there is no need for additional cleanup.

Part of tarantool/tarantool-ee#585

NO_DOC=will be added to Enterprise Edition
NO_TEST=will be added to Enterprise Edition
NO_CHANGELOG=will be added to Enterprise Edition

When integrity protection is enabled, the whole process environment is
purged to avoid any side effects from particular environment variables.
To totally collapse the effect of the environment, <clearenv> is invoked
at the beginning of Tarantool startup. Unfortunately, <proc_title_init>
helper relied on <environ> not being NULL, which led to the platform
crash during startup. The patch fixes this misbehaviour, either.

Part of tarantool/tarantool-ee#585

NO_DOC=will be added to Enterprise Edition
NO_TEST=will be added to Enterprise Edition
NO_CHANGELOG=will be added to Enterprise Edition

It turns out, GitHub actions don't allow `env` usage in their
definition. This patch fixes this issue in submodule bump action
by moving the environment definition into the executed shell
script.

NO_DOC=CI
NO_TEST=CI
NO_CHANGELOG=CI

Vshard now supports names instead of UUIDs in configuration. Let's
use them and finally drop generating UUIDs in config.

From now on we can recover instances without explicitly passing
UUIDs to configuration.

NO_DOC=<not released yet>
NO_CHANGELOG=<not released yet>

Co-authored-by: Mergen Imeev <imeevma@tarantool.org>

It fails in a quite stable manner after the previous commits. Let's
workaround it and fix it in the LuaJIT submodule later.

NO_DOC=no API changes
NO_CHANGELOG=no behavior changes
NO_TEST=nothing to test

This patch changes format of iproto.listen and introduces support for
TLS options in the iproto.listen config option.

Part of #8862

@TarantoolBot document
Title: new format of the `iproto.*` config options.

The config options `iproto.listen`, `iproto.advertise.peer` and
`iproto.advertise.sharding` now have a new format.

The `iproto.listen` config option now has the following format:
```
iproto:
  listen:
    - uri: <string>
      params:
        transport: <'plain' or 'ssl'>
        ssl_ca_file: <string>
        ssl_cert_file: <string>
        ssl_ciphers: <string>
        ssl_key_file: <string>
        ssl_password: <string>
        ssl_password_file: <string>
    - uri: <string>
      params:
        transport: <'plain' or 'ssl'>
        ssl_ca_file: <string>
        ssl_cert_file: <string>
        ssl_ciphers: <string>
        ssl_key_file: <string>
        ssl_password: <string>
        ssl_password_file: <string>
...
```
The `iproto.listen` config option is now an array, and its elements are
`records` that can only have a required `uri` field and an optional
`params` field.

The `uri` field is of type `string` and must be a single valid URI. The
URI cannot contain parameters, login and password.

The `params` field is a `record` with the fields `transport`,
`ssl_ca_file`, `ssl_cert_file`, `ssl_ciphers`, `ssl_key_file`,
`ssl_password`, `ssl_password_file`. The `transport` field can have one
of two values: `plain` or `ssl`, the default is `plain`. If `transport`
is set to `plain`, no additional fields are required. When `transport`
is set to `ssl`, the fields `ssl_key_file` and `ssl_cert_file` are
required, and the remaining fields are optional.

The `iproto.advertise.peer` and `iproto.advertise.sharding` now have the
following format:
```
iproto:
  advertise:
    <'peer' or 'sharding'>:
      uri: <string>
      login: <string>
      password: <string>
      params:
        transport: <'plain' or 'ssl'>
        ssl_ca_file: <string>
        ssl_cert_file: <string>
        ssl_ciphers: <string>
        ssl_key_file: <string>
        ssl_password: <string>
        ssl_password_file: <string>
```

The `iproto.advertise.peer` and `iproto.advertise.sharding` are now
`records`. All of their fields are optional, however, if the `params`
field can be set only if the `uri` field is set and the `password` field
can be set only if the `login` field is set.

The `uri` field is of type `string` and must be a single valid URI.
The URI cannot contain parameters, login and password. If the `uri` and
`params` fields are missing, the appropriate `iproto.listen` element is
selected. An element is eligible if, after parsing its `url` field using
`require('uri').parse()`, the `ipv4` result field is not equal to
`'0.0.0.0'`, the `ipv6` result field is not equal to `'::'`, and the
`service` result field is not equal to `'0'`.

The `login` and `password` fields are of type `string`.

For description of the `params` field see description of
`iproto.listen`.

This patch changes the structures of the iproto.advertise.peer and
iproto.advertise.sharding options and introduces TLS options for these
options.

There is no proper integration test for the new TLS options as our
iproto.listen does not currently support these options. The test will
be added along with the addition of TLS support in iproto.listen.

Part of #8862

NO_DOC=will be added later.