Currently, we create a database read view only to create a memtx
snapshot or join a replica, but there's already quite a bit of code
duplication between these two scenarios. In the future, we will need
the same functionality to create a user read view. So let's factor out
this code into a separate module - read_view.

The API of the read_view module is quite simple - there are just two
methods: open and close a read view. The user can pass a space and index
filter while opening a read view to skip certain spaces. E.g. we skip
all temporary spaces and secondary indexes when we create a memtx
snapshot. A read_view object has a list of space_read_view objects, one
per each space included into the read view. A space_read_view object, in
turn, has a map of all index_read_view objects (introduced earlier)
corresponding to space indexes. There's nothing like a space cache - the
user can create one if required.

An engine that supports creation of a read view (currently, only memtx)
is supposed to set the ENGINE_SUPPORTS_READ_VIEW flag and implement the
create_read_view engine method in addition to the create_read_view index
method. The engine method should do some engine-wide read view related
preparations. For example, in case of memtx, it suspends tuple garbage
collection.

Closes #7363

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

downstream lag is the difference in time between the moment a
transaction was written to master's WAL and the moment an ack for it
arrived.

Its calculation is supported by replicas sending the last applied row
timestamp. When there is no replication, the last applied row timestamp
stays the same, so in this case downstream lag grows as time passes.

Once an old master is replaced by a new one, it notices changes in peer
vclocks and tries to update downstream lag unconditionally. This makes
the lag appear to be growing indefinitely, showing the time since the
last transaction on the old master:

```
 downstream:
   status: follow
   idle: 0.018218606001028
   vclock: {1: 3, 2: 2}
   lag: 34.623061401367
```

The commit 56571d83 ("raft: make followers notice leader hang")
made relay exchange information with tx even when there are no new
transactions, so the issue became even easier to reproduce.

The issue itself was present since downstream lag introduction in commit
29025bce ("relay: provide information about downstream lag").

Closes #7581

NO_DOC=bugfix

The 'log' module uses fibers internally for logs rotation sake and
before we can free log's resources (on program exit) we need to wait
until rotation is complete, which implies that events loop is still
running. But we break the event loop in `on_shutdown_f` trigger and
calling any events based functionality later cause unexpected results
because fibers are no loner valid to use. Thus move `say_logger_free`
call into `on_shutdown_f` body where fibers are still alive.

N.B. Testing the issue is sensitive to timings, during local tests
found that minimal delay 1ms is enough to trigger, thus ERRINJ_LOG_ROTATE
get increased.

Fixes #4450

NO_DOC=bugfix

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

Some functions is src/main.cc are declared as global
while they used in file scope only. Declare them as
appropriate.

NO_DOC=cleanup
NO_CHANGELOG=cleanup
NO_TEST=cleanup

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

If two or more fibers are yielding in fiber_join_timeout(), one of them
will eventually join and recycle the fiber, while the rest will crash
on accessing the recycled fiber's struct. Fix this by doing fiber_find()
again after each waiting attempt in lbox_fiber_join().

Closes #7489
Closes #7531

NO_DOC=bugfix

It is separated from fiber_join_timeout(), and will be used
in lbox_fiber_join() too.

Part of #7489
Part of #7531

NO_DOC=internal
NO_CHANGELOG=internal

Currently `make` in `static-build` doesn't rebuild Tarantool when source
files are changed. Fix this by setting BUILD_ALWAYS option, which forces
rescan for changes of the external project [1]:

> This option is not normally needed unless developers are expected to
> modify something the external project's build depends on in a way that
> is not detectable via the step target dependencies (e.g. SOURCE_DIR is
> used without a download method and developers might modify the sources
> in SOURCE_DIR).

It is available since CMake 3.1, so update cmake_minimum_required, as we
already require it (fa8d70ca).

[1] https://cmake.org/cmake/help/latest/module/ExternalProject.html

Part of #7536

NO_DOC=build
NO_TEST=build
NO_CHANGELOG=minor

Test that an expected Lua function can be found in one of frames.
C function is already covered by this test.

Closes #7535

NO_DOC=test
NO_CHANGELOG=test

CMake accepts the following case-insensitive values as true: 1, ON, YES,
TRUE, Y, or a non-zero number (including floating point numbers). This
complicates the parsing of ENABLE_BACKTRACE in `tarantool.build.options`.
Fix this by defining it to TRUE for any true value.

Part of #7535

NO_DOC=internal
NO_CHANGELOG=internal

As the underlying problem behind this injection is fixed in #7357 it can
be removed and `-i` flag could be used as initially intended.

Closes #7554
Requires #7357
NO_DOC=refactoring
NO_CHANGELOG=refactoring

We will add all source files related to user read views under this
option.

Needed for https://github.com/tarantool/tarantool-ee/issues/191

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

We need these functions to implement format-less tuple comparison.

Needed for https://github.com/tarantool/tarantool-ee/issues/191

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

This patch adds a temporary workaround for LuaJIT profiling tests to
avoid runners shutdown due to no space left on the disk. The profiling
tests may produce profiles until fully fill up the runner [1].

The workaround is based on implementing disk quotas.

In two words, it creates a 1GB file (disk image), formats this file as
an ext4 filesystem, mounts this filesystem to some mount pont and sets
`LUAJIT_TEST_VARDIR=<mount point>`. In this case LuaJIT tests will use
this dir for storing various test data/profiles and not be able to fill
up the runner.

[1] https://github.com/tarantool/tarantool/issues/7472

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

To make a memtx snapshot, we use the create_snapshot_iterator index
method. The method creates a 'frozen' iterator over an index - changes
done to the index after the iterator was created don't affect the
iterator output. Also, the iterator is safe to use from any thread.
This API works just fine for snapshots, but it's too limited to allow
creation of user read views so we need to rework it.

To make the existing snapshot infrastructure suitable for user read
views, this commit replaces the create_snapshot_iterator method with
create_read_view. The new method returns an index_read_view object,
which has the API similar to the read-only API of an index. A read view
object may only be created and destroyed in the tx thread, but it may be
used in any thread.

Currently, index_read_view has the only method - create_iterator, which
takes iterator type and key and returns an index_read_view_iterator
object. The iterator type and key arguments are ignored and we always
assume the iterator type to be ITER_ALL (asserted), but later on we will
fix this and also add a method to look up a tuple by key.

Closes #7194

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Since commit f167c1af ("memtx: decompress tuples in snapshot
iterator") a snapshot iterator may allocate the result tuple on the
fiber region - the caller is supposed to clean the region after usage.
So we don't need to store the tuple in sequence_data_iterator anymore -
we can allocate it on the fiber region instead, which is simpler and
more straightforward.

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

The create_snapshot_iterator index callback is used by the memtx engine
to create a consistent read view of data stored in memtx so that it can
be written to a snapshot or sent to a remote replica. We also define and
use this callback internally in vinyl to implement initial join.

Actually, there's no need to have this code wrapped in a callback in
vinyl, because it's never called from outside the vinyl internals. Let's
inline it and drop the callback for vinyl. This will simplify further
refactoring of the internal index read view API.

Needed for #7194

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

'get_raw' is a misleading name, because usually we append the '_raw'
suffix to functions that work with raw MsgPack while 'get_raw' actually
returns a formatted tuple. The function is used internally in memtx to
implement tuple compression. Let's call it 'get_internal' to emphasize
that.

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

With current leader fencing implementation old leader doesn't resign
it's leadership before new leader may be elected. Because of this
several "leaders" might coexist in replicaset for some time.

This commit changes replication_disconnect_timeout that it is twice
as short for current raft leader (2*replication_timeout) if strict
fencing is enabled. Assuming that replication_timeout is the same for
every replica in replicaset this makes it less probable that new
leader can be elected before old one resigns it's leadership.

Old fencing behaviour can be enabled by setting fencing to soft mode.
This is useful when connection death timeouts shouldn't be affected
(e.g. different replication_timeouts are set to prioritize some
replicas as leader over the others).

Closes #7110

@TarantoolBot document
Title: Strict fencing

In `box.cfg` option `election_fencing_enabled` is deprecated in favor
of `election_fencing_mode`. `election_fencing_mode` can be set to one
of the following values:
'off' - fencing turned off (same as `election_fencing_enabled` set to
false before).
Connection death timeout is 4*replication_timeout for all nodes.

'soft' (default) - fencing turned on, but connection death timeout is
the same for leader and followers in replicaset. This is enough to
solve cluster being readonly and not being to elect a new leader in
some situations because of pre-vote.
Connection death timeout is 4*replication_timeout for all nodes.

'strict' - fencing turned on. In this mode leader tries its best to
resign leadership before new leader can be elected. This is achived
by halving death timeout on leader.
Connection death timeout is 4*replication_timeout for followers and
2*replication_timout for current leader.

Currently box_raft asserts that raft is initialized when it is called.
For strict fencing box_raft will be called in
replication_disconnect_timeout to set different timeouts for leader and
follower. Sometimes replication_disconnect_timeout is called before raft
is initialized.

This commit changes box_raft behaviour, removing the assertion and
returning NULL instead of pointer to global raft state, if raft isn't
initialized. This makes it possible to call box_raft even before raft
has been initialized, checking that return value isn't NULL.

Assuming that this assertion didn't trigger anywhere
else, there is no need to check for box_raft returning NULL anywhere
except new calls. Even if in future this will change it will trigger
segmentation fault and the problem could be easily localized.

Part of #7110

NO_DOC=internal changes
NO_TEST=internal changes
NO_CHANGELOG=internal changes

Before, we used to modify box.cfg.replication to reproduce network
problems in our test. This worked fine in most situations, but doesn't
work in others: when instance gets disconnected by modifying
box.cfg.replication, it closes its connection immediately (in terms of
realtime), and this is noticed almost immediately by its neighbours in
replica set (because they receive EOF). This made it impossible to test
some things, that rely on specific timeouts in our code (e.g. strict
fencing).

This commits adds helper, which acts as UNIX socket proxy, and can block
connection transparently for tarantool instances. It makes it possible
to write some tests, that were not possible before. It is also possible
to inject arbitrary packets between instance, which are interconnected
via proxy.

Usage:

                  +-------------------+
                  |tarantool server 1 |
                  +-------------------+
                            |
                            |
                            |
                   .-----------------.
                  (   /tmp/test-out   )
                   `-----------------'
                            |
                            |
                            |
                  +-------------------+
                  |       proxy       |
                  +-------------------+
                            |
                            |
                            |
                   .-----------------.
          +-------(   /tmp/test-in    )--------+
          |        `-----------------'         |
          |                                    |
          |                                    |
          |                                    |
+-------------------+                +-------------------+
|tarantool server 2 |                |tarantool server 3 |
+-------------------+                +-------------------+

tarantool server 1 init.lua:
box.cfg{listen = '/tmp/test-out'}
box.once("schema", function()
    box.schema.user.grant('guest', 'super')
end)

tarantool server 2 and tarantool server 3 init.lua:
box.cfg{replication = '/tmp/test-in'}

proxy init.lua:
-- Import proxy helper
Proxy = require('test.luatest_helpers.proxy.proxy')

-- Create proxy, which will (when started) listen on client_socket_path
-- and accept connection when client tries to connect. The accepted
-- socket connection is then passed to new Connection instance.
proxy = Proxy:new({
    -- Path to UNIX socket, where proxy will await new connections.
    client_socket_path = '/tmp/test-in',

    -- Path to UNIX socket where tarantool server is listening.
    server_socket_path = '/tmp/test-out',

    -- Table, describing how to process client socket. Optional.
    -- Defaults used and described:
    process_client = {
        -- function(connection) which, if not nil, will be called once
        -- before client socket processing loop.
        pre = nil,

        -- function(connection, data) which, if not nil, will be called
        -- in loop, when new data is received from client socket.
        -- Connection.forward_to_server(connection, data) will:
        -- 1) Connect server socket to server_socket_path, if server
        --    socket is not connected.
        -- 2) Write data to server socket, if connected and writable.
        func = Connection.forward_to_server,

        -- function(connection) which, if not nil, will be called once
        -- after client socket processing loop.
        -- Connection.close_client_socket(connection) will shutdown and
        -- close client socket, if it is connected.
        post = Connection.close_client_socket,
    },

    -- Table, describing how to process server socket. Optional.
    -- Defaults used and described:
    process_server = {
        -- function(connection) which, if not nil, will be called once
        -- before server socket processing loop.
        pre = nil,

        -- function(connection, data) which, if not nil, will be called
        -- in loop, when new data is received from server socket.
        -- Connection.forward_to_client(connection, data) will write data
        -- to client socket, if it is connected and writable
        func = Connection.forward_to_client,

        -- function(connection) which, if not nil, will be called once
        -- after server socket processing loop.
        -- Connection.close_server_socket(connection) will shutdown and
        -- close server socket, if it is connected.
        post = Connection.close_server_socket,
    }

})

-- Bind client socket (defined by proxy.client_socket_path) and start
-- accepting connections on it in a new fiber. If opts.force is set to
-- true, it will remove proxy.client_socket_path file before binding to
-- it. After proxy is started it will accept client connections and
-- create Connection instance for each connection.
proxy:start({force = false})

-- Stop accepting new connetions on client socket and join the fiber,
-- created by proxy:start(), and close client socket. Also stop all
-- active connections (see Connection:stop()).
proxy:stop()

-- Pause accepting new connections and pause all active connections (see
-- Connection:pause()).
proxy:pause()

-- Resume accepting new connections and resume all paused connections
-- (see Connection:resume())
proxy:resume()

-- Connection class:
Connection:new({
    {
        -- Socket which is already created (by Proxy class for example).
        -- Optional, may be nil.
        client_socket = '?table',

        -- Path to connect server socket to. Will try to connect on
        -- initialization, and in Connection.forward_to_server.
        -- Can connect manually by calling
        -- Connection:connect_server_socket().
        server_socket_path = 'string',

        -- See Proxy:new()
        process_client = '?table',

          -- See Proxy:new()
        process_server = '?table',
    },
})

-- Start processing client socket, using functions from
-- Connection:process_client.
Connection:start()

-- Connect server socket to Connection.server_socket_path (if not
-- connected already). Start processing server socket, if successfully
-- connected (using functions from Connection.process_server).
Connection:connect_server_socket()

-- Pause processing packets (both incoming from client socket and server
-- socket).
Connection:pause()

-- Resume processing packets (both incoming from client socket and
-- server socket).
Connection:resume()

-- Close server socket, if open.
Connection:close_server_socket()

-- Close client socket, if open.
Connection:close_client_socket()

-- Close client and server sockets, if open, and wait for processing
-- fibers to die.
Connection:stop()

NO_DOC=test helpers
NO_CHANGELOG=test helpers

This patch introduces the new `report-job-status` action that is used
for sending notifications to the VK Teams messenger on the job failure.
It replaces the old `send-telegram-notify` action sending similar text
notifications to the Telegram messenger.

The new action selects the corresponding chat for the notification on
failure and uses the `tarantool/actions/report-job-status` action [1]
for sending it to:

* the team chat (Tarantool CI/CD reports) - if the job started on any
  event in the master or release branch (1.10, 2.10, etc);

* a personal chat (created by the committer) – if the job started on
  creating/updating a pull request or any event in other branches.

[1] https://github.com/tarantool/actions/tree/master/report-job-status

Resolves tarantool/tarantool-qa#258

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

- Fix checkout with submodules
- Fix publish action version

NO_CHANGELOG=ci
NO_DOC=ci
NO_TEST=ci

The interactive mode has been ignored when stdin was not a tty and is no
more. Now results of another command can be handled by tarantool.
Before the patch:
```
$ echo 42 | tarantool -i
LuajitError: stdin:1: unexpected symbol near '42'
fatal error, exiting the event loop
```

After the patch:
```
$ echo 42 | tarantool -i
Tarantool 2.5.0-130-ge3cf64a6c
type 'help' for interactive help
tarantool> 42
---
- 42
...

```

Closes #5064

NO_DOC=bugfix

Inspired by gh-5064, that breaks the previous version of the test on
openSUSE. When using `io.popen:write()` on tarantool with `-i` flag, it
failed to run the command on openSUSE. This happened because before
gh-5064 patch it used to employ `luaL_loadfile()` that interprets EOF
as the end of the command, while when it is loaded as a string openSUSE
expects it to end with '\n'.

Needed for #5064
NO_DOC=test fix
NO_TEST=test fix
NO_CHANGELOG=test fix

NO_DOC=internal
NO_TEST=internal

NO_CHANGELOG=Update doxygen config
NO_DOC=Update doxygen config
NO_TEST=Update doxygen config

It is counter-intuitive to see options of a component that is disabled
at build time. Especially, when the returned value means that the
component is enabled (while it is not so).

Before this patch (on `-DENABLE_FEEDBACK_DAEMON=OFF` build):

```yaml
tarantool> box.cfg()
tarantool> box.cfg.feedback_enabled
---
- true
...
```

After this patch (on `-DENABLE_FEEDBACK_DAEMON=OFF` build):

```yaml
tarantool> box.cfg()
tarantool> box.cfg.feedback_enabled
---
- null
...
```

NB: The following test cases in cartridge are failed with
`-DENABLE_FEEDBACK_DAEMON=OFF` (as before as well as after the patch):

* integration.feedback.test_feedback
* integration.feedback.test_rocks

Since they verify cartridge's additions for the feedback daemon, it is
expected outcome of disabling the component entirely. Ideally we should
conditionally disable those test cases, but it is out of scope here.

Follows up #3308

NO_DOC=I think it is expected behavior and unlikely it requires any
       change in the documentation
NO_TEST=a test would verify behavior of the particular build type, but
        we have no such configuration in CI, so the test would be pretty
        useless
NO_CHANGELOG=seems too minor to highlight it for users

It is convenient to fast check, whether the option was enabled or
disabled. Especially, when cmake is called indirectly, say, by a package
manager on Gentoo.

Follows up #3308

NO_DOC=quite minor build process change
NO_TEST=has no relation to tarantool behavior
NO_CHANGELOG=see NO_DOC

To avoid potential buffer overflows and to make static analyzers happy.

Fixed CWE-120:
- sprintf: does not check for buffer overflows
- strcpy: does not check for buffer overflows when copying to destination
- strcat: does not check for buffer overflows when concatenating to
  destination

Closes #7534

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

strlcat is a function from BSD, which is designed to be safer, more
consistent, and less error prone replacement for strcat and strncat.

NO_DOC=internal
NO_CHANGELOG=internal

Part of #7534

The hash index doesn't create a snapshot clarifier, which is used for
filtering out uncommitted tuples from a snapshot. Fix this. Also fix
a bug in hash_snapshot_iterator_next, where we passed a wrong argument
to tuple_data_range. It hasn't fired, because the clarifier didn't work.

Fixes commit ee8ed065 ("txm: clarify all fetched tuples").
Fixes commit f167c1af ("memtx: decompress tuples in snapshot
iterator").

Closes #7539

NO_DOC=bug fix

Gap tracking does not handle gap writes when the key has the same value as
the gap item: review the whole gap write handling logic, refactor it and
fix handling of corner cases along the way.

Co-authored-by: Alexander Lyapunov <alyapunov@tarantool.org>

Closes #7375

NO_DOC=bugfix

Since `ITER_ALL` is an alias to `ITER_GE` in context of TREE index,
denormalize it during iterator creation.

Needed for #7375

NO_CHANGELOG=refactoring
NO_DOC=refactoring
NO_TEST=refactoring

The problem is described in #7073. It was fixed only for
`tree_iterator_start_raw` next method, but other methods used for reverse
iterators are also subject to this bug: move tuple clarification from the
wrapper of iterator `next` methods to individual iterator methods.

Closes #7432

NO_DOC=bugfix

LuaJIT tests are disabled in the scope of the issue [1].

[1] https://github.com/tarantool/tarantool/issues/4819

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

The Rust module (see the issue) needs a getter and a setter for decimal
values on the Lua stack. Let's make them part of the module API.

Part of #7228

@TarantoolBot document
Title: Lua/C functions for decimals in the module API

The following functions are added into the module API:

```c
/**
 * Allocate a new decimal on the Lua stack and return
 * a pointer to it.
 */
API_EXPORT box_decimal_t *
luaT_newdecimal(struct lua_State *L);

/**
 * Allocate a new decimal on the Lua stack with copy of given
 * decimal and return a pointer to it.
 */
API_EXPORT box_decimal_t *
luaT_pushdecimal(struct lua_State *L, const box_decimal_t *dec);

/**
 * Check whether a value on the Lua stack is a decimal.
 *
 * Returns a pointer to the decimal on a successful check,
 * NULL otherwise.
 */
API_EXPORT box_decimal_t *
luaT_isdecimal(struct lua_State *L, int index);
```

This change follows the previous commits regarding decimal, uuid and
datetiem functions. See them for details.

Part of #7228

NO_DOC=refactoring, no user-visible changes
NO_TEST=refactoring, no behavior changes
NO_CHANGELOG=refactoring, no user-visible changes

This change follows the previous commits regarding
`luaT_{new,push}decimal()` and `luaT_{new,push}uuid()`. See them for
details.

Part of #7228

NO_DOC=refactoring, no user-visible changes
NO_TEST=refactoring, no behavior changes
NO_CHANGELOG=refactoring, no user-visible changes

This change follows the previous commit regarding `luaT_newdecimal()`
and `luaT_pushdecimal()`, see explanation and details there.

Also changed the `luaL_` prefix to more appropriate `luaT_`. The `struct
tt_uuid` is our own type, the functions are specific to tarantool. So
`luaT_`.

Part of #7228

NO_DOC=refactoring, no user-visible changes
NO_TEST=refactoring, no behavior changes
NO_CHANGELOG=refactoring, no user-visible changes

`luaT_pushdecimal()` now accepts a decimal argument to copy into the Lua
managed memory.

`luaT_newdecimal()` now doing what `luaT_pushdecimal()` did before: just
allocates a storage for decimal on the Lua stack.

This naming looks much more friendly. It also seems that it follow Lua
API names: `lua_push*()` accepts what to push, `lua_new*()` doesn't.

A couple of notes around the change:

* On the first glance it seems that `luaT_pushdecimal()` is redundant,
  because it can be written using `luaT_newdecimal()` + copying. That's
  truth in contexts, where we know size of the internal `decimal_t`
  structure. A user of the module API don't know it and should pass
  `box_decimal_t *` pointer to `luaT_pushdecimal()` to write the value.
* I use `memcpy()` instead of just `*a = *b` in `luaT_pushdecimal()` to
  copy the padding byte content. Who knows, maybe this not-so-legal way
  to hold extra information may be crucial for some use case or will
  allow us to add one field into the structure.

This is preparatory commit for exposing `luaT_*decimal()` functions into
the module API.

Next commits will change uuid, datetime, interval functions in the same
way.

Part of #7228

NO_DOC=refactoring, no user-visible changes
NO_TEST=will be tested in a next commit, after exposing to the
        module API
NO_CHANGELOG=refactoring, no user-visible changes