In commit (35334ca1) qsort was fixed but unfortunately a small
typo was introduced. Due to that typo the qsort made its job wrong.

Fix the problem and add unit test for qsort.

Unfortunately the test right from the issue runs extremely long,
so it should go to long-tests.

Closes #7605

NO_DOC=bugfix

It is unused and misleading. Let's remove them so that now we have
single entry point for log subsystem - `say()`.

NO_DOC=<Refactoring>
NO_CHANGELOG=<Refactoring>
NO_TEST=<Refactoring>

Let's introduce on_log_level static variable which is assumed to be
configured in `say_set_log_callback()`. on_log_level is assumed to be
log level of `log->on_log` callback (i.e. if entry to be logger features
higher log level - it is simply skipped). Note that now casual log_level
is calculated as MAX(level, on_log_level) since log_level is the single
guard for passing execution flow to `log_vsay()` where both things (to
be precise on_log callback invocation and ordinary logging) happens.

This change is required since if log_level has lower magnitude than
on_log_level - on_log callback will be skipped.

NO_DOC=<Internal change>
NO_TEST=<Internal change>
NO_CHANGELOG=<Internal change>

This patch fixes a number of issues with trigger_clear() while the
trigger list is being run:
1) clearing the next-to-be-run trigger doesn't prevent it from being run
2) clearing the next-to-be-run trigger causes an infinite loop or a
   crash
3) swapping trigger list head before the last trigger is run causes an
   infinite loop or a crash (see space_swap_triggers() in alter.cc, which
   had worked all this time by miracle: space _space on_replace trigger
   swaps its own head during local recovery, and that had only worked
   because the trigger by luck was the last to run)

This is fixed by adding triggers in a separate run list on trigger_run.
This list may be iterated by `rlist_shift_entry`, which doesn't suffer
from any of the problems mentioned above.

While being bad in a number of ways, old approach supported practically
unlimited number of concurrent trigger_runs for the same trigger list.
The new approach requires the trigger to be in as many run lists as
there are concurrent trigger_runs, which results in quite a big
refactoring.

Add a luatest-based test and a unit test.

Closes #4264

NO_DOC=bugfix

struct trigger is about to get a new field, and it's mandatory that this
field is specified in all initializers. Let's introduce a macro to avoid
adding every new field to all the initializers and at the same time keep
the benefits of static initialization.

Also while we're at it fix `lbox_trigger_reset` setting all trigger
fileds manually.

Part-of #4264

NO_DOC=refactoring
NO_CHANGELOG=refactoring
NO_TEST=refactoring

Make trigger_fiber_run return an error, when it occurs, so that the
calling code decides how to log it.
Also, while I'm at it, simplify trigger_fiber_run's code a bit.

In-scope-of #4264

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

cord_exit should be always called in the exiting thread. It's a single
place to call all the thread-specific module deinitalization routines.

In-scope-of #4264

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Unit test compilation with `#define UNIT_TAP_COMPATIBLE 1` might fail
with an error complaining that <stdarg.h> is not included. Fix this.

In-scope-of #4264

NO_CHANGELOG=testing stuff
NO_DOC=testing stuff

Our triggers support recursive invocation: for example, an on_replace
trigger on a space may do a replace in the same space.

However, this is not tested and might get broken easily. Let's add a
corresponding test.

In-scope-of #4264

NO_DOC=testing
NO_CHANGELOG=testing

We will use it in the read view Lua API. For memtx tree and hash
indexes, we add a stub function with a possibility of override if
ENABLE_READ_VIEW is set. For the sequence data space, we raise an
error if this function is called, because we aren't planning to
implement it in the EE repository.

Needed for https://github.com/tarantool/tarantool-ee/issues/197

NO_DOC=internal
NO_TEST=ee
NO_CHANGELOG=internal

We aren't planning to support all iterator types for _sequence read
views, but once we implement a Lua API for read views, the user will
still be able to create a read view of this space. So let's replace
the assertions with a graceful error.

Needed for https://github.com/tarantool/tarantool-ee/issues/197

NO_DOC=internal
NO_TEST=ee
NO_CHANGELOG=internal

Tuples stored in a read view might not have a format, because we
overwrite the tuple format id when we free a tuple. So for anything more
sophisticated than a full-scan, we need to use special comparators that
don't access the tuple format. To address that, let's provide a way to
reset tree and hash key_def if ENABLE_READ_VIEW is defined.

The stub functions simply reset the read view key_def to NULL, because
it isn't supposed to be used in the CE version.

Needed for https://github.com/tarantool/tarantool-ee/issues/197

NO_DOC=internal
NO_TEST=ee
NO_CHANGELOG=internal

For memtx tree and hash indexes: if the ENABLE_READ_VIEW macro is
defined, include a source file with the read view iterator
implementation instead of defining it directly. We don't need to support
any iterator types except ALL in the CE repository, because user read
views will be available only in EE. All other iterator types will be
defined in the EE repository. We include source files (not headers),
because to implement a read view iterator, we need access to index
internals, which are defined in memtx_hash.cc and memtx_tree.cc.

The included implementation source file is supposed to define
the {tree,hash}_read_view_iterator_start function, which positions the
iterator to the given key. It's called directly by create_iteartor index
read view method.

For the memtx tree index, the iterator implementation needs to know the
iteration key (for handling EQ/REQ requests) so we also add the key to
the iterator struct in this commit.

Needed for https://github.com/tarantool/tarantool-ee/issues/197

NO_DOC=internal
NO_TEST=ee
NO_CHANGELOG=internal

We can't return tuples retrieved from a consistent index read view as
is. We need to filter out dirty tuples and decompress compressed tuples.
Let's introduce a helper function for this. Currently, this helper
function is used only in the ALL iterator implementation of the tree and
hash indexes. Later, we will use it in all other iterator types, which
will be implemented in the EE repository.

Needed for https://github.com/tarantool/tarantool-ee/issues/197

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

To implement all iterator types for memtx tree and hash index read
views, we need to have the key definition. We can't just use the key
definition stored in the index directly, because, despite the fact that
a read view keeps a reference the index, its index definition may still
be changed by alter (when compatible changes happen).

To solve this problem, let's store a copy of the index definition in
memtx tree and hash index read views. The index definition also contains
the index name, which will be useful for exporting the read view to Lua.

Needed for https://github.com/tarantool/tarantool-ee/issues/197

NO_DOC=internal
NO_TEST=ee
NO_CHANGELOG=internal

When an iterator gets exhausted, we often set the iterator callback
to a stub function that always returns NULL. Let's add a global stub
function for that so that we can reuse it.

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

See d3f32d18 for explanation.

NO_CHANGELOG=ci
NO_DOC=ci
NO_TEST=ci

Publishing workflow will run building module API documentation for
opened or reopened pull request with label "full-ci" and will publish
documentation on Github Pages on push to the master branch.

NO_CHANGELOG=ci
NO_DOC=ci
NO_TEST=ci

Fixup of f06417 ("doc: publish autogenerated module api documentation").

https://tarantool.github.io/tarantool/api/html/module_8h.html

NO_CHANGELOG=ci
NO_DOC=ci
NO_TEST=ci

Report workflow failures in PRs made by TarantoolBot to the same chat
as with stable branches. Such PRs are used for automated integration
testing, so it's important for the team to notice failures in them.
There is no personal chat for TarantoolBot and no need to make one.

NO_DOC=CI reporting
NO_TEST=CI reporting
NO_CHANGELOG=CI reporting

This patch allows to call `report-job-status` action with only one
input: `bot-token`. VK Teams chat ID has the default value in current
action, API URL has the default value in [1].

[1] `tarantool/actions/report-job-status`

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

This is a degradation introduced by commit 26f7056f ("Introduce
internal database read view API"): tuple garbage collection is never
resumed after a snapshot. This happens, because we don't add engine read
views to the read view engine list on read view construction. As a
result, read_view_close() never closes the memtx engine read view.

NO_DOC=bug fix
NO_CHANGELOG=unreleased

When multiline commands were loaded from .tarantool_history, they were
treated as a bunch of oneline commands. Now readline is configured to
write timestamps in .tarantool_history as delimiters and multiline
commands are handled correctly.

If there is already a .tarantool_history file, readline will set
timestamps automatically, nothing will be lost.

Closes #7320
NO_DOC=bugfix
NO_TEST=impossible to check readline history from lua

This patch adds a warning about using the action in pull requests
created from forks. GitHub secrets are not passed to workflows in
this case [1] and due to this limitation the action will not work
correctly for such pull requests.

[1] https://docs.github.com/en/actions/security-guides/encrypted-secrets#using-encrypted-secrets-in-a-workflow

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

The problem could be easily reproduced with following command line:
./test/test-run.py
	$(yes app-luatest/http_client_test.lua | head -n 1000).

Before this commit we did a socket binding to know a free network port,
then close a socket and started httpd.py on that network port. However
it was not reliable and even with socket options SO_REUSE_PORT start of
httpd.py has failed. With proposed patch schema is changed: we start
httpd.py and pass only a socket family (AF_INET for TCP connection and
AF_UNIX for connection via Unix socket) and then reading output from a
process. Successfully started httpd.py prints a path to a Unix socket or
a pair of IP address and network port split with ":".

With proposed patch test has passed 1000 times without any problems.
Tests previously marked as "fragile" are passed too:

./test/test-run.py --builddir=$(pwd)/build box-tap/net.box.test.lua \
	box-tap/cfg.test.lua box-tap/session.storage.test.lua \
	box-tap/session.test.lua app-tap/tarantoolctl.test.lua \
	app-tap/debug.test.lua app-tap/inspector.test.lua \
	app-tap/logger.test.lua app-tap/transitive1.test.lua \
	app-tap/csv.test.lua app-luatest/http_client_test.lua

P.S. The problem with "fragile" tests is that rerunning hides other
problems. [1] is about "Address already in use" and [2] is about hangs
in test. I made a pull request with changes in http client module and
triggered CI run. Job has been passed, but in log [3] I see three test
restarts due to fails in http_client test related to my changes.

1. https://github.com/tarantool/tarantool-qa/issues/186
2. https://github.com/tarantool/tarantool-qa/issues/31
3. https://github.com/tarantool/tarantool/runs/7726358823?check_suite_focus=true

Closes https://github.com/tarantool/tarantool-qa/issues/186
Closes https://github.com/tarantool/tarantool-qa/issues/31

NO_CHANGELOG=testing
NO_DOC=testing
NO_TEST=testing

Currently, we create a database read view only to create a memtx
snapshot or join a replica, but there's already quite a bit of code
duplication between these two scenarios. In the future, we will need
the same functionality to create a user read view. So let's factor out
this code into a separate module - read_view.

The API of the read_view module is quite simple - there are just two
methods: open and close a read view. The user can pass a space and index
filter while opening a read view to skip certain spaces. E.g. we skip
all temporary spaces and secondary indexes when we create a memtx
snapshot. A read_view object has a list of space_read_view objects, one
per each space included into the read view. A space_read_view object, in
turn, has a map of all index_read_view objects (introduced earlier)
corresponding to space indexes. There's nothing like a space cache - the
user can create one if required.

An engine that supports creation of a read view (currently, only memtx)
is supposed to set the ENGINE_SUPPORTS_READ_VIEW flag and implement the
create_read_view engine method in addition to the create_read_view index
method. The engine method should do some engine-wide read view related
preparations. For example, in case of memtx, it suspends tuple garbage
collection.

Closes #7363

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

downstream lag is the difference in time between the moment a
transaction was written to master's WAL and the moment an ack for it
arrived.

Its calculation is supported by replicas sending the last applied row
timestamp. When there is no replication, the last applied row timestamp
stays the same, so in this case downstream lag grows as time passes.

Once an old master is replaced by a new one, it notices changes in peer
vclocks and tries to update downstream lag unconditionally. This makes
the lag appear to be growing indefinitely, showing the time since the
last transaction on the old master:

```
 downstream:
   status: follow
   idle: 0.018218606001028
   vclock: {1: 3, 2: 2}
   lag: 34.623061401367
```

The commit 56571d83 ("raft: make followers notice leader hang")
made relay exchange information with tx even when there are no new
transactions, so the issue became even easier to reproduce.

The issue itself was present since downstream lag introduction in commit
29025bce ("relay: provide information about downstream lag").

Closes #7581

NO_DOC=bugfix

The 'log' module uses fibers internally for logs rotation sake and
before we can free log's resources (on program exit) we need to wait
until rotation is complete, which implies that events loop is still
running. But we break the event loop in `on_shutdown_f` trigger and
calling any events based functionality later cause unexpected results
because fibers are no loner valid to use. Thus move `say_logger_free`
call into `on_shutdown_f` body where fibers are still alive.

N.B. Testing the issue is sensitive to timings, during local tests
found that minimal delay 1ms is enough to trigger, thus ERRINJ_LOG_ROTATE
get increased.

Fixes #4450

NO_DOC=bugfix

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

Some functions is src/main.cc are declared as global
while they used in file scope only. Declare them as
appropriate.

NO_DOC=cleanup
NO_CHANGELOG=cleanup
NO_TEST=cleanup

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

If two or more fibers are yielding in fiber_join_timeout(), one of them
will eventually join and recycle the fiber, while the rest will crash
on accessing the recycled fiber's struct. Fix this by doing fiber_find()
again after each waiting attempt in lbox_fiber_join().

Closes #7489
Closes #7531

NO_DOC=bugfix

It is separated from fiber_join_timeout(), and will be used
in lbox_fiber_join() too.

Part of #7489
Part of #7531

NO_DOC=internal
NO_CHANGELOG=internal

Currently `make` in `static-build` doesn't rebuild Tarantool when source
files are changed. Fix this by setting BUILD_ALWAYS option, which forces
rescan for changes of the external project [1]:

> This option is not normally needed unless developers are expected to
> modify something the external project's build depends on in a way that
> is not detectable via the step target dependencies (e.g. SOURCE_DIR is
> used without a download method and developers might modify the sources
> in SOURCE_DIR).

It is available since CMake 3.1, so update cmake_minimum_required, as we
already require it (fa8d70ca).

[1] https://cmake.org/cmake/help/latest/module/ExternalProject.html

Part of #7536

NO_DOC=build
NO_TEST=build
NO_CHANGELOG=minor

Test that an expected Lua function can be found in one of frames.
C function is already covered by this test.

Closes #7535

NO_DOC=test
NO_CHANGELOG=test

CMake accepts the following case-insensitive values as true: 1, ON, YES,
TRUE, Y, or a non-zero number (including floating point numbers). This
complicates the parsing of ENABLE_BACKTRACE in `tarantool.build.options`.
Fix this by defining it to TRUE for any true value.

Part of #7535

NO_DOC=internal
NO_CHANGELOG=internal

As the underlying problem behind this injection is fixed in #7357 it can
be removed and `-i` flag could be used as initially intended.

Closes #7554
Requires #7357
NO_DOC=refactoring
NO_CHANGELOG=refactoring

We will add all source files related to user read views under this
option.

Needed for https://github.com/tarantool/tarantool-ee/issues/191

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

We need these functions to implement format-less tuple comparison.

Needed for https://github.com/tarantool/tarantool-ee/issues/191

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

This patch adds a temporary workaround for LuaJIT profiling tests to
avoid runners shutdown due to no space left on the disk. The profiling
tests may produce profiles until fully fill up the runner [1].

The workaround is based on implementing disk quotas.

In two words, it creates a 1GB file (disk image), formats this file as
an ext4 filesystem, mounts this filesystem to some mount pont and sets
`LUAJIT_TEST_VARDIR=<mount point>`. In this case LuaJIT tests will use
this dir for storing various test data/profiles and not be able to fill
up the runner.

[1] https://github.com/tarantool/tarantool/issues/7472

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

To make a memtx snapshot, we use the create_snapshot_iterator index
method. The method creates a 'frozen' iterator over an index - changes
done to the index after the iterator was created don't affect the
iterator output. Also, the iterator is safe to use from any thread.
This API works just fine for snapshots, but it's too limited to allow
creation of user read views so we need to rework it.

To make the existing snapshot infrastructure suitable for user read
views, this commit replaces the create_snapshot_iterator method with
create_read_view. The new method returns an index_read_view object,
which has the API similar to the read-only API of an index. A read view
object may only be created and destroyed in the tx thread, but it may be
used in any thread.

Currently, index_read_view has the only method - create_iterator, which
takes iterator type and key and returns an index_read_view_iterator
object. The iterator type and key arguments are ignored and we always
assume the iterator type to be ITER_ALL (asserted), but later on we will
fix this and also add a method to look up a tuple by key.

Closes #7194

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Since commit f167c1af ("memtx: decompress tuples in snapshot
iterator") a snapshot iterator may allocate the result tuple on the
fiber region - the caller is supposed to clean the region after usage.
So we don't need to store the tuple in sequence_data_iterator anymore -
we can allocate it on the fiber region instead, which is simpler and
more straightforward.

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal