When Lua main script was launched, the sched fiber passed its own
diag to the script's fiber. When the script was finished, it put
its error into the diag. The sched fiber then checked if the diag
is empty to detect an error.

But it wasn't really correct. The error could also happen right in
the scheduler fiber in a libev callback. For example, in one of
ev_io callbacks in SWIM. Then the process would end with an error
even if the script was finished successfully.

These errors were not related to the main fiber executing the
script.

The patch makes so the scheduler fiber's diag no longer is used as
an indication of an error in the script. Instead, a new diag is
created on the stack of the scheduler's fiber, where the Lua
script saves the error.

Closes #5864

Swim node couldn't talk to broadcast network interfaces because
the option SO_BROADCAST wasn't set.

It worked fine for localhost broadcast, but failed for all the
other IPs. There is no a test, because the tests work for the
localhost only anyway.

It still fails on Mac though in case the swim node was bound to
127.0.0.1. Then somewhy sendto() raises EADDRNOTAVAIL on attempt
to broadcast beyond the local machine. It happens on Linux too,
but with EINVAL error. These errors are ignored because are not
critical.

Part of #5864

Was caught by base64 test with enabled ASAN.

It also caused data corruption - garbage instead of "extra bits" was
saved into state->result if there was no space in output buffer.

Decode state removed along with helper functions.

Added test for "zero-sized output buffer" case.

Fixes: #3069
(cherry picked from commit 7214add2c7f2a86265a5e08f2184029a19fc184d)

Since the introduction of asynchronous commit, which doesn't wait for a
WAL write to succeed, it's quite easy to clog WAL with huge amounts
write requests. For now, it's only possible from an applier, since it's
the only user of async commit at the moment.

This happens when replica is syncing with master and reads new
transactions at a pace higher than it can write them to WAL (see docbot
request for detailed explanation).

To ameliorate such behavior, we need to introduce some limit on
not-yet-finished WAL write requests. This is what this commit is trying
to do.
A new counter is added to wal writer: queue_size (in bytes) together with a
corresponding configuration setting: `wal_queue_max_size`.
The counter is increased on every new submitted request, and decreased once
the tx thread receives a confirmation that a specific request was written.

Actually, the limit is added to an abstract journal queue, but
currently works only for wal writer, since it's the only possible journal
when applier is working.

Once size reaches its maximum value, applier is blocked until
some of the write requests are finished.

The size limit isn't strict, i.e. if there's at least one free byte, the
whole write request fits and no blocking is involved.

The feature is ready for `box.commit{is_async=true}`. Once it's
implemented, it should check whether the queue is full and let the user
decide what to do next. Either wait or roll the tx back.

Closes #5536

@TarantoolBot document
Title: new configuration option: 'wal_queue_max_size'

`wal_queue_max_size` puts a limit on the amount of concurrent write requests
submitted to WAL.
`wal_queue_max_size` is measured in number of bytes to be written (0
means unlimited, which was the default behaviour before).
The option only affects replica behaviour at the moment, and defaults
to 16 megabytes. The option limits the pace at which replica reads new
transactions from master.

Here's when the option comes in handy:

Before this option was introduced such a situation could be possible:
there are 2 servers, a master and a replica, and the replica is down for
some period of time. While the replica is down, master serves requests
at a reasonable pace, possibly close to its WAL throughput limit. Once the
replica reconnects, it has to receive all the data master has piled up and
there's no limit in speed at which master sends the data to replica, and,
without the option, there was no limit in speed at which replica submitted
corresponding write requests to WAL.

This lead to a situation when replica's WAL was never in time to serve the
requests and the amount of pending requests was constantly growing.
There was no limit for memory WAL write requests take, and this clogging
of WAL write queue could even lead to replica using up all the available
memory.

Now, when `wal_queue_max_size` is set, appliers will stop reading new
transactions once the limit is reached. This will let WAL process all the
requests that have piled up and free all the excess memory.

Implemented on_shutdown API, which allows to register functions
that will be called when the tarantool stopped. Functions will
be called in the reverse order they are registered. So the module
developer registers one fuction that starts module termination and
waits for its competition. This function should be fast or used an
asynchronous waiting mechanism (coio_wait or cord_cojoin for example).

Closes #5723

@TarantoolBot document
Title: Implement on_shutdown API
Implemented on_shutdown API, which allows to register functions
that will be called when the tarantool stopped. Functions will
be called in the reverse order they are registered. So the module
developer registers one fuction that starts module termination and
waits for its competition. This function should be fast or used an
asynchronous waiting mechanism (coio_wait or cord_cojoin for example).

Previously lua on_shutdown triggers were started sequentially, now
each of triggers starts in a separate fiber. Tarantool waits for 3.0
seconds to their completion by default. User has the option to change
this value using new implemented box.ctl.set_on_shutdown_timeout function.
If timeout has expired, tarantool immediately stops, without waiting for
other triggers completion.
Also moved ev_break from trigger to the on_shutdown_f function, after
calling all on_shutdown lua triggers, because now all triggers are
started asynchronously in fibers, and we should call ev_break only
after all triggers are finished.

Part of #5723

@TarantoolBot document
Title: Changed Lua on_shutdown triggers behaviour.
Previously lua on_shutdown triggers were started sequentially, now
each of triggers starts in a separate fiber. Tarantool waits for 3.0
seconds to their completion by default. User has the option to change
this value using new implemented box.ctl.set_on_shutdown_timeout function.
If timeout has expired, tarantool immediately stops, without waiting for
other triggers completion.

Since the function for registering on_shutdown triggers for
tarantool modules was decided to be named box_on_shutdown,
the head of the trigger list with a similar name was renamed.

Part of #5723

Implemented function for starting a chain of triggers
in separate fibers, which is required for on_shutdown
API implementation.

Part of #5723

Implemented fiber_join_timeout function, which allows to wait
for the completion of the fiber for a specified period of time.
Function returns fiber execution status to the caller or -1
if the timeout exceeded and set diag. Needed for further
on_shutdown API implementation.

Part of #5723

Renamed granularity option to slab_alloc_granularity, according
to the name of the other options for small allocator.

Follow-up #5518

This test-run update offers fixes of two problems:

* Unhandled OSError exception that occurs rarely, under a heavy load
  (see [1]).
* The 'attempt to compare nil with number' error on test_run:wait_lsn(),
  when an instance is just bootstrapped (see [2]).

[1]: https://github.com/tarantool/test-run/issues/270
[2]: https://github.com/tarantool/test-run/issues/226

LuaJIT submodule is bumped to introduce the following changes:
* test: disable LuaJIT CLI tests in lua-Harness suite
* test: set USERNAME env var for lua-Harness suite
* test: adjust lua-Harness tests that use dofile
* test: adjust lua-Harness suite to CMake machinery
* test: add lua-Harness test suite

Within this changeset lua-Harness suite[1] is added to Tarantool
testing. Considering Tarantool specific changes in runtime the suite
itself is adjusted in LuaJIT submodule.

However, Tarantool provides and unconditionally loads TAP module
conflicting with the one used in the new suite. Hence, the Tarantool
built-in module is "unloaded" in test/luajit-test-init.lua.

Furthermore, Tarantool provides UTF-8 support via another built-in
module. Its interfaces differ from the ones implemented in Lua5.3 and
moonjit. At the same time our LuaJIT fork provides no UTF-8 support, so
lua-Harness UTF-8 detector is simply confused with non-nil utf8 global
variable. As a result, utf8 is set to nil in test/luajit-test-init.lua.

There are also some tests launching Lua interpreter, so strict need to
be disabled for their child tests too. Hence `strict.off()` is added to
`progname` (i.e. arg[-1] considering the way Tarantool parses its CLI
arguments) command used in these tests.

[1]: https://framagit.org/fperrad/lua-Harness/tree/a74be27/test_lua



Closes #5844
Part of #4473

Reviewed-by: Sergey Ostanevich <sergos@tarantool.org>
Reviewed-by: Igor Munkin <imun@tarantool.org>
Signed-off-by: Igor Munkin <imun@tarantool.org>

xdir_say_gc() takes errno and return code of unlink() sys call. If RC
is negative (meaning that unlink failed) we reset errno to given value
and log corresponding error message (it is done this way since eio saves
errno to internal structure so we have to restore it manually). Before
this patch, unlink() call was "in-place" of argument. However, the order
of argument evaluation is unspecified. So it may turn out that we assign
errno to the previous value, which is obviously wrong. To fix it let's
firstly invoke unlink() and then pass the result of call to
xdir_say_gc().

Added manual and backend triggers to run test workflows. It will
give the ability to run missed/needed workflows in Github Actions
and to use standalone backend scripts to run test workflows.

In case if there only one snapshot or xlog file
there is no need to call sorting procedure at all.

In-scope-of #5806

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

Currently we use unsined int for "cleanup" schedule
counting, this is safe while this routine is not
called too often. Still there is a chance to hit
a number wrap on code modification because there
is no strict rule on how to use this garbage collector.
Lets use wide integers instead, we have only one gc
instance and such approach eliminates potential problems
in future (actually this should had been done from the
beginning since the current gc code flow developed
without wrapping in mind).

In-scope-of #5806

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

After porting LuaJIT build system to CMake in commit
07c83aab ('build: adjust LuaJIT build
system'), its build options are not fully maintained in Tarantool. E.g.
several compile flags, such as -fomit-frame-pointer, are set within
LuaJIT CMake machinery and there is no way to tweak them outside. As a
result ASAN + LSAN build in Tarantool CI[1] reports new leaks related to
LuaJIT runtime, but there is none of them actually (no source code
changes are made in scope of the applied patchset). Hence it was decided
to consider all LuaJIT related warnings as false positives for now and
suppress them until #5878 is resolved.

[1]: https://github.com/tarantool/tarantool/runs/1999839396



Follows up #4862
Relates to #5878

Reviewed-by: Alexander V. Tikhonov <avtikhon@tarantool.org>
Reviewed-by: Kirill Yukhin <kyukhin@tarantool.org>
Signed-off-by: Igor Munkin <imun@tarantool.org>

Closes #5652

We use cmake instead of autotools to build curl. Change was introduced
in commit 11dcc9cb ("build: remove
autotools from packages spec"). No sense to require autoconf, automake
and libtool to build tarantool.

Part of #5652

Part of #5652

Fix package names with Python modules on Centos 7, OpenSUSE, Fedora.
All supported RPM-based distributives (Fedora 33, Fedora 32, OpenSUSE 15.2,
OpenSUSE 15.1, CentOS 8) contains python3, python3-six and python3-gevent
packages except CentOS 7 and CentOS 6.
We don't run tests when make packages for CentOS 6. For CentOS 7 we create
our own RPM packages for Python3 dependencies.

Patch unblocks regression testing on Fedora 33.

Part of #5652
Closes tarantool/tarantool-qa#17

- install python3 on mac machines
- run test-run without specifying path to python interpreter
- get rid our own tarantool brew-tap that depends on Python 2

Part of #5652

getaddrinfo() is a POSIX function that returns addrinfo structure in
success and error value for the failure.POSIX standard [1]
describes possible error values but description have a status of
recommendation. When non-existent hostname is passed to getaddrinfo() it
fails with error, but error value may be different on Linux and FreeBSD.
We have testcases where we check error messages obtained from
getaddinfo() ignoring this fact and sometime test fails on FreeBSD [2].

1. https://pubs.opengroup.org/onlinepubs/9699919799/functions/freeaddrinfo.html
2. https://github.com/tarantool/tarantool/runs/2107188391

Part of #5652

Part of #5652

Response object is a unpacked msgpack structure, returned by Tarantool.
Nor msgpack-python [1] that used in tarantool-python nor msgpack [2]
itself cannot guarantee an order of keys in unpacked dictionaries.
Therefore we have different keys order with running tests under Python 2
and Python 3, for example box-py/call.test.py. To workaround a problem
proposed a conversion of dictionaries in a tuple to lists before
printing Response object.

Requires changes in tarantool-python [3].

1. https://github.com/msgpack/msgpack-python/pull/164
2. https://github.com/msgpack/msgpack/issues/215
3. https://github.com/tarantool/tarantool-python/pull/186

Part of #5652
Follows up #5538

Test output with running on Python 2 and Python 3 is not the same.

[009] box-py/iproto.test.py                                           [ fail ]
[009]
[009] Test failed! Result content mismatch:
[009] --- box-py/iproto.result  Sat Mar 13 17:46:54 2021
[009] +++ /rw_bins/test/var/rejects/box-py/iproto.reject        Sat Mar 13 17:50:51 2021
[009] @@ -29,7 +29,7 @@
[009]  IPROTO_UPDATE
[009]  query {'IPROTO_CODE': 4} {'IPROTO_SPACE_ID': 280}
[009]  True
[009] -query {'IPROTO_CODE': 4} {'IPROTO_SPACE_ID': 280, 'IPROTO_KEY': (1,)}
[009] +query {'IPROTO_CODE': 4} {'IPROTO_KEY': (1,), 'IPROTO_SPACE_ID': 280}
[009]  True
[009]
[009]
[009]
[009] Last 15 lines of Tarantool Log file [Instance "box"][/rw_bins/test/var/009_box-py/box.log]:

1. https://github.com/tarantool/tarantool/pull/5751/checks?check_run_id=2103097418

Part of #5652
Follows up #5538

Part of #5652

Part of #5652

golden100 is an array of int32_t elements. So sizeof(uint64_t)
is exactly double array element size. In other words, only half
of golden100 array is checked. This patch makes calculation of
array size type-independed.

Reviewed-by: Cyrill Gorcunov <gorcunov@gmail.com>

The reason why tarantool -e always enters interactive mode is that
statement after option -e isn't considered as a script.

In man PUC-Rio lua there are different names for statement -e (stat)
and script, but they have the same behavior regarding interactive
mode. (Also cases, when interpreter loads stdin, have the same behaviour).

NOTE: test for this code fix uses errinjs, and the last one  should work only
in debug mode, so added `release_disabled` in suite.ini. But there is a bug in
test-run: `release_disable` disables tests at each build type. Partially this
problem is descripted in tarantool/test-run#199.

Fixes #5040

Sometimes, it's useful to set error injections via environment
variables and this commit adds this opportunity.

e.g: `ERRINJ_WAL_WRITE=true tarantool` will be launched with
ERRINJ_WAL_WRITE setted to true.

Errinjs with bool parameters can be set to "true", "false",
"True", "False", "TRUE", "FALSE", etc. (case-insensitive variable).

Errinjs with int or double parameters should be whole valid ("123s" is invalid).
e.g. for int or double: "123", "-1", "2.34", "+2.34".

NOTE: errinjs should work only in debug mode, so added `release_disabled` in
suite.ini. But there is a bug in test-run: `release_disable` disables tests at
each build type. Partially this problem is descripted in tarantool/test-run#199.

Part of #5040

Seems previously "path" wasn't installed to index schema. This
patch fixes it and introduces a test.

Closes #5451

This commit adds the misc module support into the pretest_clean
machinery: so `_G.misc` and `package.loaded.misc` are not cleared before
running of a test of the 'core = tarantool' type. See [1] for details.

[1]: https://github.com/tarantool/test-run/commit/34ea4afd6543d8bc29c371ec2effb756a7460520

Follows up #5187

Relay structure must have alignas(CACHELINE_SIZE), but when
we use simple malloc or same function to allocate memory for
this structure we get pointer with incorrect alignment.
So now we allocate memory for this struct using aligned_alloc.

Granularity is an option that allows user to set
multiplicity of memory allocation in small allocator.
Granulatiry must be exponent of two and >= 4. By default
granularity value == sizeof(intptr_t), as it was before,
when this option was not provided.

@TarantoolBot document
Title: Add 'granularity' option to box.cfg{}
Add granularity option that allows user to set multiplicity
of memory allocation in small allocator. Granularity determines
not only alignment of objects, but also size of the objects in
the pool. Thus, the greater the granularity, the greater the
memory loss per one memory allocation, but tuples with different
sizes are allocated from the same mempool, and we do not lose
memory on the slabs, when we have highly distributed tuple sizes.
This is somewhat similar to a large alloc factor. The smaller the
granularity, the less memory loss per allocation, if the user has
many small tuples of approximately the same size, it will be nice
to set granularity == 4 to save memory.

This option must be set once during start, default value
== sizeof(intptr_t) (8 on 64 bit platforms), as it was before, when
this option was not provided. Granularity must be exponent of two
and >= 4. Together with the slab_alloc_factor, this option gives you
full control over the behavior of small allocator.

Closes #5518

Found that ubuntu-latest that switched in Github Actions from
ubuntu-18.04 to ubuntu-20.04, doesn't have createrepo package
available in Ubuntu repositories to be installed. It happened
because createrepo written on python2, which is not supported
by ubuntu-20.04. Anyway createrepo tool was rewritten on C to
make it available for the later distributions, its new naming
is createrepo_c. It is already avaliable in Ubuntu 21.04, but
still not backported to 20.04 version. So we need to wait its
avalaibility in Ubuntu 22.04 either its backported version in
20.04 [1]. As for now we need to use ubuntu-18.04 in Actions.

[1]: https://answers.launchpad.net/createrepo/+question/690448

LuaJIT submodule is bumped to introduce the following changes:
* test: adjust LuaJIT test suite for Tarantool
* test: change LuaJIT test suite to match b4e6bf0
* test: change LuaJIT test suite to match 5a61e1a
* test: change LuaJIT test suite to match c198167
* test: change LuaJIT test suite to match de5568e
* test: add LuaJIT-test-cleanup test suite
* test: fix Lua command in utils.selfrun
* test: fix luacheck invocation for non-real paths

Within this changeset LuaJIT-test-cleanup suite[1] is added to Tarantool
testing. Considering Tarantool specific changes in runtime the suite
itself is adjusted in LuaJIT submodule. However, there is <strict>
module enabled by default in Debug build, so the testing environment
need to be tweaked via test/luajit-test-init.lua script to be run prior
to every LuaJIT suite test is started.

[1]: https://github.com/LuaJIT/LuaJIT-test-cleanup/tree/014708b/test



Closes #4064
Part of #4473

Reviewed-by: Sergey Kaplun <skaplun@tarantool.org>
Reviewed-by: Sergey Ostanevich <sergos@tarantool.org>
Signed-off-by: Igor Munkin <imun@tarantool.org>

On my ARM system __CC_ARM is not defined. Checking whether we don't run on non-x86 may also make supporting other architectures (like POWER or RISC-V) easier.

This commit updates test-run with an update of tarantool-python, which
eliminates the test-run submodule from tarantool-python. In other words,
we have no test-run submodule inside test-run anymore.

The goal is to don't confuse luacheck with old revisions of test-run /
tarantool-python code and to make recursive clone faster. See [1] and
[2] for details.

[1]: https://github.com/tarantool/test-run/pull/268
[2]: https://github.com/tarantool/tarantool-python/commit/dccf18fd07ac684195fdd386d2a5285b9582d380

Now `make luacheck` gracefully handles different cases[^1]: in-source
and out-of-source build (within the source tree or outside), current
working directory as a real path or with symlink components.

As result of looking into those problems I filed the issue [1] against
luacheck. It seems, there are problems around absolute paths with
symlinks components.

[^1]: We have the similar problems with LuaJIT's luacheck rule. They
      will be fixed in a separate patch.

[1]: https://github.com/mpeterv/luacheck/issues/208



Reviewed-by: Sergey Bronnikov <sergeyb@tarantool.org>
Reviewed-by: Igor Munkin <imun@tarantool.org>