Currently, we don't check tuple format in before_replace triggers,
that's why some bugs happen if we don't use the triggers correctly.

Let's check tuple format before execution of before_replace triggers
and after each before_replace trigger. The check will be disabled
during recovery for backward compatibility.

Closes #6780

NO_DOC=bug fix

(cherry picked from commit 884b3ff3)

The test checks key validation in before_replace trigger, but it
uses replace() to check it, and, as we plan to check tuple format in
before_replace trigger, we need to use something that works with
key, not tuple: delete(), for example.

Part of #6780

NO_DOC=refactoring
NO_CHANGELOG=refactoring

(cherry picked from commit 512ba6d5)

A secondary index creation proceeds as follows:
 1. Build the new index by inserting statements from the primary index,
    see vinyl_space_build_index().
 2. Dump the new index and wait for the dump to complete.
 3. Commit the index creation record to the WAL.

While the new index is being dumped at step 2, new statements may be
inserted into the space. We need to insert those statements during
recovery, see vy_build_recover(). We identify such statements by
comparing LSN to vy_lsm::dump_lsn, see vy_build_recover_stmt().

It might occur that the newly built index is empty while the primary
index memory level isn't - if all statements cancel each other. In this
case, the secondary index won't be dumped during creation and its
dump_lsn will be set to -1, see the vy_lsm_is_empty() check in
vinyl_space_build_index(). This would break the assumption made on
recovery: that all statements with LSN > vy_lsm::dump_lsn should be
inserted into the secondary index. If a statement like this isn't
compatible with the new index, we will get a crash trying to insert it.

Let's fix this issue by skipping vy_build_recover() in case the new
secondary index was never dumped.

Closes #6778

NO_DOC=bug fix

(cherry picked from commit dadb8d70)

Follow-up c28a3b2f
("test: remove checksums from suite.ini config files")

Return a more meaningful message about a malformed opts to
select, count and pairs methods of index object. Do not fail
with luajit error.

Also allow to pass specific iterator as a box.index.{ALL,...} directly.
For example, `s:pairs(nil, box.index.GT)`.

Fixes #6501

NO_DOC=bug fix

(cherry picked from commit 3a98e079)

Previous behaviour was quite illogical and unexpected. If user
once defined strict mode at start of tests it's expected that all
subtests also will have strict mode. However before this patch it
wasn't so. And wasn't clear at start why enabled strict mode
didn't work.
After this patch subtests strict mode will be the same as for
parent. This behaviour wasn't tested anyhow and wasn't documented.

Follow-up #4125

@TarantoolBot document
Title: clarify 'strict' behaviour in tap tests
Defined for root tap object strict mode will be the same for all
subtests.

Example:
```lua
t = require('tap').test('123')
t.strict = true

t:is_deeply({a = box.NULL}, {}) -- false

t:test('subtest', function(t)
    t:is_deeply({a = box.NULL}, {}) -- also false
end)
```

(cherry picked from commit 676582c4)

Before this patch comparison of two `false` values produced
incorrect result. That because nil and false are consieded in the
same way by if operator. This patch fixes an issue and introduces
corresponding tests.

Closes #6386

NO_DOC=bugfix

(cherry picked from commit 840a45c4)

The luatest_helpers.lua module confuses developers because luatest
itself has the helpers.lua module as well. It is unclear which one
should be used and when because it is hard to remember what stuff
provides each of them.

Actually, luatest_helpers.lua has functions that can be safely moved to
more proper places (luatest_helpers/server.lua & instances/default.lua).

The `get_vclock` function logically belongs to server stuff, so it was
moved to server.lua and slightly changed (`eval` call was replaced by
`exec`).

The `wait_vclock` function logically belongs to server stuff as well, so
it was moved to server.lua without changes.

The `instance_uri` function logically belongs to server stuff as well,
so it was moved to server.lua and slightly changed (it was renamed into
`build_instance_uri` and redundant `instance_id` param was dropped).

The `default_cfg`, `env_cfg`, `box_cfg` functions logically belongs to
instance stuff, so they were moved to default.lua without changes.

Thus, the luatest_helpers.lua module was dropped and tests using this
module were updated.

Closes tarantool/luatest#192

NO_DOC=testing stuff
NO_CHANGELOG=testing stuff

(cherry picked from commit c3253d89)

The luatest_helpers/asserts.lua module confuses developers because
luatest itself has the asserts.lua module as well. It is unclear which
one should be used and when because it is hard to remember what stuff
provides each of them.

Actually, luatest_helpers/asserts.lua has functions that are not really
about asserting and can be safely moved to more proper places like the
luatest_helpers/server.lua and luatest_helpers/cluster.lua modules.

The `assert_server_follow_upstream` function logically belongs to server
stuff, so it was moved to server.lua and slightly changed (renamed into
`assert_follows_upstream` and `eval` calls were replaced by `exec`).

The `wait_fullmesh` function logically belongs to cluster stuff, so it
was moved to cluster.lua and slightly changed (now it accepts one param
that can include wait timeout and delay, otherwise, defaults applied).

Thus, the luatest_helpers/asserts.lua module was dropped and tests using
this module were updated.

Part of tarantool/luatest#192

NO_DOC=testing stuff
NO_CHANGELOG=testing stuff

(cherry picked from commit dbcb2d53)

It takes box.cfg config as an argument. And changes the argument
by adding a new key 'command'. If the caller wants to pass the
same box.cfg or slightly modified to several build_server() calls,
it won't work - all options will be the same on all instances.

For example:

    local cfg = {...}
    cfg.replication = {url1}
    cluster:build_server(cfg)
    cfg.replication = {url2}
    cluster:build_server(cfg)

It will not work. Both servers will get the same 'command' and the
same 'replication'.

(cherry picked from commit 33ea6824)

We have recently dropped 'checksum' machinery in test-run
(tarantool/test-run#321) and now there is no need to keep
checksums in suite.ini config files. So removing them.

Closes tarantool/tarantool-qa#152

(cherry picked from commit fc177b33)

* test: adapt tests checking loading bytecode files
* test: adapt disabled tests from PUC-Rio
* test: adapt tests checking traceback in tail call
* test: adapt test checking global environment

Closes #5687
Closes #5691
Closes #5703
Closes #5713
Part of #5870

NO_DOC=LuaJIT submodule bump
NO_CHANGELOG=no visible changes

Bump test-run to new version with the following improvements:

  - drop 'checksum' machinery (tarantool/test-run#316)
  - rerun fragile tests only with status == 'fail'

NO_DOC=testing stuff
NO_CHANGELOG=testing stuff

(cherry picked from commit a70797f1)

The function of the question (`tarantoolsqlIdxKeyCompare()`) compares a
tuple with a key. It extracts necessary fields from the tuple and runs
the compare function, which is basically `memcmp()` of two buffers (for
`varbinary` values -- numeric comparisons are different).

In the Debug build the function also re-verifies the result using a
separate comparison code. And here we compare two `memcmp()` return
values.

We shouldn't do it directly, because `memcmp()` only guarantees sign of
the result. Please, consider the linked issue for details.

NO_DOC=only debug build is affected
NO_CHANGELOG=only debug build is affected
NO_TEST=hard to create a stable reproducer

Fixes #6849

(cherry picked from commit 330100bf)

- Improve reporting from failed luatest tests (tarantool/test-run#319)
- Update luatest to 0.5.7 (tarantool/test-run#322)

(cherry picked from commit e4f9d040)

* gdb: add mmudata value to lj-gc output
* gdb: unwind Lua stack top part

It was added to the master by a patch that introduced a new feature so
we it wasn't backported. As a result, the tests were never run.

The is_multikey option must be wrapped in opts, because commit
66f5368b ("schema: allow to pass
is_multikey func opt without sub-table") wasn't backported.

From time to time we can see failures of one of `box-luatest` tests.
It's `gh_6794_recover_nonmatching_xlogs.test_ignore_with_force_recovery`
test:

    [021] box-luatest/gh_6794_recover_nonmatching_xlogs_>
    [021] not ok 2 box-luatest.gh_6794_recover_nonmatching_xlogs.test_ignore_with_force_recovery #
    [021] Rejected result file: /tmp/tnt/rejects/box-luatest/gh_6794_recover_nonmatching_xlogs.reject
    [021] [ fail ]

More detailed failure log:

    not ok 2	box-luatest.gh_6794_recover_nonmatching_xlogs.test_ignore_with_force_recovery
    #   .../tarantool/test/luatest_helpers/server.lua:99: Waiting for "readiness" on server master-UCg6PXluSqYt (PID 85920) timed out
    #   stack traceback:
    #   	.../tarantool/test/luatest_helpers/server.lua:99: in function 'wait_for_readiness'
    #   	.../tarantool/test/luatest_helpers/server.lua:146: in function 'start'
    #   	...t/box-luatest/gh_6794_recover_nonmatching_xlogs_test.lua:32: in function 'box-luatest.gh_6794_recover_nonmatching_xlogs.test_ignore_with_force_recovery'
    #   	...
    #   	[C]: in function 'xpcall'

If we take a look at the log of a tarantool server that is used for
testing, we will see the failure reason:

    2022-02-04 21:06:12.063 [85920] main/103/default.lua evio.c:240 E> SocketError: unlink, called on fd 27, aka unix/:(socket), peer of unix/:(socket): Address already in use
    2022-02-04 21:06:12.063 [85920] main/103/default.lua F> can't initialize storage: unlink, called on fd 27, aka unix/:(socket), peer of unix/:(socket): Address already in use

Both tests from the gh_6794_recover_nonmatching_xlogs_test.lua file used
the same alias (master) for the tarantool server, hence in both tests
the tarantool server used a socket with the same name (master.iproto).
So we had a race here: if the socket from the previous test was released
in time, our test succeeded, otherwise, failed.

To fix the issue it was decided to create tarantool servers with
different aliases. This gives us sockets with different names, hence no
socket intersection.

Closes tarantool/tarantool-qa#151

NO_CHANGELOG=test stuff
NO_DOC=testing stuff

(cherry picked from commit 3f86cd04)

All tests from the update_optimize.test.lua file were fully migrated
to the luatest framework. The old test files were deleted.

Closes #6556

NO_CHANGELOG=testing stuff
NO_DOC=testing stuff

(cherry picked from commit 1b0bd504)

On completion of compaction tasks we remove compacted run files created
after the last checkpoint immediately to save disk space. In order to
perform this optimization we compare the unused runs' dump LSN with the
last checkpoint's one.

But during replica's initial JOIN stage we set the LSN of all rows
received from remote master to 0 (see
box/box.cc/boostrap_journal_write). Considering that the LSN of an
initial checkpoint is also 0, our optimization stops working, and we
receive a huge disk space usage spike (as the unused run files will
only get removed when garbage collection occurs).

We should check the vinyl space engine's status and perform
our optimization unconditionally if we are in replica's initial JOIN
stage.

Closes #6568

(cherry picked from commit 599f0c13)

tarantool/luatest#214 adds copy directory semantics to
luatest_helpers.Server: reuse it in the scope of this test.

Follow-up #tarantool/luatest#214

NO_DOC=refactoring
NO_CHANGELOG=refactoring

(cherry picked from commit c32273ce)

Currently, one can only specify a work directory that will be directly
passed to box.cfg (via workdir option passed to luatest.Server, via the
TARANTOOL_WORKDIR environment variable, or via explicitly specifying
work_dir in box.cfg), which implies that the test server will be run in
the specified directory.

One might need to start a Tarantool instance from a certain snapshot or
set of xlogs — for instance, to test Tarantool over an older schema
version.

For diff-tests the test-run harness provided a 'workdir=' option when
calling inspector:cmd('create server ...'), which solved this issue by
copying the specified directory's contents into the test server's
actual working directory (see tarantool/test-run/blob/
d16cbdc2702e7d939d5d34e41730fd83d2c65879/lib/preprocessor.py#L274-L284).

To solve this issue using the luatest_helpers harness we introduce a
datadir option for luatest_helpers.Server with the same semantics.

Needed for #6571
Closes tarantool/luatest#214

NO_DOC=luatest helpers
NO_CHANGELOG=luatest helpers

(cherry picked from commit b42a27fc)

Because of the typo, the test fails on my localhost like this:

  There is no test with name 'test_ignore_with_force_recovery'
    but hook 'after_test' is defined for it

(I use luatest installed with luarocks).

Follow-up #6794
Follow-up 8c8b7739
  ("recovery: fix recovering from 1.6 xlogs/snaps")

(cherry picked from commit bf475693)

As soon as replica subscribes, relay starts scanning the first requested
WAL file to find the position from which to start replication. While the
scan is in progress, relay never sends heartbeats to the replica, so a
really long scan can make the replica timeout before relay finally finds
the point to replicate from.

Two similar problems were already addressed in
30ad4a55 ("relay: yield explicitly every N sent rows") and
17289440 ("recovery: make it yield when positioning in a WAL")

These patches do not fix the issue completely for relay: even though now
it yields while scanning a WAL, it doesn't send heartbeats anyway.

Fix the issue by not only yielding, but sending heartbeats as well while
WAL scan is in progress.

Follow-up #5979
Closes #6706

(cherry-picked from ee6de025)

Sent rows were counted explicitly before the patch
17289440 ("recovery: make it yield when positioning in a WAL").
Now all the row count details are hidden inside the xstream.

Remove the redundant field.

Follow-up #5979

(cherry-picked from ee6de025)

Tarantool 1.6 didn't rotate xlogs on snap creation.
This means, that Tarantool 1.6 data dir may contain files like
0...0.xlog and 0...01.snap, where 0...0.xlog covers changes both before
and after the snap creation.

New Tarantool versions fail to recover from such files: they recover the
data correctly, but fail to update the vclock accordingly.

Let's allow recovering from such files and update the vclock
accordingly, but only when force_recovery is set.

Patch 634f59c7 ("recovery: panic in case of recovery and replicaset
vclock mismatch") added the panic, which helped to discover the issue.
Before that patch Tarantool would silently recover with an outdated
vclock.

Also, while we're at it, refactor the check leading to a panic: it never
happens after hot_standby, so move it into "else" branch of hot_standby.

Closes #6794

(cherry picked from commit 8c8b7739)

This change splits the debug_coverage.yml workflow into two independent
workflows: debug.yml and coverage.yml.

The first one will be run on push to a branch and creating/updating
external pool requests. In other words, it will be run always.

The second one will be run on push to a branch only if the branch has
the `full-ci` suffix (some-branch-full-ci) or the `full-ci` label is set
on a pool request, internal or external - it doesn't matter.

Why do we need it?

1. The coverage testing is heavy testing due to enabling long tests
   and takes at least 2 times more time than debug build testing. So
   testing that is done at the first stage (lint, release, debug) will
   pass faster.
2. The coverage report as a pool request comment from coveralls.io can
   be obtained only if a pool request event happened. It is a limitation
   of the `coverallsapp/github-action` action [1]. So there is no sense
   to run the coverage workflow on push events. But the coverage report
   on push still can be seen on coveralls.io if the branch has the
   `full-ci` suffix.
3. The logic of the debug_coverage.yml workflow is tricky enough and no
   one understands how it works. So we would like to have an absolutely
   transparent and simple workflow.

[1] https://github.com/coverallsapp/github-action/tree/8cbef1dea373ebce56de0a14c68d6267baa10b44#coveralls-github-action

Closes tarantool/tarantool-qa#147

NO_DOC=ci
NO_CHANGELOG=ci

(cherry picked from commit 5fdc1ea2)

According to the readline library documentation, the callback passed to
rl_callback_handler_install is supposed to free the string passed to it:

> As with readline(), the handler function should free the line when it
> it finished with it.

( See https://tiswww.case.edu/php/chet/readline/readline.html#SEC41 )

Our callback leaks it.

Closes #6817

NO_DOC=bug fix

(cherry picked from commit 2bbd13b4)

A memtx tree iterator remembers the last tuple returned to the user and
its hint so that it can restore iteration if the index is changed. To
prevent the tuple from being freed, it references it. The problem is
it's not enough for a functional index, because the latter allocates key
parts separately from tuples (it stores pointers to them in memtx tree
hints). As a result, if a tuple is deleted from the tree, its key parts
will be immediately freed, even if the tuple itself is referenced. Since
key parts are necessary to restore an iterator, this results in a use
after free bug.

To fix the issue, let's store a copy of the current tuple key part in
the iterator along with the tuple. If a key part is small, the copy is
stored in a preallocated fixed-size buffer, otherwise it's allocated
with malloc.

Closes #6786

(cherry picked from commit 51d0a9b2)

So as not to set tree_iterator::current directly. Currently, we only
need to ref/unref tuple when we update current, but in the future we
will also alloc/free func index key part, a pointer to which is stored
in a hint.

Needed for #6786

(cherry picked from commit d8a352b9)

These functions are used only by memtx functional indexes and they
depend on neither tuple_format nor tuple. No need to keep them in
tuple_format_vtab.

Needed for #6786

(cherry picked from commit c7d38b34)

Fix static build for macOS 11.5 or higher.
On macOS SDK ver. 11.5 some `*.dylib` files was replaced with `*.tbd`.
So we replace `libunwind.dylib` on `libunwind.tbd`.
Because of macOS 10.15 support being dropped
conditional is not needed.

Closes #6052

(cherry picked from commit a7724d8f)

* Actually implement maxirconst trace limit.
* Fix string.char() recording with no arguments.

Closes #6371
Part of #6548

Now the tarantool/testing:debian-stretch image from Docker Hub is used.

Closes tarantool/tarantool-qa#136

(cherry picked from commit eadadbe8)

Add GitHub Actions workflows for Fedora 35
Closes: #6692

NB: removed aarch64 variant for 2.8 and 1.10 branches.

(cherry picked from commit d3d04b49)

ev_timer.at was used as timeout. But after ev_timer_start() it
turns into the deadline - totally different value.

The patch makes sure ev_timer.at is not used in raft at all.

To test that the fakeev subsystem is patched to start its time not
from 0. Otherwise ev_timer.at often really matched the timeout
even for an active timer.

(cherry picked from commit e51c61ae)

It used to crash if done during election on a node voted for
anybody, it is a candidate, it doesn't know a leader yet, but has
a WAL write in progress.

Thus it could only happen if the term was bumped by a message from
a non-leader node and wasn't flushed to the disk yet.

The patch makes the reconfig check if there is a WAL write in
progress. Then don't do anything.

Could also check for volatile vote instead of persistent, but it
would create the same problem for the case when started writing
vote for self and didn't finish yet. Reconfig would crash.

(cherry picked from commit 82757e55)

* Enable parallel invocation of fragile tests

(cherry picked from commit 91d8fef5)

The commit fixes the following test failure:

```
[082] vinyl/gh-4810-dump-during-index-build.test.lua                  Test timeout of 310 secs reached	[ fail ]
[082]
[082] Test failed! Result content mismatch:
[082] --- vinyl/gh-4810-dump-during-index-build.result	Thu Dec  9 05:31:17 2021
[082] +++ /build/usr/src/debug/tarantool-2.10.0~beta1.324.dev/test/var/rejects/vinyl/gh-4810-dump-during-index-build.reject	Thu Dec  9 06:51:03 2021
[082] @@ -117,34 +117,3 @@
[082]  for i = 1, ch:size() do
[082]      ch:get()
[082]  end;
[082] - | ---
[082] - | ...
[082] -
...
```

The test hangs waiting for the test fibers to exit. There are two test
fibers - one builds an index, another populates the test space. The
latter uses pcall so it always returns. The one that builds an index,
however, doesn't. The problem is index build may fail because it builds
a unique index while the fiber populating the space may insert
non-unique values. Fix this by building a non-unique index instead,
which should never fail. To reproduce the issue the test checks is fixed
one can build any index, unique or non-unique, so it should be fine.

Closes #5508

(cherry picked from commit 5cd399b7)