ER_TUPLE_FOUND message shows only space and index, let's also show old
tuple and new tuple.

This commit changes error message in code and in tests. Test sql/checks
and sql-tap/aler remain the same due to problems in showing their old
and new tuples in error message.

Closes #5567

Add field name to field mismatch error message.

Part of #4707

Add got type to field mismatch error message.

Part of #4707

Previously tuple_field_u32 and tuple_next_u32 stored uint64_t value in
uint32_t field. This commit fixes it.

Part of #4707

Bump `feedback_version` to 7 and introduce a new field: `feedback.events`.
It holds a counter for every event we may choose to register later on.

Currently the possible events are "create_space", "drop_space",
"create_index", "drop_index".

All the registered events and corresponding counters are sent in a
report in `feedback.events` field.

Also, the first registered event triggers the report sending right away.
So, we may follow such events like "first space/index created/dropped"

Closes #5750

Feedback daemon used to generate report before waiting (for an hour by
default) until it's time to send it. Better actualize the reports and
generate them right when it's time to send them.

Part of #5750

Send the first report as soon as instance's initial configuration
finishes.

Part of #5750

feedback_daemon.send() will come in handy once we implement triggers to
dispatch feedback after some events, for example, right on initial
instance configuration.

So, it's not a testing method anymore, hence the new name.

Part of #5750

We are going to send feedback right after initial `box.cfg{}` call, so
include server uptime in the report to filter out short-living CI
instances.

Also, while we're at it, fix a typo in feedback_daemon test.

Prerequisite #5750

In commit 14fa5fd8 (cfg: support symbolic evaluation of
replication_synchro_quorum) we implemented support of
symbolic evaluation of `replication_synchro_quorum` parameter
and there is no easy way to obtain it current run-time value,
ie evaluated number value.

Moreover we would like to fetch queue length on transaction
limbo for tests and extend this statistics in future. Thus
lets add them.

Closes #5191

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

@TarantoolBot document
Title: Provide `box.info.synchro` interface

The `box.info.synchro` leaf provides information about details of
synchronous replication.

In particular `quorum` represent the current value of synchronous
replication quorum defined by `replication_synchro_quorum`
configuration parameter because it can be set as dynamic formula
such as `N/2+1` and the value depends on current number of replicas.

Since synchronous replication does not commit data immediately
but waits for its propagation to replicas the data sits in a queue
gathering `commit` responses from remote nodes. Current number of
entries waiting in the queue is shown via `queue.len` member.

A typical output is the following

``` Lua
tarantool> box.info.synchro
---
- queue:
    len: 0
  quorum: 1
...
```

The `len` member shows current number of entries in the queue.
And the `quorum` member shows an evaluated value of
`replication_synchro_quorum` parameter.

For self-hosted runners run w/o restart may need to kill hanged
processes that could be leaved from the previous workflows runs.
This patch implements in cleanup with kill for such processes in
'environment' actions which is called for all workflows and runs
before main steps. Cleanup searches for two names patterns of
running processes:

  - ' tarantool '
  - 'test-run.py '

Closes tarantool/tarantool-qa#114

Enabling ubuntu-20.04 hosts for packaging workflows found that DEB
package Github Actions workflows do not need to install createrepo
tool. Also found that createrepo is not ready for ubuntu-20.04 as
described in (till ubuntu-21.04 where it is available as the new
version of this tool named as 'createrepo_c' as DEB package):

  3a7c2102 ('github-ci: ubuntu-20.04 misses createrepo package')

To fix it added 'createrepo_c' build and installation from sources
and changed in update_repo tool 'createrepo' tool to 'createrepo_c'.
This patch is needed to use these workflows on self-hosted runners
which run under ubuntu-20.04 by default for now.

Also checking the patch on ubuntu-20.04 hosts got the following issue:

  Regenerated DEB file: pool/xenial/main/t/tarantool/tarantool-common_2.8.0.185.g4c3e0eb-1_all.deb

  <botocore.awsrequest.AWSRequest object at 0x7f7998a4ca90>

  <botocore.awsrequest.AWSRequest object at 0x7f627d965070>
  make: *** [.gitlab.mk:131: deploy] Error 255
  Error: Process completed with exit code 2.

Found that there is already issue exists in Github Actions on it [1].
Provided solution to setup AWS region in environment helped to
workaround the issue [2].

Closes tarantool/tarantool-qa#110
Closes tarantool/tarantool-qa#111

[1]: https://github.com/aws/aws-cli/issues/5234
[2]: https://github.com/aws/aws-cli/issues/5234#issuecomment-635459464

Created local composite action 'pack_and_deploy' for Tarantool packages
creation and deployment. It was created using local scripts in packages
workflows. It let to change common parts of packages creations and
deployment in each packaging workflow to call for this action. It helped
to consolidate all the instructions on packages creation and deployment
in a single place.

LuaJIT submodule is bumped to introduce the following changes:
* test: fix dynamic modules loading on MacOS
* test: make utils.selfrun usage easier
* test: remove excess dependency for tests target

Within this changeset SIP issues are worked around and dynamic modules
loading on MacOS is fixed. As a result LuaJIT tests can be enabled for
static build target on MacOS.

Closes #5959
Follows up #4862

Reviewed-by: Alexander V. Tikhonov <avtikhon@tarantool.org>
Reviewed-by: Sergey Kaplun <skaplun@tarantool.org>
Signed-off-by: Igor Munkin <imun@tarantool.org>

Found that running package removement command from S3:

  ./tools/update_repo.sh -o=fedora -d=30 -b=s3://tarantool_repo/live/1.10 -r=tarantool-1.10.9.108

it searched to remove:

  Searching to remove: s3://tarantool_repo/live/1.10/fedora/30/SRPMS/Packages/tarantool-1.10.9.108-1.fedora30.src.rpm

while it had to be:

  Searching to remove: s3://tarantool_repo/live/1.10/fedora/30/SRPMS/Packages/tarantool-1.10.9.108-1.fc30.src.rpm

where 'fc30' had to be instead of 'fedora30'. This patch fixes it.

For self-host runners, where work path with sources is saving between
different workflows runs, it is needed to be sure that all permissions
correct for running user to be sure that later call to sources checkout
by 'actions/checkout' action run by the same user won't fail on sources
cleanup.

Closes tarantool/tarantool-qa#102

Workflow 'coverity' produces and uploads results to 'coverity.com' web
site. Message about it can be shown in PR within each run was done.
This patch adds the ability to send message in available PR otherwise
it is skipped.

Part of #5644

Test checks possibility of recovery with force_recovery option.
For these purpose, snapshot is damaged in test and possibility of
recovery is checked. In previous version snapshot size was too small,
so sometimes in test, system data, needed for recovery was corrupted.
Also moved same code to functions, deleted one of two identical test
case (in previous version we write different count of garbage in the
middle of snapshot, it's meaningless, because result is almost the same,
only the amount of data that can be read from snapshot differs).
Follow-up #5422

List of changes in test-run:

* readme: add memcached to users ([PR #285][1])
* test: add integration tests ([PR #273][2])
* test: enable in continuous integration ([PR #273][2])
* Fix slowdown on Python 3 ([PR #290][3])

Updated .luacheckrc to exclude newly added 'core = tarantool' tests
(ones that check test-run itself).

[1]: https://github.com/tarantool/test-run/pull/285
[2]: https://github.com/tarantool/test-run/pull/273
[3]: https://github.com/tarantool/test-run/pull/290

It was used so as to recover synchronous auto-commit transactions
in an async way (not blocking the fiber). But it became not
necessary since #5874 was fixed. Because recovery does not use
auto-commit transactions anymore.

Closes #5194

Recovery used to be performed row by row. It was fine because
anyway all the persisted rows are supposed to be committed, and
should not meet any problems during recovery so a transaction
could be applied partially.

But it became not true after the synchronous replication
introduction. Synchronous transactions might be in the log, but
can be followed by a ROLLBACK record which is supposed to delete
them.

During row-by-row recovery, firstly, the synchro rows each turned
into a sync transaction. Which is probably fine. But the rows on
non-sync spaces which were a part of a sync transaction, could be
applied right away bypassing the limbo leading to all kind of the
sweet errors like duplicate keys, or inconsistency of a partially
applied transaction.

The patch makes the recovery transactional. Either an entire
transaction is recovered, or it is rolled back which normally
happens only for synchro transactions followed by ROLLBACK.

In force recovery of a broken log the consistency is not
guaranteed though.

Closes #5874

During recovery and xlog replay vinyl skips the statements already
stored in runs. Indeed, their re-insertion into the mems would
lead to their second dump otherwise.

But that results into an issue that the recovery transactions in
vinyl don't have a write set - their tx->log is empty. On the
other hand they still are added to the write set (xm->writers).
Probably so as not to have too many checks "skip if in recovery"
all over the code.

It works fine with single-statement transactions, but would break
on multi-statement transactions. Because the decision whether
need to add to the write set was done based on the tx's log
emptiness. It is always empty, and so the transaction could be
added to the write set twice and corrupt its list-link member.

The patch makes the decision about being added to the write set
based on emptiness of the list-link member instead of the log so
it works fine both during recovery and normal operation.

Needed for #5874

There was a bug in box_process_register. It decoded replica's vclock but
never used it when sending the registration stream. So the replica might
lose the data in range (replica_vclock, start_vclock).

Follow-up #5566

Both box_process_register and box_process_join had guards ensuring that
not a single rollback occured for transactions residing in WAL around
replica's _cluster registration.
Both functions would error on a rollback and make the replica retry
final join.

The reason for that was that replica couldn't process synchronous
transactions correctly during final join, because it applied the final
join stream row-by-row.

This path with retrying final join was a dead end, because even if
master manages to receive no ROLLBACK messages around N-th retry of
box.space._cluster:insert{}, replica would still have to receive and
process all the data dating back to its first _cluster registration
attempt.
In other words, the guard against sending synchronous rows to the
replica didn't work.

Let's remove the guard altogether, since now replica is capable of
processing synchronous txs in final join stream and even retrying final
join in case the _cluster registration was rolled back.

Closes #5566

Now applier assembles rows into transactions not only on subscribe
stage, but also during final join / register.

This was necessary for correct handling of rolled back synchronous
transactions in final join stream.

Part of #5566

applier->last_row_time is updated in applier_read_tx_row, which's called
at least once per each subscribe loop iteration. So there's no need to
have a separate last_row_time update inside the loop body itself.

Part of #5566

Once apply_synchro_row() failed, applier_apply_tx() would simply raise
an error without unlocking replica latch. This lead to all the appliers
hanging indefinitely on trying to lock the latch for this replica.

In scope of #5566

The new routine, called apply_plain_tx(), may be used not only by
applier_apply_tx(), but also by final join, once we make it
transactional, and recovery, once it's also turned transactional.

Also, while we're at it. Remove excess fiber_gc() call from
applier_subscribe loop. Let's better make sure fiber_gc() is called on
any return from applier_apply_tx().

Prerequisite #5874
Part of #5566

Introduce a new routine, set_next_tx_row(), which checks tx boundary
violation and appends the new row to the current tx in case everything
is ok.

set_next_tx_row() is extracted from applier_read_tx() because it's a
common part of transaction assembly both for recovery and applier.

The only difference for recovery will be that the routine which's
responsible for tx assembly won't read rows. It'll be a callback ran on
each new row being read from WAL.

Prerequisite #5874
Part-of #5566

Since the introduction of synchronous replication it became possible for
final join to fail on master side due to not being able to gather acks
for some tx around _cluster registration.

A replica receives an error in this case: either ER_SYNC_ROLLBACK or
ER_SYNC_QUORUM_TIMEOUT. The errors lead to applier retrying final join,
but with wrong state, APPLIER_REGISTER, which should be used only on an
anonymous replica. This lead to a hang in fiber executing box.cfg,
because it waited for APPLIER_JOINED state, which was never entered.

Part-of #5566

In swim Lua code none of the __serialize methods checked the
argument type assuming that nobody would call them directly and
mess with the types. But it happened, and is not hard to fix, so
the patch does it.

The serialization functions are sanitized for the swim object,
swim member, and member event.

Closes #5952

In Lua swim object's method member_by_uuid() could crash if called
with no arguments. UUID was then passed as NULL, and dereferenced.

The patch makes member_by_uuid() treat NULL like nil UUID and
return NULL (member not found). The reason is that
swim_member_by_uuid() can't fail. It can only return a member or
not. It never sets a diag error.

Closes #5951

We should remove a tag after fetching of a remote repository.
Ported the following commits:

  0f575e01 ('gitlab-ci: fix tag removal for a branch push job')
  0f564f34 ('gitlab-ci: remove tag from pushed branch commit')

Drop a tag that points to a current commit (if any) on a job triggered
by pushing to a branch (as against of pushing a tag). Otherwise we may
get two jobs for the same x.y.z-0-gxxxxxxxxx build: one is run by
pushing a branch and another by pushing a tag. The idea is to hide the
new tag from the branch job as if a tag would be pushed strictly after
all branch jobs for the same commit.

Closes tarantool/tarantool-qa#103

The key difference between lbox_encode_tuple_on_gc() and
luaT_tuple_encode() is that the latter never raises a Lua error, but
passes an error using the diagnostics area.

Aside of the tuple leak, the patch fixes fiber region's memory 'leak'
(till fiber_gc()). Before the patch, the memory that is used for
serialization of the key is not freed (region_truncate()) when the
serialization fails. It is verified in the gh-5388-<...> test.

While I'm here, added a test case that just verifies correct behaviour
in case of a key serialization failure (added into key_def.test.lua).
The case does not verify whether a tuple leaks and it is successful as
before this patch as well after the patch. I don't find a simple way to
check the tuple leak within a test. Verified manually using the
reproducer from the linked issue.

Fixes #5388

Got warning message:

  [014] WARGING: unix socket's "/home/ubuntu/actions-runner/_work/tarantool/tarantool/test/var/014_box/gh-5422-broken_snapshot.socket-iproto" path has length 108 symbols that is longer than 107. That likely will cause failing of tests.

It caused the following fail:

  [038] Starting instance autobootstrap_guest1...
  [038] Start failed: builtin/box/console.lua:865: failed to create server unix/:/home/ubuntu/actions-runner/_work/tarantool/tarantool/test/var/038_replication/autobootstrap_guest1.socket-admin: No buffer space available

To avoid of it use vardir option in tests runs to decrease paths length.

Closes tarantool/tarantool-qa#104

Changed the following workflows:

  luacheck
  debug_coverage
  release*
  static_build
  static_build_cmake_linux

It was changed the OS in which the test run from debian to ubuntu.
Also changed the way how this OS was booted - before the change it
was booted as docker container using Github Actions tag from inside
the worklfows. And it caused all the workflow steps to be run inside
it. After the change no container run anymore on the running host.
Github Actions host uses for now with its native OS set in 'runs'
tag. It was decided to use the latest one OS `ubuntu-20.04` which is
already the default for 'ubuntu-latest' tag.

This change gave us the abilities to:
 - Remove extra container step in workflow.
 - Switch off swap using 'swapoff' command.
 - Use the same OS as Github Actions uses by default.
 - Setup our local hosts using Github Actions image snapshot.
 - Enable use of actions/checkout@v2.3.4 which is better than v1.
 - Light bootstrap of packages in local .*.mk makefile for:
     build: libreadline-dev libunwind-dev
     tests: pip install -r test-run/requirements.txt

Closes tarantool/tarantool-qa#101

As a follow-up to the previous patch, let's check also emptiness of the
vylog being removed. During vylog rotation all entries are squashed
(e.g. "delete range" annihilates "insert range"), written to the new
vylog and at the end of new vylog SNAPSHOT marker is placed. If the last
entry in the vylog is SNAPSHOT, we can safely remove it without
hesitation.  So it is OK to remove it even during casual recovery
process. However, if it contains rows after SNAPSHOT marker, removal of
vylog may cause data loss. In this case we still can remove it only in
force_recovery mode.

Follow-up #5823

Having data in different engines checkpoint process is handled this way:
 - wait_checkpoint memtx
 - wait_checkpoint vinyl
 - commit_checkpoint memtx
 - commit_checkpoint vinyl

In contrast to commit_checkpoint which does not tolerate fails (if
something goes wrong e.g. renaming of snapshot file - instance simply
crashes), wait_checkpoint may fail. As a part of wait_checkpoint for
vinyl engine vy_log rotation takes place: old vy_log is closed and new
one is created. At this moment, wait_checkpoint of memtx engine has
already created new *inprogress* snapshot featuring bumped vclock.
While recovering from this configuration, vclock of the latest snapshot
is used as a reference.

At the initial recovery stage (vinyl_engine_begin_initial_recovery),
we check that snapshot's vclock matches with vylog's one (they should be
the same since normally vylog is rotated along with snapshot). On the
other hand, in the directory we have old snapshot and new vylog (and new
.inprogress snapshot). In such a situation recovery (even in force mode)
was aborted. The only way to fix this dead end, user has to manually
delete last vy_log file.

Let's proceed with the same resolution while user runs force_recovery
mode: delete last vy_log file and update vclock value. If user uses
casual recovery, let's print verbose message how to fix this situation
manually.

Closes #5823

It is conditional injection that terminates execution calling assert(0)
if given condition is true. It is quite useful since allows us to
emulate situations when instance is suddenly shutdown: due to sigkill
for example.

Needed for #5823

Needed for #5823