Added cleanup functionality for the meta files.
Script may have the following situations:

 - package files removed at S3, but it still registered:
   Script stores and registers the new packages at S3 and
   removes all the other registered blocks for the sames
   files in meta files.

 - package files already exists at S3 with the same hashes:
   Script passes it with warning message.

 - package files already exists at S3 with the old hashes:
   Script fails w/o force flag, otherwise it stores and
   registers the new packages at S3 and removes all the other
   registered blocks for the sames files in meta files.

Added '-s|skip_errors' option flag to skip errors on changed
packages to avoid of exits on script run.

Part of #4839

Returned the static build based on Dockerfile to gitlab-ci release branches
testing after the issues with missed openssl version fixed at PR #4831.

Follow up #4831

(cherry picked from commit b09f44b856e91f1006bd5b3e226a7be0b65b7859)

Found that static build based on Dockerfile used external link
and missed that it was removed, like it was in #4830. To avoid
of the same issues the cache for building the Dockerfile was
disabled with '--no-cache' option at docker build command.

Follow up #4830

(cherry picked from commit 1207821e4fc18312a9916d81a55a8eacd75a67b3)

By default lcov collects line coverage only. It would be useful to
collect function and branch coverage too.

Closes #4888

Test was a flaky from the beginning 39d0e427
Time of building indexes varies from time to time and the problem was due to
abcense of synchronization in index building and checking numbers of these
indexes.

Fixes #4353

relay_subscribe_f sets a recovery trigger notifying tx when a full log
is read and gc consumer corresponding to the replica may be advanced.
Since anonymous replicas do not have gc consumers, the trigger isn't
added for them. However, on relay exit, the trigger deletion depends
on replica->anon flag. This is buggy in case relay stalls on exit due to
replica disconnect. Replica has time to reconnect and register as a
normal instance, hence its replica->anon flag will be false by the time
we check whether to clear triggers or not, effectively making us to
clear unset triggers and segfault.

Fix this by initializing the triggers with trigger_create(), which
allows a trigger_clear() call, even if the triggers are not set, and
omit the replica->anon check.

Closes #4731

Acked-by: Cyrill Gorcunov <gorcunov@gmail.com>

Sign local space requests with a zero instance id. This allows to split
local changes aside from the changes, which should be visible to the whole
cluster, and stop sending NOPs to replicas to follow local vclock.

Moreover, it fixes the following bug with local spaces and replication.
In a situation when there are a master and a replica set up, replica may
still write to local spaces even if it's read-only. Local space
operations used to promote instance's lsn before this patch. Eventually,
master would have vclock {1:x} and replica'd have vclock {1:x, 2:y},
where y > 0, due to local space requests performed by the replica.
If a snapshot happens on replica side, replica will delete it's .xlog
files prior to the snapshot, since no one replicates from it and thus it
doesn't have any registered GC consumers.
From this point, every attempt to configure replication from replica to
master will fail, since master will try to fetch records which account
for the difference in master's and replica's vclocks: {1:x} vs {1:x,2:y},
even though master will never need the rows in range {2:1} - {2:y},
since they'll be turned to NOPs during replication.

Starting from this patch, in the situation described above, replica's
clock will be {0:y, 1:x}, and, since local space requests are now not
replicated at all, master will be able to follow replica, turning the
configuration to master-master.

Closes #4114

The current WAL GC implementation tracks consumers (i.e. remote replicas) by
their vclock signature, which is the sum of all vclock components.
This approach is wrong, and this can be shown by a little example.
The example will be a little synthetic, but it'll illustrate the problem.
Say, you have 2 masters, A and B with ids 1 and 2 respectively, and a replica C
with id 3. Say, С replicates from both A and B, and there is no replication
between A and B (say, the instances were reconfigured to not replicate from each
other). Now, say replica C has followed A and B to vclock {1:5, 2:13}. At the
same time, A has lsn 10 and B has lsn 15. A and B do not know about each other’s
changes, so A’s vclock is {1:10} and B’s vclock is {2:15}. Now imagine A does a
snapshot and creates a new xlog with signature 10. A’s directory will look like:
00…000.xlog 00…010.snap 00….010.xlog
Replica C reports its vclock {1:5, 2:13} to A, A uses the vclock to update the
corresponding GC consumer. Since signatures are used, GC consumer is assigned a
signature = 13 + 5 = 18. This is greater than the signature of the last xlog on
A (which is 10), so the previous xlog (00…00.xlog) can be deleted (at least A
assumes it can be). Actually, replica still needs 00…00.xlog, because it
contains rows corresponding to vclocks {1:6} - {1:10}, which haven’t been
replicated yet.

If instead of using vclock signatures, gc consumers used vclocks, such a problem
wouldn’t arise. Replica would report its vclock {1:5, 2:13}. The vclock is NOT
strictly greater than A’s most recent xlog vclock ({1:10}), so the previous log
is kept until replica reports a vclock {1:10, 2:something} or {1:11, …} and so
on.

Rewrite gc to perform cleanup based on finding minimal vclock
components present in at least one of the consumer vclocks instead of
just comparing vclock signatures.

Prerequisite #4114

If an anonymous replica is promoted to a normal one and becomes
replication master later, its vclock contains a non-empty zero
component, tracking local changes on this replica from the time when it
had been anonymous. No need to pollute joining instance's vclock with
our non-empty 0 component.
When an anonymous replica reports its status to a remote instance it
should also hide its 0-th vclock component.

This is needed for backward compatibility with old instances, which
don't ignore 0th vclock component coming from a remote instance by
default.

In order to do so, introduce a new function - vclock_size_ignore0(),
which doesn't count 0th clock component, and patch xrow_encode_vclock()
to skip 0th clock component if it's present.

Also make sure that new instances ignore 0th vclock component coming
from an unpatched remote instance.

Follow-up #3186
Prerequisite #4114

API is different from box.session.push() - sync argument was
removed. It will disappear from Lua API as well, because it just
is not needed here. Session is omitted as well. Indeed, a user
can't push to a foreign session, and the current session can be
obtained inside box_session_push(). And anyway session is not in
the public C API.

Internally dump into iproto is done using obuf_dup(), just like
tuple_to_obuf() does. obuf_alloc() would be a bad call here,
because it wouldn't be able to split the pushed data into several
obuf chunks, and would cause obuf fragmentation.

Dump into plain text behaves just like a Lua push - it produces a
YAML formatted text or Lua text depending on output format. But to
turn MessagePack into YAML or Lua text an intermediate Lua
representation is used, because there are no a MessagePack -> YAML
and MessagePack -> Lua text translators yet.

Closes #4734

@TarantoolBot document
Title: box_session_push() C API

There is a new function in the public C API:
```C
    int
    box_session_push(const char *data, const char *data_end);
```

It takes raw MessagePack, and behaves just like Lua
`box.session.push()`.

The policy for check of luarocks flags has been changed
(moved from tarantoolctl to luarocks).
Chdir has been moved to luarocks.

Closes #4629

@TarantoolBot document
Title: Update tarantoolctl rocks
tarantoolctl rocks commands has been added:
build
config
download
init
lint
new_version
purge
which
write_rockspec

https://github.com/luarocks/luarocks/wiki/luarocks

- Add the chdir option for make.

box.session.storage is a general-purpose table, which can be
used by user. Therefore, we shouldn't store any internal details
in it.

Needed for #4686

box.session.push() encodes data as a YAML document independent on
the current console output format. This patch adds handling for Lua
as well.

Closes #4686

Currently if a user wants to change session setting with SQL, one has
to execute UPDATE query like:
[[UPDATE "_session_settings" SET "value" = true WHERE "name" = 'name']]
However, direct access to system spaces isn't considered to be a good practice.
To avoid that and a bit simplify user's life, we introduce SQL shortcut command
SET SESSION.

Closes #4711

@TarantoolBot document
Title: API for accessing _session_settings space.
There are two ways of updating values of session settings:
via Lua and SQL.

Lua:
box.session.settings is a table, which is always accessible
to user. The syntax is the following:
`box.session.settings.<setting_name> = <new_value>`.

Example of usage:
```
tarantool> box.session.settings.sql_default_engine
---
- memtx
...

tarantool> box.session.settings.sql_default_engine = 'vinyl'
---
...

```

The table itself represents the (unordered) result of select
from _session_settings space.

SQL:
Instead of typing long UPDATE query one can use the SET SESSION command:
`box.execute([[SET SESSION "<setting_name>" = <new_value>]])`.
Note, that this query is case sensitive so the name must be quoted.
Also, SET SESSION doesn't provide any implicit casts, so <new_value> must
be of the type corresponding to the setting being updated.

Example:
```
tarantool> box.execute([[set session "sql_default_engine" = 'memtx']])
---
- row_count: 1
...

tarantool> box.execute([[set session "sql_defer_foreign_keys" = true]])
---
- row_count: 1
...

```

- space_object:update() is hard to use for configuring session settings,
so we provide box.session.settings table, which can be used in a much more
native way.

- Prior to this patch sql settings were not accessible before box.cfg()
call, even though these flags can be set right after session creation.

Part of #4711

Currently we have an array of modules and each module has
its own array of session_settings. This patch merges these
into one array, as long as it turned out, that we cannot
represent every setting as a part of some module.

Part of #4711

It is now possible to create an index over UUID values, returned by
`uuid.new()`.

Closes #4268
Closes #2916

@TarantoolBot document
Title: Document uuid field type.

There's a new field type -- UUID, it accepts values returned by
`uuid.new()`.

The index may be either unique or non-unique, nullable or non-nullable,
and may be a primary key.

The values in an index are ordered lexicographically by their string
representation.

To create an index over a uuid field for space `test`, say:
```
box.space.test:create_index("pk", {parts={1, 'uuid'}})
```
Now you may insert uuids into the space:
```
tarantool> box.space.test:insert{uuid.new()}
---
- [e631fdcc-0e8a-4d2f-83fd-b0ce6762b13f]
...

tarantool> box.space.test:insert{uuid.fromstr('64d22e4d-ac92-4a23-899a-e59f34af5479')}
---
- [64d22e4d-ac92-4a23-899a-e59f34af5479]
...

tarantool> box.space.test:select{}
---
- - [64d22e4d-ac92-4a23-899a-e59f34af5479]
  - [e631fdcc-0e8a-4d2f-83fd-b0ce6762b13f]
...

```

A special format for encoding UUIDs to MsgPack is introduced.
It is supported by both lua and C encoders/decoders, and it is now
possible to insert UUIDs into spaces, but only into unindexed fields
without format for now.

Prerequisite #4268

@TarantoolBot document
Title: Internals: msgpack format for UUID

UUID values share the MessagePack type with decimals:
both use MP_EXT. A new subtype is introduced for UUIDs,
MP_UUID = `0x02`
UUID is encoded as follows:
```
    +--------+---------+-----------+
    | MP_EXT | MP_UUID | UuidValue |
    +--------+---------+-----------+
```
Since UUID is 16 bytes in size, the header, MP_EXT, is always the same:
`0xd8`. MP_UUID = `0x02` follows. The header is followed by the 16
bytes of the UuidValue.

UuidValue consists of 11 fields, which are encoded as big endian
unsigned integers in the following order: `time_low` (4 bytes), `time_mid`
(2 bytes), `time_hi_and_version` (2 bytes), `clock_seq_hi_and_reserved` (1
byte), `clock_seq_low` (1 byte), `node[0], ..., node[5]` (1 byte each).

The total size of such a representation is 18 bytes, whereas storing
uuids as strings requires from 34 (when '-'s are ommitted) to 38 bytes
per UUID, giving a 2x space usage improvement.

Expose the code used to check UUID variant to a separate function:
tt_uuid_validate(). The function will be used for msgpack uuid decoding
when checking whether the buffer contains a valid uuid value.

Prerequisite #4268

mpstream is part of core library, which, since the introduction of
messagepack extension types, leads to circular dependencies between core
and any other library having extension type encoding/decoding methods
(e.g. uuid library which will soon be expanded with such methods).

Prerequisite #4268

I guess a compiler assumes that the code inside vfork may use the stack
slot that corresponds the variable and so clobber it. The recent commit
07a07b3c ('popen: decouple logger fd
from stderr') adds read from this variable after vfork() in the parent
process.

The warning is produced on RelWithDebInfo build with LTO enabled on GCC
9.2.0 (locally) and on GCC 8.3.0 (in CI).

Part of #4031

Acked-by: Cyrill Gorcunov <gorcunov@gmail.com>

popen read / write functions are written in C and intended to be used
from C: the contract is to return -1 at failure and set an entry to the
diagnostics area. However a C++ exception from coio read / write
functions would pass over a popen function stack frame.

The solution is to use the recently introduced coio exception safe
functions.

Part of #4031

Acked-by: Cyrill Gorcunov <gorcunov@gmail.com>

The popen implementation is written in C and uses coio read / write
functions. If an exception occurs, it'll pass through the C code. It
should be catched to proceed correctly.

We usually have foo() and foo_xc() (exception) functions when both
variants are necessary. Here I added non-conventional *_noxc() functions
as the temporary solution to postpone refactoring of the code and all
its usages.

Part of #4031

Acked-by: Cyrill Gorcunov <gorcunov@gmail.com>

The comment was added in 52765de6, but
becomes non-actual since 1.6.6-21-gc74abc786 ('Implement special
TimedOut exception type and use it in coio and latch.')

Part of #4031

Acked-by: Cyrill Gorcunov <gorcunov@gmail.com>

Our usual convention for C code is to return a negative value at failure
and set an entry to the diagnostics area.

When code uses this convention consistently, it is much easier to handle
failures when using it: you always know where to find an error type and
message and how to pass the error to a C or Lua caller.

See also the previous commit ('popen: add missed diag_set in
popen_signal/delete').

Part of #4031

Acked-by: Cyrill Gorcunov <gorcunov@gmail.com>

The function already checks flags to find out whether the file
descriptor should be available for reading / writing. When it is so, the
corresponding fd is great or equal to zero.

The further commits will add missed diagnostics for IO functions and it
is hard to write a meaningful error message for a situation that is not
possible. Moreover, we would obligated to document the error as one of
possible failures in a function contract (while it can't occur).

Part of #4031

Acked-by: Cyrill Gorcunov <gorcunov@gmail.com>

Before this change popen_read_timeout() waits until a passed buffer will
be fully filled (or until EOF / timeout / IO error occurs). Now it waits
for any amount of data (but at least one byte).

It allows to communicate with an interactive child program: write, read,
repeat.

Part of #4031

Acked-by: Cyrill Gorcunov <gorcunov@gmail.com>

The buffer is for reading, we're not intend to change it.

Part of #4031

Acked-by: Cyrill Gorcunov <gorcunov@gmail.com>

The default logger configuration writes logs to stderr.

Popen implementation holds a logger fd until execve() to be able to
write debug entries or information about a failure from a child. However
when popen flags requires to close stderr in the child, the logger fd
becomes closed: logging will fail.

Another problem appears when a user want to capture stderr and
tarantool's log level is set to debug (7). Since the logger uses stderr
and it is fed to the parent using a pipe, the logger output will not
shown on the 'real' stderr, but will be captured together with child's
program debugging output.

This commit duplicates a logger file descriptor that allows to close or
redirect child's stderr without described side effects.

See also 86ec3a5c ('popen: add logging
in child process').

Areas for improvements:

* Copy logger fd at module initialization time instead of copying of
  each popen call.

Alternatives:

* Extend logger to allow to accumulate log entries in a buffer. Flush
  the buffer from the parent process. (It is possible since vfork does
  not split a virtual memory space).

Part of #4031

Acked-by: Cyrill Gorcunov <gorcunov@gmail.com>

It is necessary to decouple stderr from a logger file descriptor in the
popen implementation.

Part of #4031

Acked-by: Cyrill Gorcunov <gorcunov@gmail.com>

It is useful for debugging popen behaviour around file descriptors.

Part of #4031

Acked-by: Cyrill Gorcunov <gorcunov@gmail.com>

Lua API will use content of the diagnostics area to report an error to a
caller, so it is critical to always have proper diagnostics at failure.

Part of #4031

Acked-by: Cyrill Gorcunov <gorcunov@gmail.com>

After the previous commit ('popen: require popen handle to be non-NULL')
it turns out that popen_state() function always succeeds. There is no
reason to return a success / failure value from it.

Part of #4031

Acked-by: Cyrill Gorcunov <gorcunov@gmail.com>

Further commits will add proper entries into the diagnostics area for
failures inside popen functions. We should either report handle == NULL
case via the diagnostics area or ensure that the NULL handle case is not
possible.

The latter approach is implemented in this commit. There are two
reasons for this:

* This way simplifies function contracts (one less kind of failure).
* The popen Lua module (that will be implemented in the further commits)
  will not construct any logic using NULL as a handle. When 'NULL
  handle' error is not possible in the C API, it will be easier to
  verify that this failure is not possible the Lua API.

A user of the C API should take care to don't call those functions with
NULL handle.

Part of #4031

Acked-by: Cyrill Gorcunov <gorcunov@gmail.com>

Test that diag_raise doesn't happen if async transaction
fails inside replication procedure.

Side note: I don't like merging tests with patches in
general and I hate doing so for big tests with a passion
because it hides the patch code itself. So here is a
separate patch on top of the fix.

Test-of #4730

Acked-by: Serge Petrenko <sergepetrenko@tarantool.org>
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

Currently when transaction rollback happens we just drop an existing
error setting ClientError to the replicaset.applier.diag. This action
leaves current fiber with diag=nil, which in turn leads to sigsegv once
diag_raise() called right after applier_apply_tx():

 | applier_f
 |   try {
 |   applier_subscribe
 |     applier_apply_tx
 |       // error happens
 |       txn_rollback
 |         diag_set(ClientError, ER_WAL_IO)
 |         diag_move(&fiber()->diag, &replicaset.applier.diag)
 |         // fiber->diag = nil
 |       applier_on_rollback
 |         diag_add_error(&applier->diag, diag_last_error(&replicaset.applier.diag)
 |         fiber_cancel(applier->reader);
 |     diag_raise() -> NULL dereference
 |   } catch { ... }

Thus:
 - use diag_set_error() instead of diag_move() to not drop error
   from a current fiber() preventing a nil dereference;
 - put fixme mark into the code: we need to rework it in a
   more sense way.

Fixes #4730

Acked-by: Serge Petrenko <sergepetrenko@tarantool.org>
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

To make it a bit more readable.

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

For some reason the replica_by_id member (which is an
array of pointers) is allocated dynamically. Moreover
VCLOCK_MAX = 32 by now and extending it to some new
limit will require a way more efforts than just increase
the number.

Thus reserve memory for replica_by_id inside replicaset
statically. This allows to simplify code a bit and
drop calloc/free calls.

The former code comes from edd76a2a without any
explanation why the dynamic member is needed.

Acked-by: Konstantin Osipov <kostja.osipov@gmail.com>
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

In case if we're hitting memory limit allocating triggers
we should setup diag error to prevent nil dereference
in diag_raise call (for example from applier_apply_tx).

Note that there are region_alloc_xc helpers which are
throwing errors but as far as I understand we need the
rollback action to process first instead of immediate
throw/catch thus we use diag_set.

Acked-by: Sergey Ostanevich <sergos@tarantool.org>
Acked-by: Konstantin Osipov <kostja.osipov@gmail.com>
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>