This patch allows VARBINARY to be returned for user-defined LUA
functions. However, there are currently no values that can be
interpreted as VARBINARY by the serializer, so the only way to get a
VARBINARY result for user-defined LUA functions is to return a UUID or
DECIMAL. Both types are not supported by SQL and are treated as
VARBINARY.

Closes #6024

This patch allows VARBINARY to be returned for user-defined C functions.
There is currently no support for UUID and DECIMAL in SQL, so they are
also treated as VARBINARY.

Part of #6024

In 0e37af31 an optimization eliminating INSERT+DELETE and DELETE+INSERT
statements by the same key in write set was introduced. It is fine until
it comes for secondary index build. While we are building secondary
index we save current lsn, set on_replace trigger forwarding new
requests to the secondary index and copy row-by-row tuples (to be more
precise keys) to secondary index until lsn of tuple is less than the one
we preserved at the start. Now, if during index build we execute request
replacing key that hasn't been already transferred to secondary index,
we will get missing key in secondary index since:
a) In on_replace trigger replace is split into DELETE+INSERT and
   eliminated by mentioned optimization (the same concerns simple pair
   of DELETE+INSERT requests made in single transaction - so that they
   get into one write set);
b) It is skipped in the main loop transferring tuples from PK to SK
   since lsn of modified tuples is greater than saved lsn.

In this respect, we may get missing tuples in secondary index.
The proposed solution is quite trivial: we are able to track that index
is still being created (see previous commit) so we won't apply
INSERT+DELETE annihilation if index build is not finished.

Closes #6045

It tells whether LSM tree is currently being constructed, or is already
built and committed.

Needed for #6045

Remote node doing the subscribe might be from a different
replicaset.

Before this patch the subscribe would be retried infinitely
because the node couldn't be found in _cluster, and the master
assumed it must have joined to another node, and its ID should
arrive shortly (ER_TOO_EARLY_SUBSCRIBE).

The ID would never arrive, because the node belongs to another
replicaset.

The patch makes so the master checks if the peer lives in the same
replicaset. Since it is doing a subscribe, it must have joined
already and should have a valid replicaset UUID, regardless of
whether it is anonymous or not.

Correct behaviour is to hard cut this peer off immediately,
without retries.

Closes #6094
Part of #5613

This update offers one tiny change in the pretest_clean functionaly:
it'll not remove jit.dis_arm64 from the package.loaded table anymore.

Relates to #5983

Sometimes jobs on CI with Jepsen tests failed on installation
dependencies:

```
sudo -S -u root bash -c "cd /; env DEBIAN_FRONTEND=noninteractive apt-get install -y --force-yes apt-transport-https libzip4 ntpdate faketime"

STDIN:
null

STDOUT:
Reading package lists...
Building dependency tree...
Reading state information...

STDERR:
W: --force-yes is deprecated, use one of the options starting with --allow instead.
E: Unable to locate package libzip4
E: Unable to locate package ntpdate
E: Unable to locate package faketime
```

Problem looks as a flaky, I couldn't reproduce it locally.  I suspect
the root cause is an infrastructure problem and to get more details
about it I have enabled debug options in apt-get and added
`set -o errexit` as it is recommended in documentation [1]
(see Note section).

1. https://www.terraform.io/docs/language/resources/provisioners/remote-exec.html#argument-reference

Part of: https://github.com/tarantool/jepsen.tarantool/issues/87

It is possible that a new async transaction is added to the limbo
when there is an in-progress CONFIRM WAL write for all the pending
sync transactions.

Then when CONFIRM WAL write is done, it might see that the limbo
now in the first place contains an async transaction not yet
written to WAL. A suspicious situation - on one hand the async
transaction does not have any blocking sync txns before it and
can be considered complete, on the other hand its WAL write is not
done and it is not complete.

Before this patch it resulted into a crash - limbo didn't consider
the situation possible at all.

Now when CONFIRM covers a not yet written async transactions, they
are removed from the limbo and are turned to plain transactions.

When their WAL write is done, they see they no more have
TXN_WAIT_SYNC flag and don't even need to interact with the limbo.

It is important to remove them from the limbo right when the
CONFIRM is done. Because otherwise their limbo entry may be not
removed at all when it is done on a replica. On a replica the
limbo entries are removed only by CONFIRM/ROLLBACK/PROMOTE. If
there would be an async transaction in the first position in the
limbo queue, it wouldn't be deleted until next sync transaction
appears.

This replica case is not possible now though. Because all synchro
entries on the applier are written in a blocking way. Nonetheless
if it ever becomes non-blocking, the code should handle it ok.

Closes #6057

Currently `log` module accepts only numeric values of
logging levels. I turn `box.cfg` interface supports
symbolic names (such as 'fatal', 'crit' and etc).

Thus we should support the same in `log` module.

Closes #5882

Reported-by: Alexander Turenko <alexander.turenko@tarantool.org>
Acked-by: Alexander Turenko <alexander.turenko@tarantool.org>
Acked-by: Serge Petrenko <sergepetrenko@tarantool.org>
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

Found that commit message may consists of special characters which can
be used be Markdown as commands, like '`' or '```'. To avoid of it these
characters must be changed to some predefined names like for:

'\' - BACKSLASH
'`' - BACKTICK

Also added filter block to avoid of other not known symbols which we
could miss. This block converts commit message to HTML and then takes
only text from it.

Found that on self-hosted runners where CentOS 7 is the base OS,
'send-telegram-notify' action creates message with syntax error:

  --------------'\n't'\n'```'\n'')) ; \
  ^
  SyntaxError: unexpected character after line continuation character

It happened because of extra quotes at '\n' while it had to be \n.
To avoid of it the same message changes must be done as for OSX
hosts are doing. These changes should be done when self-hosted
runners uses RHEL or Fedora as base OS.

After commit:

  58fe0fcb ('github-ci: avoid of use container tags in actions')

We began to use not the docker containers, but native github hosts.
To avoid of permissions fails on native github actions runners apt
command must run using sudo. Added flag '-n|--non-interactive' to
sudo command to avoid prompting the user for input of any kind which
could hang it. Added '-y' flag to apt update command to accept changes.

For now python3 is used as the default python on all OS and it is
needed to enable it in send-telegram-notify action.

Found issue:

  Traceback (most recent call last):
    File "<string>", line 3, in <module>
  AttributeError: module 'urllib' has no attribute 'quote_plus'

In Python 3 quote_plus included into urllib.parse.

Check documentaion [1]:

  Note The urllib module has been split into parts and renamed in Python 3 to urllib.request, urllib.parse, and urllib.error.

Check the same issue [2].

This patch changes use of all needed routines just from 'urllib'.

Closes tarantool/tarantool-qa#112

[1]: https://docs.python.org/2/library/urllib.html
[2]: https://github.com/web2py/web2py/issues/1822

box.ctl.promote() used to assume that the last synchronous entry is
already written to WAL by the time it's called. This is not the case
when promote is executed on the limbo owner. The last synchronous entry
might still be en route to WAL.

In order to fix the issue, wait until all the limbo entries are written
to disk via wal_sync(). After this happens, it's safe to proceed to
gathering quorum in promote.

Closes #6032

Occasionally, test/app-tap/gh-5040-inter-mode-isatty-via-errinj.test.lua
failed because it used output file with the same name as
test/app-tap/gh-4983-tnt-e-assert-false-hangs.test.lua and the last one
didn't remove file after usage.

Added removal of output file to test for 4983 and also changed file names
to distinguish outputs of these tests better in case of failure.

Fixes tarantool/tarantool-qa#122

There are several updates in test-run:

* Unit testing and coverage collecting for test-run ([1]).
  (Plus a fixup for `--disable-schema-upgrade` for Python 3.5.)
* Allow to miss a timeout in test_run:wait_log(): it is 60 seconds by
  default ([2]).
* pretest_clean: add built-in `UUID()` SQL function ([3]).

[1]: https://github.com/tarantool/test-run/pull/283
[2]: https://github.com/tarantool/test-run/pull/306
[3]: https://github.com/tarantool/test-run/pull/307

It is strange to create a new fiber and see that it has yielded 100
times, when in fact it never actually did it.

The patch makes fiber->csw = 0 for each created fiber.

Follow-up #5799

If you want to get information or get csw (Context SWitch) of some fiber
you need to call fiber.info(), but it creates table with information about
all the fibers. This patch introduces fiber_object:info() and
fiber_object:csw() - functions to solve this problem.

Closes #5799

@TarantoolBot document
Title: introduce fiber_object:info() and fiber_object:csw()
```
-- fiber_object:info() is the same as fiber.info(), but show information only
about one alive fiber.
-- fiber_object:csw() show csw (Context SWitch) of alive fiber.
```

This patch moves MEM types from the 'u32 flags' field to the new
'enum mem_type type' field. Now, we can be sure that only one type is
set for MEM. In addition, it is now easier to distinguish MAP and ARRAY
from VARBINARY, and this makes it easier to add extension types - UUID
and DECIMAL.

Closes #4906

After this patch, the mem_is_bin() function will return 'true' only if
the value that the MEM contains is of type VARBINARY. This patch also
adds the mem_is_bin_ext() function, which is used to check if a MEM
contains value of type VARBINARY or value of types that are currently
considered VARBINARY extensions - MAP and ARRAY.

Part of #4906

This patch adds proper initialization for the MEM, which is used in the
aggregate functions min() and max().

Part of #4906

Lua json module used to have a global buffer for all encodings. It
was reused by each next encode().

This was not correct, because during encode() might happen a GC
step, which might call encode() again and spoil the global buffer.

The same problem was already fixed for the global static buffer in
scope of #5632. Similarly to that time, the patch makes Lua json
module use cord_ibuf to prevent "concurrent" usage of the buffer
data. The global buffer is deleted.

According to a few microbenchmarks it didn't affect the perf
anyhow.

Core part of the patch is strbuf changes. Firstly, its destruction
is now optional, cord_ibuf can free itself on a next yield.
Secondly, its reallocation algorithm is kept intact - ibuf is used
as an allocator, not as the buffer itself. This is done so as not
to be too intrusive in the third party module which might need an
upgrade to the upstream in the future.

Closes #6050

Script used to run Jepsen tests uses Terraform to prepare test environment.
When I made this script I had a wrong assumption about working directory
used by Terraform. Due to this sometimes job with Jepsen tests
failed on cleanup with message:

```
Error: Could not load plugin

Plugin reinitialization required. Please run "terraform init".

Plugins are external binaries that Terraform uses to access and manipulate
resources. The configuration provided requires plugins which can't be located,
don't satisfy the version constraints, or are otherwise incompatible.

Terraform automatically discovers provider requirements from your
configuration, including providers used in child modules. To see the
requirements and constraints, run "terraform providers".

Failed to instantiate provider
"registry.terraform.io/terraform-providers/openstack" to obtain schema:
unknown provider "registry.terraform.io/terraform-providers/openstack"
```

Terraform documentation describes how Terraform uses working directories [1]
and there are at least to ways to resolve an issue. First one is using always
one directory before running terraform subcommands. Second one is using option
`-chdir` in all terraform commands [2].

1. https://www.terraform.io/docs/cli/init/index.html
2. https://www.terraform.io/docs/cli/commands/index.html#switching-working-directory-with-chdir

Closes #6089

Prior to this patch, UUID was not part of SCALAR. However, this should
be changed to comply with the RFC "Consistent Lua/SQL types".

Closes #6042

@TarantoolBot document
Title: UUID is now part of SCALAR

The UUID field type is now part of the SCALAR field type. This means
that now values of type UUID can be inserted into the SCALAR field, and
these values can participate in the sorting of the SCALAR fields. The
order is as follows: boolean < number < string < varbinary < uuid.

This allows `xlog` Lua module to decode appropriate
types into symbolic form.

For example with the patch we should see raft and
promote types in output.

 | $ tarantoolctl cat 00000000000000000004.xlog
 | ---
 | HEADER:
 |   lsn: 2
 |   group_id: 1
 |   type: RAFT
 |   timestamp: 1621541912.4588
 | BODY:
 |   0: 3
 |   1: 4
 | ---
 | HEADER:
 |   lsn: 1
 |   replica_id: 4
 |   type: PROMOTE
 |   timestamp: 1621541912.4592
 | BODY:
 |   2: 0
 |   3: 0
 |   83: 3

Fixes #6088

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

Now test-run reports memory occupied by tests and their duration (see
[1] and [2]).

Aside of this, there is a tiny code health change around waiting for
`box.info.status` value: it should not affect a visible test-run
behaviour ([3]).

[1]: https://github.com/tarantool/test-run/issues/277
[2]: https://github.com/tarantool/test-run/issues/286
[3]: https://github.com/tarantool/test-run/issues/301

commit fbc04dfb
("static-build: use hashes of downloaded files") added hashes for
ExternalProject_Add() to avoid extra downloads on project rebuild.
Changes for libicu and libiconv were missed and this patch fixes it.

Follows-up #5761

Co-authored-by: Sergey Bronnikov <sergeyb@tarantool.org>

Since commit c9d88d5f ('Fix #984: add
jit.* library to the binary') all required modules implemented in Lua
are bundled (i.e. compiled into the executable as a C literal) into
Tarantool binary. While making Tarantool work on ARM64 platforms, it
turned out the arch-specific module (namely, jit/dis_arm64.lua) is not
bundled. Within this patch the missing sources are added and jit.dump
works fine on ARM64 hosts as a result.

Part of #5983
Relates to #5629
Follows up #984

Reviewed-by: Sergey Kaplun <skaplun@tarantool.org>
Reviewed-by: Sergey Ostanevich <sergos@tarantool.org>
Signed-off-by: Igor Munkin <imun@tarantool.org>

* FFI/ARM64/OSX: Fix vararg call handling.
* OSX/iOS: Handle iOS simulator and ARM64 Macs.
* build: pass sysroot to MacOS SDK

Closes #6065
Closes #6066
Part of #5629
Relates to #5983

Workflow 'debug_coverage' produces and uploads results to 'coveralls.io'
web site. Message about it can be shown in PR within each run was done.
This patch adds the ability to send message in available PR otherwise it
is skipped. Also added 'coverage.info' file to artifacts list.

Found that used 'coverallsapp/github-action' action checks if the
trigger was 'pull_request' used to run the workflow [1]. And only in
this way it writes results message to PR. Previously 'pull_request'
trigger was blocked to avoid duplication with 'push' trigger. To make
able to run workflow by any single trigger and to send message to PR
if it exits, it was implemented the following logic:

  - run workflows on both triggers 'push' and 'pull_request';
  - for workflow with 'push' trigger check if PR exists then skip all
    jobs otherwise continue running;
  - for workflow with 'pull_request' trigger continue running;

To avoid of issue coverallsapp/github-action#55 [2], sources checkout
should be based on 2nd version and later.

Closes #5644

[1]: https://github.com/coverallsapp/github-action/blob/master/src/run.ts#L38
[2]: https://github.com/coverallsapp/github-action/issues/55#issuecomment-644927165

The problem that is fixed by this test-run update is the following. A
test that is listed in the 'release_disabled' field in a suite.ini file
of a 'core = app' or a 'core = unittest' test suite was disabled at all,
not only for the release tarantool build. A 'core = tarantool' test
suite was not affected by the problem.

Disabled sql-tap/debug_mode_only.test.lua test, because it does not work
anymore. We plan to revisit this test in the scope of #3694.

See https://github.com/tarantool/test-run/issues/199

There are a few issues concerning build of coro on Mac M1.

Firstly, Mach-O assembler does not support .type directive. So let's put
it under ifndef __APPLE__ preprocessing guard.

Secondly, according to OS X calling convention symbols (at least those
which are not exported) are mangled with underscore ("_") prefix:

Undefined symbols for architecture arm64:
  "_coro_startup", referenced from:
      _coro_create in libcoro.a(coro.c.o)

The same concerns "abort" function call. So to call function from assembly
inline we have to specify exact name (i.e. prefixed with _).

Part of #5983

This test-run update offers several changes.

One changeset renews test-run dependencies (see [1]):

* 27e097e Eliminate six dependency
* 0868e26 Drop unused dependencies
* e698240 Relax dependency version requirements

Another one fixes --reproduce mode (see [2]):

* 9fe1cc9 Fix --reproduce mode for fragile tests

[1]: https://github.com/tarantool/test-run/pull/288
[2]: https://github.com/tarantool/test-run/issues/233

This fixes a nit in commit 6af47377
("fiber: use uint64_t for fiber IDs").

Since lua_pushfstring doesn't support %llu format [1] it
breaks the test

app-tap/gh-1700-abort-recording-on-fiber-switch.test.lua

[1] https://www.lua.org/manual/5.1/manual.html#lua_pushfstring



Follows up #5846

Reviewed-by: Igor Munkin <imun@tarantool.org>
Reviewed-by: Alexander V. Tikhonov <avtikhon@tarantool.org>
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
Signed-off-by: Igor Munkin <imun@tarantool.org>

Update of an absent field could crash when had 2 operations in
reversed order both on not specified fields.

The problem was that the rope item was created incorrectly.

Rope item is a range of array fields and consists of 2 parts:
xrow_update_field and a tail. The xrow field should describe
exactly one array field (first in the given range) and its update
operation. When no operation, it is NOP.

The tail includes the rest of the array fields not affected by
any operations yet.

But in the code it was so the rope item was created with its xrow
field covering multiple array fields, and with zero tail. As a
result, split of that item didn't work because it can't split an
xrow field.

The bug was in the nil-autofill for absent array fields. The patch
fixes it so the newly created item with nils has its xrow field
containing one nil, and the other nils in the tail. This allows to
split the item correctly if necessary.

Closes #6069

The position of first argument to check from is 6, not 0.
Looks like our tests with CFORMAT was simply not working
since commit 7d12d66e (Refactor macroses for compiler
builtins and function attributes).

Lets turn them all back.

Follow-up #5846

Reported-by: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

 - vclock_get returns int64_t
 - vclock_sum returns int64_t

Follow-up #5846

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

Use explicit cast for 64 bit number.

Follow-up #5846

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

Use explicit conversion for 64 bit number.

Follow-up #5846

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>