It didn't have a single fail path. That led to some amount of code
duplication, and it complicated future patches where the journal
entries are going to get a proper error reason instead of default
-1 without any details.

The patch is a preparation for #6027 where it is wanted to have
more detailed errors on journal entry/transaction fail instead
of ER_WAL_IO for everything. Sometimes it can override a real
error like a cascade rollback, or a transaction conflict.

Part of #6027

It used to simply return -1 and set a diag only when OOM happened
inside.

The caller was forced either to ignore the result or set its own
diag regardless of what really happened.

The patch makes journal_write() set a correct diag error when it
returns -1. The only implementation to change was
wal_write_async(). The other implementations always return 0.

Part of #6027

The script name was too long. It was also used as a name for the
unix socket file on which the replica listens. As a result, the
test couldn't start, at least on my machine.

Besides, the script was not any different from the existing
replica.lua, except a couple of not important settings.

The patch drops it and makes gh-4730-applier-rollback.test.lua use
replica.lua. Now it can run on my machine.

Done as a preparation for #6027, which is slightly related to the
test - it is also about errors in applier and their display.

It was called ER_CHECKPOINT_ROLLBACK but was set only when there
is a cascade rollback in WAL.

The new error name is going to be used in the next patches, where
not only checkpoint can fail due to a cascade rollback.

Part of #6027

`VERSION` files in small subproject and in tarantool are treated as C++
standard library on a filesystem with case-insensitive names. So we have
to delete the root of tarantool project from `include_directories` in
tarantool CMake. Also we have to change `include_directories` in tarantool
CMake from the root of `small` project to `include` subfolder in `small`
project.
Closes #6076

* build: fix tarantool build failure on xcode 12.5

In scope of this commit new GitHub Actions workflows for testing
Tarantool on M1 hosts are added:

  Release: .github/workflows/osx_arm64_11_2.yml
  Debug: .github/workflows/osx_debug_arm64_11_2.yml

Since GitHub Actions uses x86_64 environment by default on M1 targets,
'arch -arm64' prefix is specified in GitHub Actions workflow to make all
commands in .travis.mk run in ARM64 environment.

Introduced a new temporary target in .travis.mk Makefile to run only
specific LuaJIT test suites on M1. Now it runs only the following LuaJIT
test targets:
* PUC-Rio-Lua-5.1-tests
* lua-Harness-tests
* tarantool-tests

Python 3.9 is installed by default on M1 hosts, but gevent is required
for Tarantool tests, which installation fails with the following error:

  Using cached gevent-21.1.2.tar.gz (5.9 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... error
    ERROR: Command errored out with exit status 1:
     command: /opt/homebrew/opt/python@3.9/bin/python3.9 /opt/homebrew/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /var/folders/b0/1vlv5rvn77x2rn6zbl2p4tqr0000gp/T/tmpyy59ae2p
         cwd: /private/var/folders/b0/1vlv5rvn77x2rn6zbl2p4tqr0000gp/T/pip-install-msbf7_vz/gevent_c2956687bb0d4de9bfb5f0660da759ee
    Complete output (42 lines):
      ...
      File "/private/var/folders/b0/1vlv5rvn77x2rn6zbl2p4tqr0000gp/T/pip-build-env-1lesbbxi/overlay/lib/python3.9/site-packages/cffi/api.py", line 48, in __init__
        import _cffi_backend as backend
    ImportError: dlopen(/private/var/folders/b0/1vlv5rvn77x2rn6zbl2p4tqr0000gp/T/pip-build-env-1lesbbxi/overlay/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so, 2): no suitable image found.  Did find:
    	/private/var/folders/b0/1vlv5rvn77x2rn6zbl2p4tqr0000gp/T/pip-build-env-1lesbbxi/overlay/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so: mach-o, but wrong architecture
    	/private/var/folders/b0/1vlv5rvn77x2rn6zbl2p4tqr0000gp/T/pip-build-env-1lesbbxi/overlay/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so: mach-o, but wrong architecture

This issue is described in gevent/gevent#1721. Fortunately, gevent can
be successfully installed via Python 3.8, hence to avoid this failure,
python3 is pinned to the specific version (i.e. python@3.8) until the
mentioned issue is resolved.

Closes tarantool/tarantool-qa#120
Relates to #6068

To avoid targets duplication in the later use of .travis.mk file, it was
decided to parameterize OSX jobs and move all fine tuning manipulations
related to these pipelines from .travis.mk to the corresponding GitHub
Actions workflows.

* ARM64: Fix xpcall() error case (really).
* ARM64: Fix xpcall() error case.
* test: add arch-specific skipcond for memprof
* ARM, ARM64, PPC: Fix TSETR fallback.

Closes #6084
Closes #6093
Part of #5629

All iproto threads listening same socket, and if user change listen
address, this socket is closed in each iproto thread. This patch fix
this error, now socket is closed only in main thread, and in other threads
we are only stop listening, without socket closing. Also this patch fix
error, related to the fact, that tarantool did not delete the unix socket
path, when it's finishing work.

In previous version `box_listen` and all associated functions raise
exception in case of error. Now, in case of error, they set diag and
return -1. It's necessary for graceful resources release in case of
error.

Previously `box_check_uri` raise exception in case of error.
Now, in case of error, it set diag and return -1. This patch
needs for graceful error handling.

Add {if_not_exists = true} to box.schema.user.grant to prevent
error when server restart.

In c++11 it's ok to made initialization in struct declaration,
but this is not done anywhere else in our code.

At the moment, user can set any number of iproto threads, which leads
to incorrect behavior if the specified number of threads is less than
or equal to zero or too large. Added check for user input of the number
of iproto threads - value must be > 0 and less than or equal to 1000.

Closes #6005

@TarantoolBot document
Title: Add check for user input of the number of iproto threads
Added check for user input of the number of iproto threads - value
must be > 0 and less than or equal to 1000.

This patch fixes inaccuracy in Tarantool build configuration introduced
by commit 07c83aab ('build: adjust
LuaJIT build system'). That patch sets LUAJIT_USE_ASSERT and
LUAJIT_USE_APICHECK options for Debug build instead of LUA_USE_ASSERT
and LUA_USE_APICHECK respectively. This patch fixes these typos.

Follows up #4862

Reviewed-by: Sergey Ostanevich <sergos@tarantool.org>
Reviewed-by: Igor Munkin <imun@tarantool.org>
Signed-off-by: Igor Munkin <imun@tarantool.org>

There was a bug that a new replica at join to a election-enabled
cluster sometimes tried to register on a non-leader node which
couldn't write to _cluster, so the join failed with ER_READONLY
error.

Now in scope of #5613 the algorithm of join-master selection is
changed. A new node looks for writable members of the cluster to
use a join-master. It will not choose a follower if there is a
leader.

Closes #6127

The algorithm of looking for an instance to join the replicaset
from didn't take into account that some of the instances might be
not bootstrapped but still perfectly available.

As a result, a ridiculous situation could happen - an instance
could connect to a cluster with just read-only instances, but it
could have itself with box.cfg{read_only = false}. Then instead of
failing or waiting it just booted a brand new cluster. And after
that the node just started complaining about the others having a
different replicaset UUID.

The patch makes so a new instance always prefers a bootstrapped
join-source to a non-boostrapped one, including self. In the
situation above the new instance now terminates with an error.

In future hopefully it should start a retry-loop instead.

Closes #5613

@TarantoolBot document
Title: IPROTO_BALLOT rework and a new field

A couple of fields in `IPROTO_BALLOT 0x29` used to have values not
matching with their names. They are changed.

* `IPROTO_BALLOT_IS_RO 0x01` used to mean "the instance has
  `box.cfg{read_only = true}`". It was renamed in the source code
  to `IPROTO_BALLOT_IS_RO_CFG`. It has the same code `0x01`, and
  the value is the same. Only the name has changed, and in the doc
  should be too.

* `IPROTO_BALLOT_IS_LOADING 0x04` used to mean "the instance has
  finished `box.cfg()` and it has `read_only = true`". The name
  was wrong therefore, because even if the instance finished
  loading, the flag still was false for `read_only = true` nodes.
  Also such a value is not very suitable for any sane usage.
  The name was changed to `IPROTO_BALLOT_IS_RO`, the code stayed
  the same, and the value now is "the instance is not writable".
  The reason for being not writable can be any: the node is an
  orphan; or it has `read_only = true`; or it is a Raft follower;
  or anything else.

And there is a new field.

`IPROTO_BALLOT_IS_BOOTED 0x06` means the instance has finished its
bootstrap or recovery.

The patch refactors the algorithm of finding a join-master (in
replicaset_find_join_master()) to use scores instead of multiple
iterations with different criteria.

The original code was relatively fine as long as it had only
one parameter to change - whether should it skip
`box.cfg{read_only = true}` nodes.

Although it was clear that it was "on the edge" of acceptable
complexity due to a second non-configurable parameter whether a
replica is in read-only state regardless of its config.

It is going to get more complicated when the algorithm will take
into account the third parameter whether an instance is
bootstrapped.

Then it should make decisions like "among bootstrapped nodes try
to prefer instances not having read_only=true, and not being in
read-only state". The easiest way to do so is to use
scores/weights incremented according to the instance's parameters
matching certain "good points".

Part of #5613

The new field reports whether the instance has finished its
bootstrap/recovery, or IOW has finished box.cfg().

The new field will help in fixing #5613 so as not to try to join
to a replicaset via non-bootstrapped instances if there are
others.

The problem is that otherwise, if all nodes are booted but
are read-only, new instances bootstrap their own independent
replicaset. It would be better to just fail and terminate the
process than do such a bizarre action.

Part of #5613

Is_loading in the ballot used to mean the following: "the instance
did not finish its box.cfg() or has read_only = true". Which is
quite a strange property.

For instance, it was 'true' even if the instance is not really
loading anymore but has read_only = true.

The patch renames it to 'is_ro' (which existed here before, but
also with a wrong meaning).

Its behaviour is slightly changed to report the RO state of the
instance. Not its read_only. This way it incorporates all the
possible RO conditions. Such as not finished bootstrap, having
read_only = true, being a Raft follower, and so on.

The patch is done in scope of #5613 where the ballot is going to
be extended and used a bit differently in the join-master search
algorithm.

Part of #5613

Rename the member to show its actual meaning. It is not the
real RO state of the instance. Only how it is configured.

It can happen that the instance is read_only = false, but still is
in RO state due to other reasons.

The patch is done in scope of #5613 where the ballot is going to
be extended and used a bit differently in the join-master search
algorithm.

Part of #5613

Firstly, rename it to replicaset_find_join_master(). Now, when
there is Raft with a concept of an actual leader, the function
name becomes confusing.

Secondly, do not access ballot member in struct applier in such a
long way - save the ballot pointer on the stack. This is going to
become useful when in one of the next patches the ballot will be
used more.

Part of #5613

If Raft state machine sees the current leader has explicitly
resigned from its role, it starts a new election round right away.

But in the code starting a new round there was an assumption that
there is no a volatile state. There was, in fact.

The patch makes the election start code use the volatile state to
bump the term. It should be safe, because the other nodes won't
receive it anyway until the new term is persisted.

There was an alternative - do not schedule new election until the
current WAL write ends. It wasn't done, because would achieve the
same (the term would be bumped and persisted) but with bigger a
latency.

Another reason is that if the leader would appear and resign
during WAL write on another candidate, in the end of its WAL write
the latter would see 0 leader and would think this term didn't
have one yet. And would try to elect self now, in the current
term. It makes little sense, because it won't win - the current
term had already had a leader and the majority of votes is
already taken.

Closes #6129

After this patch, the tt_uuid_from_strl() function will check the length
of the given string before converting it to a UUID.

Follow up #5886

This patch introduces a new SQL built-in function UUID().

Closes #5886

@TarantoolBot document
Title: SQL built-in function UUID()

SQL built-in function UUID() takes zero or one argument. If no argument
is specified, a UUID v4 is generated. If the version of the UUID to
generate is specified as an argument, the function returns the new UUID
of the given version. Currently only version 4 of UUID is supported.

This patch introduces UUID to SQL. UUID is now available as a new field
type.

Part of #5886

@TarantoolBot document
Title: Field type UUID is now available in SQL

The UUID field type is now available in SQL. This means that we can
create spaces and indexes with UUID, use it in SELECT, UPDATE and
DELETE. UUID can be accepted and returned by built-in functions and
user-defined functions.

According to the comparison rules, there will be no implicit casting in
the comparison. This rule also applies to UUID values: if a value is not
part of a SCALAR field, it cannot be compared to a value of any other
type. If the value is in a SCALAR field, it can be compared to any other
scalar value according to the comparison rules for a SCALAR field.

In case a UUID value is used in an operation that is not a comparison,
it can be implicitly converted to STRING or VARBINARY.

If a STRING or VARBINARY value is used in an operation that is not a
comparison, it can be implicitly converted to a UUID.

UUID value can always be explicitly converted to STRING or VARBINARY.

A STRING or VARBINARY value can be explicitly converted to a UUID if it
conforms to the UUID standard.

This patch allows VARBINARY to be returned for user-defined LUA
functions. However, there are currently no values that can be
interpreted as VARBINARY by the serializer, so the only way to get a
VARBINARY result for user-defined LUA functions is to return a UUID or
DECIMAL. Both types are not supported by SQL and are treated as
VARBINARY.

Closes #6024

This patch allows VARBINARY to be returned for user-defined C functions.
There is currently no support for UUID and DECIMAL in SQL, so they are
also treated as VARBINARY.

Part of #6024

In 0e37af31 an optimization eliminating INSERT+DELETE and DELETE+INSERT
statements by the same key in write set was introduced. It is fine until
it comes for secondary index build. While we are building secondary
index we save current lsn, set on_replace trigger forwarding new
requests to the secondary index and copy row-by-row tuples (to be more
precise keys) to secondary index until lsn of tuple is less than the one
we preserved at the start. Now, if during index build we execute request
replacing key that hasn't been already transferred to secondary index,
we will get missing key in secondary index since:
a) In on_replace trigger replace is split into DELETE+INSERT and
   eliminated by mentioned optimization (the same concerns simple pair
   of DELETE+INSERT requests made in single transaction - so that they
   get into one write set);
b) It is skipped in the main loop transferring tuples from PK to SK
   since lsn of modified tuples is greater than saved lsn.

In this respect, we may get missing tuples in secondary index.
The proposed solution is quite trivial: we are able to track that index
is still being created (see previous commit) so we won't apply
INSERT+DELETE annihilation if index build is not finished.

Closes #6045

It tells whether LSM tree is currently being constructed, or is already
built and committed.

Needed for #6045

Remote node doing the subscribe might be from a different
replicaset.

Before this patch the subscribe would be retried infinitely
because the node couldn't be found in _cluster, and the master
assumed it must have joined to another node, and its ID should
arrive shortly (ER_TOO_EARLY_SUBSCRIBE).

The ID would never arrive, because the node belongs to another
replicaset.

The patch makes so the master checks if the peer lives in the same
replicaset. Since it is doing a subscribe, it must have joined
already and should have a valid replicaset UUID, regardless of
whether it is anonymous or not.

Correct behaviour is to hard cut this peer off immediately,
without retries.

Closes #6094
Part of #5613

This update offers one tiny change in the pretest_clean functionaly:
it'll not remove jit.dis_arm64 from the package.loaded table anymore.

Relates to #5983

Sometimes jobs on CI with Jepsen tests failed on installation
dependencies:

```
sudo -S -u root bash -c "cd /; env DEBIAN_FRONTEND=noninteractive apt-get install -y --force-yes apt-transport-https libzip4 ntpdate faketime"

STDIN:
null

STDOUT:
Reading package lists...
Building dependency tree...
Reading state information...

STDERR:
W: --force-yes is deprecated, use one of the options starting with --allow instead.
E: Unable to locate package libzip4
E: Unable to locate package ntpdate
E: Unable to locate package faketime
```

Problem looks as a flaky, I couldn't reproduce it locally.  I suspect
the root cause is an infrastructure problem and to get more details
about it I have enabled debug options in apt-get and added
`set -o errexit` as it is recommended in documentation [1]
(see Note section).

1. https://www.terraform.io/docs/language/resources/provisioners/remote-exec.html#argument-reference

Part of: https://github.com/tarantool/jepsen.tarantool/issues/87

It is possible that a new async transaction is added to the limbo
when there is an in-progress CONFIRM WAL write for all the pending
sync transactions.

Then when CONFIRM WAL write is done, it might see that the limbo
now in the first place contains an async transaction not yet
written to WAL. A suspicious situation - on one hand the async
transaction does not have any blocking sync txns before it and
can be considered complete, on the other hand its WAL write is not
done and it is not complete.

Before this patch it resulted into a crash - limbo didn't consider
the situation possible at all.

Now when CONFIRM covers a not yet written async transactions, they
are removed from the limbo and are turned to plain transactions.

When their WAL write is done, they see they no more have
TXN_WAIT_SYNC flag and don't even need to interact with the limbo.

It is important to remove them from the limbo right when the
CONFIRM is done. Because otherwise their limbo entry may be not
removed at all when it is done on a replica. On a replica the
limbo entries are removed only by CONFIRM/ROLLBACK/PROMOTE. If
there would be an async transaction in the first position in the
limbo queue, it wouldn't be deleted until next sync transaction
appears.

This replica case is not possible now though. Because all synchro
entries on the applier are written in a blocking way. Nonetheless
if it ever becomes non-blocking, the code should handle it ok.

Closes #6057

Currently `log` module accepts only numeric values of
logging levels. I turn `box.cfg` interface supports
symbolic names (such as 'fatal', 'crit' and etc).

Thus we should support the same in `log` module.

Closes #5882

Reported-by: Alexander Turenko <alexander.turenko@tarantool.org>
Acked-by: Alexander Turenko <alexander.turenko@tarantool.org>
Acked-by: Serge Petrenko <sergepetrenko@tarantool.org>
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

Found that commit message may consists of special characters which can
be used be Markdown as commands, like '`' or '```'. To avoid of it these
characters must be changed to some predefined names like for:

'\' - BACKSLASH
'`' - BACKTICK

Also added filter block to avoid of other not known symbols which we
could miss. This block converts commit message to HTML and then takes
only text from it.

Found that on self-hosted runners where CentOS 7 is the base OS,
'send-telegram-notify' action creates message with syntax error:

  --------------'\n't'\n'```'\n'')) ; \
  ^
  SyntaxError: unexpected character after line continuation character

It happened because of extra quotes at '\n' while it had to be \n.
To avoid of it the same message changes must be done as for OSX
hosts are doing. These changes should be done when self-hosted
runners uses RHEL or Fedora as base OS.

After commit:

  58fe0fcb ('github-ci: avoid of use container tags in actions')

We began to use not the docker containers, but native github hosts.
To avoid of permissions fails on native github actions runners apt
command must run using sudo. Added flag '-n|--non-interactive' to
sudo command to avoid prompting the user for input of any kind which
could hang it. Added '-y' flag to apt update command to accept changes.

For now python3 is used as the default python on all OS and it is
needed to enable it in send-telegram-notify action.

Found issue:

  Traceback (most recent call last):
    File "<string>", line 3, in <module>
  AttributeError: module 'urllib' has no attribute 'quote_plus'

In Python 3 quote_plus included into urllib.parse.

Check documentaion [1]:

  Note The urllib module has been split into parts and renamed in Python 3 to urllib.request, urllib.parse, and urllib.error.

Check the same issue [2].

This patch changes use of all needed routines just from 'urllib'.

Closes tarantool/tarantool-qa#112

[1]: https://docs.python.org/2/library/urllib.html
[2]: https://github.com/web2py/web2py/issues/1822