Cleaned up the tests code according to the Lua style guide:
- made scoped variables local
- added os.exit call at the end of the test chunk
- adjusted the messages in test:ok calls

(cherry picked from commit 3c641629)

box_check_config() didn't check memtx_memory and vinyl_memory
upper bound. As a result, it was possible to set memory size
higher than what the quota allows as maximum.

That worked only when box.cfg() was called first time, because
quota_init() does not check its value. Subsequent box.cfg() calls
use quota_set(), which aborts the program if a size is too big.
Only in debug mode. In release quota_set() also worked with any
sizes.

Closes #4705

Reviewed-by: Igor Munkin <imun@tarantool.org>
Reviewed-by: Nikita Pettik <korablev@tarantool.org>
(cherry picked from commit 74cec5f5)

Use bit_count_u32() instead of plain __builtin_popcount() for filter
size, just as we do for vclock size now.

(cherry picked from commit 044d6d58)

We're using an unsigned int to hold vclock map, but there is no
guarantee that unsigned int will be 4 bytes in size to fit all the 32
vclock components. So use uint32_t instead and add an alias to it
vclock_map_t.

(cherry picked from commit e5679980)

We need to set a thread cancellation guard, because
another thread may cancel the current thread at a
really bad time (messages flush, mutex lock)

Fixes: #4127
(cherry picked from commit 63d75e92)

Since this build flag has been removed as a result of reverting the
tarantool/luajit@d4e985a, its definition in the corresponding Tarantool
cmake file is irrelevant.

Furthermore, considering the breakage faced in #4770 the following tests
are introduced:
* the check whether space __pairs metamethod is set to space.pairs to
  create a Lua Fun iterator that handles __pairs manually underneath.
* the check whether pairs builtin behaviour doesn't change when __pairs
  is set e.g. on space object.

Follow-up #4560
Closes #4770

Reviewed-by: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Signed-off-by: Igor Munkin <imun@tarantool.org>
(cherry picked from commit 7ec7ced6)

In #4684 it was found that box.tuple.* contained some private
functions: bless(), encode(), and is().

Bless() and encode() didn't make any sense for a user, so they
were hidden into box.internal.tuple.*.

But box.tuple.is() is actually a useful thing. It is harnessed in
the tests a lot, and is likely to be already used by customers,
because it is available in box.tuple.* for a long time. It is a
matter of time when someone will open a doc ticket saying that
box.tuple.is() is not documented. The patch makes it legally
public.

Follow-up #4684

@TarantoolBot document
Title: box.tuple.is()
```Lua
box.tuple.is(object)
```
A function to check whether a given object is a tuple cdata
object. Returns true or false. Never raises nor returns an error.

(cherry picked from commit adc98575)

box.tuple.bless, .encode, and .is are internal. Their
behaviour is not documented, and they may omit some checks for the
sake of speed, and can crash if used without thinking.

Nonetheless, despite they are not documented, curious users could
notice them in box.tuple.* output via autocompletion, for example.
And they could try to use them. This is not ok.

box.tuple.bless() being called by a user leads either to a crash,
or to a leak (because it is basically tuple reference counter
increment).

box.tuple.encode() is kind of a wrapper around msgpack, and users
should not touch it. It may change, may be removed. And is just
makes no sense except some rare cases in schema.lua.

bless() and encode() were used in schema.lua only, so the patch
simply moves them to box.internal.tuple.

box.tuple.is() is kept as is, because
- this is used in the tests a lot;
- it is totally safe;
- that function actually makes sense, and some users could have
  already started using it.

There is no a test, since nothing to test - bless() is not
available for users anymore (assuming no one is brave enough to
rely on box.internal).

Closes #4684

(cherry picked from commit b2b3fb21)

This code is called from C, so it shouldn't throw.

Closes #4753

Reviewed-by: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Reviewed-by: Nikita Pettik <korablev@tarantool.org>
(cherry picked from commit afe4007d)

Use 'filename' grep_log() option instead of custom log search.

Reviewed-by: Alexander Turenko <alexander.turenko@tarantool.org>
(cherry picked from commit 54606356)

Add filename option to grep_log() (#106).

The option can be used when it is not possible to grab 'box.cfg.log'
value from a tarantool instance: say, when the instance is stopped or
crashed.

Usage:

 | local test_run = require('test_run').new()
 | test_run:grep_log(node, what, bytes, {filename = <...>})

(cherry picked from commit c401880b)

Original port_tuple in C has 'int size;' field. It was
'size_t size' in Lua. Since sizeof(size_t) usually is
8, and sizeof(int) is 4, this was a really bad typo.

(cherry picked from commit 8bea83ed)

When tarantool tries to recover rtree from a snapshot and memtx_memory value
is lower than it has been when the snapshot was created, server suffers from
segmentation fault. This happens because there is no out of memory error
handling in rtree lib. In another words, we do not check the result of
malloc operation.
The execution flow in case of recovery uses different way and the secondary
keys are build in batches. That way has no checks and reservations.
The patch adds memtx_rtree_index_reserve implementation to make sure that any
memory allocation in rtree will fail. Although this gives us no additional
optimization as in case of memtx_tree, the memory reservation prevents
tarantool from segmentation fault. If there is not enough memory to be reserved
server will fail gently with the "Failed to allocate" error message.

Closes #4619

(cherry picked from commit 3b4fbdc0)

Despite what was stated in the documentation, netbox.connect was not always
equivalent to netbox.self. In particular, they converted tuple to different
types - table and cdata respectively.

The patch fixes the issue and covers all cases where netbox.self and connect
perform conversion of types - e.g., for box.error.

Closes #4513

(cherry picked from commit 0bcf9a59)

Calling prepare and execute did not update corresponding request statistics
in the box.stat table. This happened because methods that collect stats were
never called where they should have been.

Version for 2.2 only includes box.execute.

Closes #4756

- Make string to number conversions fail on NUL char.

(cherry picked from commit 12694d72)

- gdb: adjust the extension to be used with Python 2
- gdb: introduce luajit-gdb extension

(cherry picked from commit a81453b9)

libcurl has a built-in threaded resolver used for asynchronous DNS
requests, however, when DNS server is slow to respond, the request still
hangs tarantool until it is finished. The reason is that curl calls
thread_join on the resolving thread internally upon timeout, making the
calling thread hang until resolution has ended.
Use c-ares as an asynchronous resolver instead to eliminate the problem.

Closes #4591

(cherry picked from commit 23837076)

It was possible to leak user password through setting 'replication'
configuration option in first box.cfg invocation. This happened due
to unconditional logging in load_cfg function. The patch introduces
conditional logging.

Closes #4493

(cherry picked from commit 3ce08a3e9cf0386d93ecc694aa4c4f99056ae7ca)

When checking wheter rejoin is needed, replica loops through all the
instances in box.cfg.replication, which makes it believe that there is a
master holding files, needed by it, since it accounts itself just like
all other instances.
So make replica skip itself when finding an instance which holds files
needed by it, and determining whether rebootstrap is needed.

We already have a working test for the issue, it missed the issue due to
replica.lua replication settings. Fix replica.lua to optionally include
itself in box.cfg.replication, so that the corresponding test works
correctly.

Closes #4759

(cherry picked from commit dbcfaf70)

We have a mechanism for restoring rows originating from an instance that
suffered a sudden power loss: remote masters resend the isntance's rows
received before a certain point in time, defined by remote master vclock
at the moment of subscribe.
However, this is useful only on initial replication configuraiton, when
an instance has just recovered, so that it can receive what it has
relayed but haven't synced to disk.
In other cases, when an instance is operating normally and master-master
replication is configured, the mechanism described above may lead to
instance re-applying instance's own rows, coming from a master it has just
subscribed to.
To fix the problem do not relay rows coming from a remote instance, if
the instance has already recovered.

Closes #4739

(cherry picked from commit ed2e1430)

Add a filter for relay to skip rows coming from unwanted instances.
A list of instance ids whose rows replica doesn't want to fetch is encoded
together with SUBSCRIBE request after a freshly introduced flag IPROTO_ID_FILTER.

Filtering rows is needed to prevent an instance from fetching its own
rows from a remote master, which is useful on initial configuration and
harmful on resubscribe.

Prerequisite #4739, #3294

@TarantoolBot document

Title: document new binary protocol key and subscribe request changes

Add key `IPROTO_ID_FILTER = 0x51` to the internals reference.
This is an optional key used in SUBSCRIBE request followed by an array
of ids of instances whose rows won't be relayed to the replica.

SUBSCRIBE request is supplemented with an optional field of the
following structure:
```
+====================+
|      ID_FILTER     |
|   0x51 : ID LIST   |
| MP_INT : MP_ARRRAY |
|                    |
+====================+
```
The field is encoded only when the id list is not empty.

(cherry picked from commit 45de9907)

There is an assertion in vclock_follow `lsn > prev_lsn`, which doesn't
fire in release builds, of course. Let's at least warn the user on an
attempt to write a record with a duplicate or otherwise broken lsn, and
not follow such an lsn.

Follow-up #4739

(cherry picked from commit e0750262)

is_orphan status check is needed by applier in order to tell relay
whether to send the instance's own rows back or not.

Prerequisite #4739

(cherry picked from commit 7b83b73d)

After #4736 regression fix (in fact it just reverts the new logic in
small) it is possible again that a fiber's region may hold a memory for
a while, but release it eventually. When the used memory exceeds 128 KiB
threshold, fiber_gc() puts 'garbage' slabs back to slab_cache and
subtracts them from region_used() metric. But until this point those
slabs are accounted in region_used() and so in fiber.info() metrics.

This commit fixes flakiness of test cases of the following kind:

 | fiber.info()[fiber.self().id()].memory.used -- should be zero
 | <...workload...>
 | fiber.info()[fiber.self().id()].memory.used -- should be zero

The problem is that the first `<...>.memory.used` value may be non-zero.
It depends of previous tests that were executed on this tarantool
instance.

The obvious way to solve it would be print differences between
`<...>.memory.used` values before and after a workload instead of
absolute values. This however does not work, because a first slab in a
region can be almost used at the point where a test case starts and a
next slab will be acquired from a slab_cache. This means that the
previous slab will become a 'garbage' and will not be collected until
128 KiB threshold will exceed: the latter `<...>.memory.used` check will
return a bigger value than the former one. However, if the threshold
will be reached during the workload, the latter check may show lesser
value than the former one. In short, the test case would be unstable
after this change.

It is resolved by restarting of a tarantool instance before such test
cases to ensure that there are no 'garbage' slabs in a current fiber's
region.

Note: This works only if a test case reserves only one slab at the
moment: otherwise some memory may be hold after the case (and so a
memory check after a workload will fail). However it seems that our
cases are small enough to don't trigger this situation.

Call of region_free() would be enough, but we have no Lua API for it.

Fixes #4750.

(cherry picked from commit d6cf327f)

The bug was in an attempt to update a record in _space_sequence
in-place, to add field path and number. This was not properly
supported by the system space's trigger, and was banned in the
previous patch of this series.

But delete + tuple update + insert work fine. The patch uses them.

To test it the old disabled and heavily outdated
xlog/upgrade.test.lua was replaced with a smaller analogue, which
is supposed to be created separately for each upgrade bug.
According to the new policy of creating test files.

The patch tries to make it easy to add new upgrade tests and
snapshots. A new test should consist of fill.lua script to
populate spaces, snapshot, needed xlogs, and a .test.lua file.
Fill script and binaries should be in the same folder as test file
name, which is located in version folder. Like this:

 xlog/
 |
 + <test_name>.test.lua
 |
 +- upgrade/
    |
    +- <version>/
    |   |
    |   +-<test_name>/
    |     |
    |     +- fill.lua
    |     +- *.snap
    |     +- *.xlog

Version is supposed to say explicitly what a version files in
there have.

Closes #4771

(cherry picked from commit 6d45a41e)

Anyway this does not work for generated sequences. A proper
support of update would complicate the code and won't give
anything useful.

Part of #4771

(cherry picked from commit 1a84b80e)

box.internal.bootstrap() before doing anything turns off system
space triggers, because it is likely to do some hard changes
violating existing rules. And eliminates data from all system
spaces to fill it from the scratch.

Each time when a new space is added, its erasure and turning off
its triggers should have been called explicitly here. As a result
it was not done sometimes, by accident. For example, triggers
were not turned off for _sequence_data, _sequence,
_space_sequence.

Content removal wasn't done for _space_sequence.

The patch makes a generic solution which does not require manual
patching of trigger manipulation and truncation anymore.

The bug was discovered while working on #4771, although it is not
related.

(cherry picked from commit e1c7d25f)

In case if we unable to revert guard page back to
read|write we should never use such slab again.

Initially I thought of just put panic here and
exit but it is too destructive. I think better
print an error and continue. If node admin ignore
this message then one moment at future there won't
be slab left for use and creating new fibers get
prohibited.

In future (hopefully near one) we plan to drop
guard pages to prevent VMA fracturing and use
stack marks instead.

Reviewed-by: Alexander Turenko <alexander.turenko@tarantool.org>
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
(cherry picked from commit 8d53fadc)

Both madvise and mprotect calls can fail due to various
reasons, mostly because of lack of free memory in the
system.

We log such cases via say_x helpers but this is not enough.
In particular tarantool/memcached relies on diag error to be
set to detect an error condition:

 | expire_fiber = fiber_new(name, memcached_expire_loop);
 | const box_error_t *err = box_error_last();
 | if (err) {
 |	say_error("Can't start the expire fiber");
 |	say_error("%s", box_error_message(err));
 |	return -1;
 | }

Thus lets use diag_set() helper here and instead of macros
use inline functions for better readability.

Fixes #4722

Reported-by: Alexander Turenko <alexander.turenko@tarantool.org>
Reviewed-by: Alexander Turenko <alexander.turenko@tarantool.org>
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
(cherry picked from commit c6752297)

Before the patch there were the rules:
* float +/- double = double
* double +/- double = double
* float +/- float = float

The rules were applied regardless of values. That led to a problem
when float + float exceeding maximal float value could fit into
double, but was stored as an infinity.

The patch makes so that if a floating point arithmetic operation
result fits into float, it is stored as float. Otherwise as
double. Regardless of initial types.

This alongside saves some memory for cases when doubles can be
stored as floats, and therefore takes 4 less bytes. Although
these cases are rare, because any not integer value stored in a
double may have a long garbage tail in its fraction.

Closes #4701

(cherry picked from commit fef4fdfc)

Our S3 based repositories now reflect packagecloud.io repositories
structure.

It will allow us to migrate from packagecloud.io w/o much complicating
redirection rules on a web server serving download.tarantool.org.

Deploy source packages (*.src.rpm) into separate 'SRPM' repository
like packagecloud.io does.

Changed repository signing key from its subkey to public and moved it
to gitlab-ci environment.

Follows up #3380

(cherry picked from commit 4dee6890)

Enabled Tarantool performance testing on Gitlab-CI for release/master
branches and "*-perf" named branches. For this purpose 'perf' and
'cleanup' stages were added into Gitlab-CI pipeline.

Performance testing support next benchmarks:

- cbench
- linkbench
- nosqlbench (hash and tree Tarantool run modes)
- sysbench
- tpcc
- ycsb (hash and tree Tarantool run modes)

Benchmarks use scripts from repository:
http://github.com/tarantool/bench-run

Performance testing uses docker images, built with docker files from
bench-run repository:

- perf/ubuntu-bionic:perf_master           -- parent image with
                                              benchmarks only
- perf_tmp/ubuntu-bionic:perf_<commit_SHA> -- child images used for
                                              testing Tarantool sources

@Totktonada: Harness and workloads are to be reviewed.

(cherry picked from commit 87c68344)

Found that on 19.02.2020 APT repositories with packages
for Ubuntu 18.10 Cosmic were removed from Ubuntu archive:

 E: The repository 'http://security.ubuntu.com/ubuntu cosmic-security Release' does not have a Release file.
 E: The repository 'http://archive.ubuntu.com/ubuntu cosmic Release' does not have a Release file.
 E: The repository 'http://archive.ubuntu.com/ubuntu cosmic-updates Release' does not have a Release file.
 E: The repository 'http://archive.ubuntu.com/ubuntu cosmic-backports Release' does not have a Release file.

Also found the half a year old message about Ubuntu 18.10
Cosmic EOL:
 https://fridge.ubuntu.com/2019/07/19/ubuntu-18-10-cosmic-cuttlefish-end-of-life-reached-on-july-18-2019/

Removed the Ubuntu 18.10 Cosmic from gitlab-ci and
travis-ci testings.

(cherry picked from commit 961e8c5f)

The server used to crash when any option argument was passed with
a value concatenated to it, like this: '-lvalue', '-evalue'
instead of '-l value' and '-e value'.

However this is a valid way of writing values, and it should not
have crashed regardless of its validity.

The bug was in usage of 'optind' global variable from getopt()
function family. It is not supposed to be used for getting an
option's value. It points to a next argv to parse. Next argv !=
value of current argv, like it was with '-lvalue' and '-evalue'.

For getting a current value there is a variable 'optarg'.

Closes #4775

(cherry picked from commit 29cfd564)

os.setenv() and os.environ() are Lua API for

    extern char **environ;
    int setenv();

The Open Group standardized access points for environment
variables. But there is no a word about that environ never
changes. Programs can't relay on that. For example, addition of
a new variable may cause realloc of the whole environ array, and
therefore change of its pointer value. That was exactly the case
in os.environ() - it was using value of environ array remembered
when Tarantool started.

And os.setenv() could realloc the array and turn the saved pointer
into garbage.

Closes #4733

(cherry picked from commit 954d4bdc)

Revert "build: introduce LUAJIT_ENABLE_PAIRSMM flag"

Related to #4770

(cherry picked from commit 04dd6f43)

After 7fd6c809
(buffer: port static allocator to Lua) uri started to use
static_allocator - cyclic buffer that also is used in
several modules.

However situation when uri.format output is zero-length
string was not handled properly and ffi.string could
return data that was previously written in static buffer
because use as string terminator the first zero byte.

To prevent such situation let's pass result length explicitly.

Closes #4779

(cherry picked from commit 57f6fc93)

When json.decode is used with 2 arguments, 2nd argument seeps out to the
json configuration of the instance. Moreover, due to current
serializer.cfg implementation it remains invisible while checking
settings using json.cfg table.

This fixes commit 6508ddb7 ('json: fix
stack-use-after-scope in json_decode()').

Closes #4761

(cherry picked from commit f54f4dc0)

box_process_call/eval() in the end check if there is an
active transaction. If there is, it is rolled back, and
an error is set.

But rollback is not needed anymore, because anyway in
the end of the request the fiber is stopped, and its
not finished transaction is rolled back. Just setting
of the error is enough.

Follow-up #4662

(cherry picked from commit f5d51448)