The test case has two problems that appear from time to time and lead to
flaky fails. Those fails are look as shown below in a test-run output.

 | Test failed! Result content mismatch:
 | --- box/net.box.result	Mon Jun 24 17:23:49 2019
 | +++ box/net.box.reject	Mon Jun 24 17:51:52 2019
 | @@ -1404,7 +1404,7 @@
 |  ...
 |  test_run:grep_log('default', 'ER_INVALID_MSGPACK.*')
 | ---
 | -- 'ER_INVALID_MSGPACK: Invalid MsgPack - packet body'
 | +- 'ER_INVALID_MSGPACK: Invalid MsgPack - packet length'
 | ...
 | -- gh-983 selecting a lot of data crashes the server or hangs the
 | -- connection

'ER_INVALID_MSGPACK.*' regexp should match 'ER_INVALID_MSGPACK: Invalid
MsgPack - packet body' log message, but if it is not in a log file at a
time of grep_log() call (just don't flushed to the file yet) a message
produced by another test case can be matched ('ER_INVALID_MSGPACK:
Invalid MsgPack - packet length'). The fix here is to match the entire
message and check for the message periodically during several seconds
(use wait_log() instead of grep_log()).

Another problem is the race between writing a response to an iproto
socket on a server side and closing the socket on a client end. If
tarantool is unable to write a response, it does not produce the warning
re invalid msgpack, but shows 'broken pipe' message instead. We need
first grep for the message in logs and only then close the socket on a
client. The similar problem (with another test case) is described in
[1].

[1]: https://github.com/tarantool/tarantool/issues/4273#issuecomment-508939695

Closes: #4311

(cherry picked from commit 0f9fdd72)

Under certain circumstances vy_slice_new() may create an empty slice,
e.g. on range split:

   |------------------ Slice ---------------|
                         |---- Run -----|
                     +
                  split key
   |---- Slice 1 ----||------ Slice 2 ------|
         ^^^^^^^
          Empty

vy_range_update_compaction_priority() uses the size of the last slice in
a range as a base for LSM tree level sizing. If the slice size happens
to be 0, it will simply hang in an infinite loop. Fix this potential
hang by using 1 if the last slice size is 0.

(cherry picked from commit 75dc3e64)

Enabled use_unix_sockets_iproto option to use unix sockets
iproto provides the new way to handle the problem with
'Address already in use' error. It lets test-run appoint
unix sockets for LISTEN environment variable values.

Before this change the TcpPortDispatcher was used to
eliminate the race condition when two workers trying to use
the same port: the idea was that each worker used its own
ports range. Really these ports could race with client ports
(from, say, net.box or replication), which typically didn't
use bind() and so bound to a random available port (despite
any dispatched ranges) and could produce 'Address already in
use' error.

Closes: #4008

(cherry picked from commit 60f84cbf)

- Box configuration parameter `memtx_memory` is increased, because the
  test `lua` after `tuple` failed with the error:
  `Failed to allocate 368569 bytes in slab allocator for memtx_tuple`
  despite `collectgarbage('collect')` calls after cases with huge/many
  tuples.
  The statistics before the allocation fail gives the following values:
  ```
  box.slab.info()
  ---
  - items_size: 72786472
    items_used_ratio: 4.43%
    quota_size: 107374592
    quota_used_ratio: 93.75%
    arena_used_ratio: 6.1%
    items_used: 3222376
    quota_used: 100663296
    arena_size: 100663296
    arena_used: 6105960
  ```
  The reason of the fail seems to be a slab memory fragmentation. It is
  not clear for now whether we should consider this as a tarantool
  issue.

- Test `snapshot_stress` counts snapshot files present in the
  working directory and can reach the default 'checkpoint_count' value
  `2` if a previous test write its snapshots before.

- Restarting the default server w/o cleaning a working directory
  can leave a snapshot that holds a state saved at the middle of a test,
  before dropping of the space 'tweedledum' (because WAL is disabled),
  that can cause the error `Space 'tweedledum' already exists` for a
  following test.

- Use unix sockets because of errors `Address already in use`.

Part of #2436

(cherry picked from commit d837c94b)

Current results have some tests with flaky results, which
blocks the deploy stage - decided to merge deploy stage into
test stage temporary to fix it.

Follows up #4156

(cherry picked from commit b37959be)

Homebrew now contains curl-7.65.1 which affected by curl/curl#3995 (this
problem leads to segfaults). The next version is not released yet. The
current commit downgrades the curl version to 7.65.0.

Close #4288

(cherry picked from commit 2e880af0)

Implemented GitLab CI testing process additionally to existing Travis
CI. The new testing process is added to run tests faster. It requires to
control a load of machines to avoid flaky fails on timeouts. GitLab CI
allows us to run testing on our machines.

Created 2 stages for testing and deploying packages.

The testing stage contains the following jobs that are run for all
branches:

* Debian 9 (Stretch): release/debug gcc.
* Debian 10 (Buster): release clang8 + lto.
* OSX 14 (Mojave): release.
* FreeBSD 12: release gcc.

And the following jobs that are run of long-term branches (release
branches: for now it is 1.10, 2.1 and master):

* OSX 13 (Sierra): release clang.
* OSX 14 (Mojave): release clang + lto.

The deployment stage contains the same jobs as we have in Travis CI.
They however just build tarballs and packages: don't push them to S3 and
packagecloud.

In order to run full testing on a short-term branch one can name it with
'-full-ci' suffix.

The additional manual work is needed when dependencies are changed in
.travis.mk file ('deps_debian' or 'deps_buster_clang_8' goals):

 | make GITLAB_USER=foo -f .gitlab.mk docker_bootstrap

This command pushes docker images into GitLab Registry and then they are
used in testing. Pre-built images speed up testing.

Fixes #4156

(cherry picked from commit ce623a23)

Travis-ci failed on brew install python2 due to it was
already installed on OSX 10.13 Sierra, so it needs
to be installed without fails on its exists.

Follow up #4254

(cherry picked from commit ceefd0c8)

Removed skip flag file to switch on the testing of the
http_client test. Enabled http_client test on OSX,
fixed missing of the python2 symlink. Removed the subtest
on '595 error return' from 'error' suite, due to it may
hang forever. To enable test on travis-ci reverted commit:

1d7285c4 ('Disable flaky http_client.test.lua')

Closes #4254

(cherry picked from commit 33254bd6)

Issue pending, gh-4254.

(cherry picked from commit 1d7285c4)

Disabled tarballs and packages building on short-term branches.

Removed 'allow_failures' on coverage / debug build.

Replaced matrix expansion with the list of jobs (because Travis-CI
documentation says it does not support condition jobs with matrix
expansion).

Fixes #3755.

(backported from 28fcdaa0)

Fixed app/strict.test.lua fail after a test that sets 'a' global
variable. It was due to the way how 'strict' module tracks declared
global variables and was fixed in pretest_clean.lua test-run's module.

(cherry picked from commit 1df9c3d2)

* Implement SQL driver (#4123).
* Support line carrying with backslash.
* Support new result file format (for new tests).

See the link below for a full description of the new features.

https://github.com/tarantool/test-run/commit/a04b5b096c607172ce4fc86a84e3531c9f3a7304

Fixes #4123.

(cherry picked from commit d81d7b1c)

Fixed issues:

- box-py/iproto.test.py
  1) Fixed receive_response() to wait for whole response.
  2) Clean up _cluster space.

- replication-py/cluster.test.py
  1) Clean up _cluster space.

- replication-py/multi.test.py
  1) Removed vclock checking because it fails if previous test make some
     DML and vclock was incremented. Looks like it was used for debug
     and is not part of this test case.
  2) Fixed typo in 'Synchronize' block.

The following test sequences did fail due to unexpected IDs in _cluster
space:

- [box-py/iproto.test.py, null]
- [box-py/bootstrap.test.py, null]

- [replication-py/cluster.test.py, null]
- [replication-py/multi.test.py, null]

Part of #3232

(cherry picked from commit 4ea7d729)

Closes #3824

(cherry picked from commit 2aa25ba5)

region_reset() only frees memory from the last slab. As a result, if
a vylog transaction happens to use more than one slab, memory used by
vy_log.pool won't be freed, neither will it be reused, i.e. we'll get
a memory leak. Fix it by using region_free() instead of region_reset().
It's okay from performance point of view, because vylog transactions
are rare events.

Note, the master branch isn't affected to this issue, because the vylog
memory management was completely reworked there. 2.1 isn't affected
either, because there region_reset() was modified to free all slabs.
However, rather than backporting any of those commits, I think it's
more appropriate to simply use region_free().

vy_stmt_new_surrogate_from_key() and vy_stmt_new_surrogate_delete_raw()
allocate temporary objects on the region, but don't clean up after
themselves. Those functions may be called by a vinyl reader threads:

  vy_page_read_cb
    vy_page_find_key
      vy_page_stmt
        vy_stmt_decode

In this case the region will grow infinitely, because reader threads
never call fiber_gc().

The leak was introduced to 1.10 by commit b9072317 ("vinyl: lookup
key in reader thread"), which moved vy_page_find_key() invocation to
reader threads for the sake of performance. The fix is trivial - call
region_truncate() from those functions.

Note, neither the master branch nor 2.1 is affected to this issue,
because region_truncate() was added there in the scope of the multikey
index feature.

Add 'decimal' to a built-in modules list to preserve
package.loaded.decimal when pretest_clean is set in suite.ini.

Needed for #692.

(cherry picked from commit 0b7cc526)

Built-in modules are bundled into tarantool in the following way. A lua
file from src/lua or src/box/lua is stored as a string literal in a C
file, then built and linked into tarantool. During startup tarantool
calls luaL_loadbuffer() on this string.

When a Lua source is converted to a C literal, proper escaping is
performed. However there is one case, which was not covered: trigraphs.
The patch adds escaping of question mark symbols to avoid matching ??X
sequences as trigraphs by C preprocessor.

The most simple way to check that it works is to apply the following
patch:

 | diff --git a/src/lua/string.lua b/src/lua/string.lua
 | index 6e12c59ae..2da2dbf4d 100644
 | --- a/src/lua/string.lua
 | +++ b/src/lua/string.lua
 | @@ -425,3 +425,6 @@ string.fromhex    = string_fromhex
 |  string.strip      = string_strip
 |  string.lstrip      = string_lstrip
 |  string.rstrip      = string_rstrip
 | +string.foo = function()
 | +    return '??('
 | +end

And call the function like so:

 | ./src/tarantool -e 'print(string.foo()) os.exit()'

If it printfs `??(`, then everything is okay. If it prints `[`, then
`??(` was preprocessed as the trigraph.

We hit this problem when tried to bundle luarocks-3: it contains
"^(.-)(%??)$" regexp, where `??)` was interpreted as `]`. Debug build or
a build with -DENABLE_WERROR reports an error in the case, but usual
RelWithDebInfo build passes (with -Wtrigraphs warnings) and can show
this unexpected behaviour.

Fixes #4291.

(cherry picked from commit 177a1713)

vy_tx_handle_deferred_delete() expects (righteously) that in case a
deferred DELETE overwrites a statement in a secondary index write set
and the overwritten statement wasn't skipped on commit (i.e. doesn't
have txv->is_overwritten flag set), both the old and the new statement
must be REPLACEs (see the comment to the corresponding assertions for
more details).

The problem is we don't set is_overwritten flag in case a statement
doesn't have any effect (txv->is_nop is set), even if it was, in fact,
overwritten in the primary index write set (see vy_tx_prepare). As
a result, we get an assertion failure when we delete such statement
in the same transaction, e.g.

  s = box.schema.space.create('test', {engine = 'vinyl'})
  s:create_index('pk', {parts = {1, 'unsigned'}})
  s:create_index('sk', {parts = {2, 'unsigned'}})
  s:replace{1, 1, 1}
  box.begin()
  s:update(1, {{'+', 3, 1}}) -- adds no-op REPLACE to sk write set
  s:delete(1)                -- produces deferred DELETE for sk
  box.commit()

results in

  vy_tx_handle_deferred_delete: Assertion `vy_stmt_type(stmt) == IPROTO_REPLACE' failed.

Fix this by making sure we set is_overwritten for all overwritten
statements in a secondary index write set.

Closes #4294

(cherry picked from commit 41219774)

Added to cmake environment CMAKE_DL_LIBS (The name of the library
that has dlopen and dlclose in it, usually -ldl) to openssl build
to add DL library, to fix the following fails:

Linking CXX executable crypto.test
/usr/local/lib64/libcrypto.a(dso_dlfcn.o): In function `dlfcn_globallookup':
dso_dlfcn.c:(.text+0x11): undefined reference to `dlopen'
dso_dlfcn.c:(.text+0x24): undefined reference to `dlsym'
dso_dlfcn.c:(.text+0x2f): undefined reference to `dlclose'
/usr/local/lib64/libcrypto.a(dso_dlfcn.o): In function `dlfcn_bind_func':
dso_dlfcn.c:(.text+0x334): undefined reference to `dlsym'
dso_dlfcn.c:(.text+0x3f2): undefined reference to `dlerror'
/usr/local/lib64/libcrypto.a(dso_dlfcn.o): In function `dlfcn_load':
dso_dlfcn.c:(.text+0x459): undefined reference to `dlopen'
dso_dlfcn.c:(.text+0x4c9): undefined reference to `dlclose'
dso_dlfcn.c:(.text+0x502): undefined reference to `dlerror'
/usr/local/lib64/libcrypto.a(dso_dlfcn.o): In function `dlfcn_pathbyaddr':
dso_dlfcn.c:(.text+0x5a1): undefined reference to `dladdr'
dso_dlfcn.c:(.text+0x601): undefined reference to `dlerror'
/usr/local/lib64/libcrypto.a(dso_dlfcn.o): In function `dlfcn_unload':
dso_dlfcn.c:(.text+0x662): undefined reference to `dlclose'
collect2: error: ld returned 1 exit status
make[2]: *** [test/unit/crypto.test] Error 1
make[1]: *** [test/unit/CMakeFiles/crypto.test.dir/all] Error 2

Closes #4245

(cherry picked from commit 31ab6ea9)

Run collectgarbage() between tests to ensure that there are no dangling
iterators and so on. Such objects can affect statistic counters that may
be important for a test.

Fixes #4287.

(cherry picked from commit 0cbcf265)

Currently to determine current source path, user has to do it
by getting current source info from the stack and parse it manually

This commit adds couple little helpers for this task called
`debug.sourcefile()` and `debug.sourcedir()`,
which returns a path to the file being executed.

The path is relative to the current working directory.
If such path cannot be determined, '.' is returned (i.e. interactive
mode). There is an exception if you are running script with an absolute
path. In that case, absolute path will be returned too.

```bash
tarantool /absolute/path/to/the/script.lua
```

There are also handy shortcuts for that functions: `debug.__file__` and
`debug.__dir__`

@TarantoolBot document
Title: Document new debug helper methods

There are two more helpers in debug module `sourcefile()` and
`sourcedir()`. This two helpers returns relative paths to source file
and directory respectively.

There are also `debug.__file__` and `debug.__dir__` which are just handy
ways to call `sourcefile()` and `sourcedir()`.

It should be mentioned that it is only possible to determine real path
to the file, if the function was defined in a some lua file.
`loadstring` may be exception here, since lua will store whole string
in debug info.

There is also a possible pitfall because of tail call optimization, so
if the function `fn` is defined inside `myapp/init.lua` like this:

```lua
function fn()
    return debug.sourcefile()
end
```

It would not be possible to determine file path correctly because that
function would not appear in a stacktrace. Even worse, it may return
wrong result, i.e. actual path where "parent" function was defined.

To force the interpreter to avoit this kind of optimizations you may use
parenthesis like this:

```lua
function fn()
    return (debug.sourcefile())
end
```

Also both `sourcefile` and `sourcedir` functions have optional level
argument, which allows set the level of the call stack function should
examine for source path. By default, 2 is used.

(cherry picked from commit 70b8583c)

After compacting runs, we first mark them as dropped (VY_LOG_DROP_RUN),
then try to delete their files unless they are needed for recovery from
the checkpoint, and finally mark them as not needed in the vylog
(VY_LOG_FORGET_RUN). There's a potential race sitting here: the problem
is the garbage collector might kick in after files are dropped, but
before they are marked as not needed. If this happens, there will be
runs that have two VY_LOG_FORGET_RUN records, which will break recovery:

  Run XX is forgotten, but not registered

The following patches make the race more likely to happen so let's
eliminate it by making the garbage collector the only one who can mark
runs as not needed (i.e. write VY_LOG_FORGET_RUN record). There will
be no warnings, because the garbage collector silently ignores ENOENT
errors, see vy_gc().

Another good thing about this patch is that now we never yield inside
a vylog transaction, which makes it easier to remove the vylog latch
blocking implementation of transactional DDL.

(cherry picked from commit 30c9c7c0)

Example:
```
[001] box/on_replace.test.lua                                         [ pass ]
[001] box/on_replace.test.lua                                         [ fail ]
[001]
[001] Test failed! Result content mismatch:
[001] --- box/on_replace.result	Tue Jun  4 10:44:39 2019
[001] +++ box/on_replace.reject	Tue Jun  4 10:44:50 2019
[001] @@ -4,7 +4,7 @@
[001]  -- test c and lua triggers: must return only lua triggers
[001]  #box.space._space:on_replace()
[001]  ---
[001] -- 0
[001] +- 2
[001]  ...
```
This happened because we forgot to remove triggers set on _space.
Clear them below the test case.

(cherry picked from commit d3552264)

This commit fixes a regression introduced by commit
5ab0763b
(pass operation type to lua triggers)

When passing operation type to the trigger, we relied on a corresponding
xrow_header, which would later be written to the log. When before_replace
triggers are fired, there is no row connected to the txn statement yet,
so we faked one to pass operations. We, however, didn't account for
temporary spaces, which never have a row associated with a corresponding
txn stmt. This lead to a segfault on an attemt to use on_replace triggers
with temporary spaces.

Add a fake row for temporary spaces to pass operation type in on_replace
triggers and add some tests.

Closes #4266

(cherry picked from commit 63ebc25a)

Since DDL is triggered by the admin, it can be deliberately initiated
when the workload is known to be low. Throttling it along with DML
requests would only cause exasperation in this case. So we don't apply
disk-based rate limit to DDL. This should be fine, because the
disk-based limit is set rather strictly to let the workload some space
to grow, see vy_regulator_update_rate_limit(), and in contrast to the
memory-based limit, exceeding the disk-based limit doesn't result in
abrupt stalls - it may only lead to a gradual accumulation of disk space
usage and read latency.

Closes #4238

(cherry picked from commit 6a44ac62)

If a key isn't found in the tuple cache, we fetch it from a run file. In
this case disk read and page decompression is done by a reader thread,
however key lookup in the fetched page is still performed by the tx
thread. Since pages are immutable, this could as well be done by the
reader thread, which would allow us to save some precious CPU cycles for
tx.

Close #4257

(cherry picked from commit 04b19ac1)

To handle fiber cancellation during page read we need to pin all objects
referenced by vy_page_read_task. Currently, there's the only such
object, vy_run. It has reference counting so pinning it is trivial.
However, to move page lookup to a reader thread, we need to also
reference key def, tuple format, and key. Format and key have reference
counting, but key def doesn't - we typically copy it. Copying it in this
case is too heavy.

Actually, cancelling a fiber manually or on timeout while it's reading
disk doesn't make much sense with PCIE attached flash drives. It used to
be reasonable with rotating disks, since a rotating disk controller
could retry reading a block indefinitely on read failure. It is still
relevant to Network Attached Storage. On the other hand, NAS has never
been tested, and what isn't tested, can and should be removed. For
complex SQL queries we'll be forced to rethink timeout handling anyway.

That being said, let's simply drop this functionality.

(cherry picked from commit bab04b25)

Page reading code is intermixed with the reader thread selection in the
same function, which makes it difficult to extend the former. So let's
introduce a helper function encapsulating a call on behalf of a reader
thread.

(cherry picked from commit d8a95a2a)

Since a page read task references the source run file, we don't need to
pass page info by value.

(cherry picked from commit 67d36ccc)

This function is a part of the run iterator API so we can't use it in
a reader thread. Let's make it an independent helper. As a good side
effect, we can now reuse it in the slice stream implementation.

(cherry picked from commit ac8ce023)

This warning breaks -Werror -O2 build on GCC 9.1.

(cherry picked from commit 2f1a9012)

(cherry picked from commit bfde5242)

We run testing in Travis-CI using docker, which by default enters into a
shell session as root.

This is follow up for e9c96a4c ('fio:
fix mktree error reporting').

(cherry picked from commit 8cd18ec6)

In case of, say, permission denied the function did try to concatenate a
string with a cdata<struct error> and fails.

Also unified error messages in case of a single part path and a multi
part one.

(cherry picked from commit e9c96a4c)

Tune secondary index content to make sure minor compaction expected by
the test does occur.

Fixes commit e2f5e1bc ("vinyl: don't produce deferred DELETE on
commit if key isn't updated").

Closes #4255

(cherry picked from commit a28a0d6a)

Backport of openrusty/luajit2-test-suite
commit 907c536c210ebe6a147861bb4433d28c0ebfc8cd

To test unsink 64 bit pointers

Part-of #4171

(cherry picked from commit 10142d83)

(cherry picked from commit e7b43e47)

Even if a statement isn't marked as VY_STMT_DEFERRED_DELETE, e.g. it's
a REPLACE produced by an UPDATE request, it may overwrite a statement in
the transaction write set that is marked so, for instance:

  s = box.schema.space.create('test', {engine = 'vinyl'})
  pk = s:create_index('pk')
  sk = s:create_index('sk', {parts = {2, 'unsigned'}})

  s:insert{1, 1}

  box.begin()
  s:replace{1, 2}
  s:update(1, {{'=', 2, 3}})
  box.commit()

If we don't mark REPLACE{3,1} produced by the update operatoin with
VY_STMT_DEFERRED_DELETE flag, we will never generate a DELETE statement
for INSERT{1,1}. That is, we must inherit the flag from the overwritten
statement when we insert a new one into a write set.

Closes #4248

(cherry picked from commit b54433d9)