Homebrew now contains curl-7.65.1 which affected by curl/curl#3995 (this
problem leads to segfaults). The next version is not released yet. The
current commit downgrades the curl version to 7.65.0.

Close #4288

(cherry picked from commit 2e880af0)

Implemented GitLab CI testing process additionally to existing Travis
CI. The new testing process is added to run tests faster. It requires to
control a load of machines to avoid flaky fails on timeouts. GitLab CI
allows us to run testing on our machines.

Created 2 stages for testing and deploying packages.

The testing stage contains the following jobs that are run for all
branches:

* Debian 9 (Stretch): release/debug gcc.
* Debian 10 (Buster): release clang8 + lto.
* OSX 14 (Mojave): release.
* FreeBSD 12: release gcc.

And the following jobs that are run of long-term branches (release
branches: for now it is 1.10, 2.1 and master):

* OSX 13 (Sierra): release clang.
* OSX 14 (Mojave): release clang + lto.

The deployment stage contains the same jobs as we have in Travis CI.
They however just build tarballs and packages: don't push them to S3 and
packagecloud.

In order to run full testing on a short-term branch one can name it with
'-full-ci' suffix.

The additional manual work is needed when dependencies are changed in
.travis.mk file ('deps_debian' or 'deps_buster_clang_8' goals):

 | make GITLAB_USER=foo -f .gitlab.mk docker_bootstrap

This command pushes docker images into GitLab Registry and then they are
used in testing. Pre-built images speed up testing.

Fixes #4156

(cherry picked from commit ce623a23)

Travis-ci failed on brew install python2 due to it was
already installed on OSX 10.13 Sierra, so it needs
to be installed without fails on its exists.

Follow up #4254

(cherry picked from commit ceefd0c8)

Removed skip flag file to switch on the testing of the
http_client test. Enabled http_client test on OSX,
fixed missing of the python2 symlink. Removed the subtest
on '595 error return' from 'error' suite, due to it may
hang forever. To enable test on travis-ci reverted commit:

1d7285c4 ('Disable flaky http_client.test.lua')

Closes #4254

(cherry picked from commit 33254bd6)

Issue pending, gh-4254.

(cherry picked from commit 1d7285c4)

Disabled tarballs and packages building on short-term branches.

Removed 'allow_failures' on coverage / debug build.

Replaced matrix expansion with the list of jobs (because Travis-CI
documentation says it does not support condition jobs with matrix
expansion).

Fixes #3755.

(backported from 28fcdaa0)

Fixed app/strict.test.lua fail after a test that sets 'a' global
variable. It was due to the way how 'strict' module tracks declared
global variables and was fixed in pretest_clean.lua test-run's module.

(cherry picked from commit 1df9c3d2)

* Implement SQL driver (#4123).
* Support line carrying with backslash.
* Support new result file format (for new tests).

See the link below for a full description of the new features.

https://github.com/tarantool/test-run/commit/a04b5b096c607172ce4fc86a84e3531c9f3a7304

Fixes #4123.

(cherry picked from commit d81d7b1c)

Fixed issues:

- box-py/iproto.test.py
  1) Fixed receive_response() to wait for whole response.
  2) Clean up _cluster space.

- replication-py/cluster.test.py
  1) Clean up _cluster space.

- replication-py/multi.test.py
  1) Removed vclock checking because it fails if previous test make some
     DML and vclock was incremented. Looks like it was used for debug
     and is not part of this test case.
  2) Fixed typo in 'Synchronize' block.

The following test sequences did fail due to unexpected IDs in _cluster
space:

- [box-py/iproto.test.py, null]
- [box-py/bootstrap.test.py, null]

- [replication-py/cluster.test.py, null]
- [replication-py/multi.test.py, null]

Part of #3232

(cherry picked from commit 4ea7d729)

Closes #3824

(cherry picked from commit 2aa25ba5)

region_reset() only frees memory from the last slab. As a result, if
a vylog transaction happens to use more than one slab, memory used by
vy_log.pool won't be freed, neither will it be reused, i.e. we'll get
a memory leak. Fix it by using region_free() instead of region_reset().
It's okay from performance point of view, because vylog transactions
are rare events.

Note, the master branch isn't affected to this issue, because the vylog
memory management was completely reworked there. 2.1 isn't affected
either, because there region_reset() was modified to free all slabs.
However, rather than backporting any of those commits, I think it's
more appropriate to simply use region_free().

vy_stmt_new_surrogate_from_key() and vy_stmt_new_surrogate_delete_raw()
allocate temporary objects on the region, but don't clean up after
themselves. Those functions may be called by a vinyl reader threads:

  vy_page_read_cb
    vy_page_find_key
      vy_page_stmt
        vy_stmt_decode

In this case the region will grow infinitely, because reader threads
never call fiber_gc().

The leak was introduced to 1.10 by commit b9072317 ("vinyl: lookup
key in reader thread"), which moved vy_page_find_key() invocation to
reader threads for the sake of performance. The fix is trivial - call
region_truncate() from those functions.

Note, neither the master branch nor 2.1 is affected to this issue,
because region_truncate() was added there in the scope of the multikey
index feature.

Add 'decimal' to a built-in modules list to preserve
package.loaded.decimal when pretest_clean is set in suite.ini.

Needed for #692.

(cherry picked from commit 0b7cc526)

Built-in modules are bundled into tarantool in the following way. A lua
file from src/lua or src/box/lua is stored as a string literal in a C
file, then built and linked into tarantool. During startup tarantool
calls luaL_loadbuffer() on this string.

When a Lua source is converted to a C literal, proper escaping is
performed. However there is one case, which was not covered: trigraphs.
The patch adds escaping of question mark symbols to avoid matching ??X
sequences as trigraphs by C preprocessor.

The most simple way to check that it works is to apply the following
patch:

 | diff --git a/src/lua/string.lua b/src/lua/string.lua
 | index 6e12c59ae..2da2dbf4d 100644
 | --- a/src/lua/string.lua
 | +++ b/src/lua/string.lua
 | @@ -425,3 +425,6 @@ string.fromhex    = string_fromhex
 |  string.strip      = string_strip
 |  string.lstrip      = string_lstrip
 |  string.rstrip      = string_rstrip
 | +string.foo = function()
 | +    return '??('
 | +end

And call the function like so:

 | ./src/tarantool -e 'print(string.foo()) os.exit()'

If it printfs `??(`, then everything is okay. If it prints `[`, then
`??(` was preprocessed as the trigraph.

We hit this problem when tried to bundle luarocks-3: it contains
"^(.-)(%??)$" regexp, where `??)` was interpreted as `]`. Debug build or
a build with -DENABLE_WERROR reports an error in the case, but usual
RelWithDebInfo build passes (with -Wtrigraphs warnings) and can show
this unexpected behaviour.

Fixes #4291.

(cherry picked from commit 177a1713)

vy_tx_handle_deferred_delete() expects (righteously) that in case a
deferred DELETE overwrites a statement in a secondary index write set
and the overwritten statement wasn't skipped on commit (i.e. doesn't
have txv->is_overwritten flag set), both the old and the new statement
must be REPLACEs (see the comment to the corresponding assertions for
more details).

The problem is we don't set is_overwritten flag in case a statement
doesn't have any effect (txv->is_nop is set), even if it was, in fact,
overwritten in the primary index write set (see vy_tx_prepare). As
a result, we get an assertion failure when we delete such statement
in the same transaction, e.g.

  s = box.schema.space.create('test', {engine = 'vinyl'})
  s:create_index('pk', {parts = {1, 'unsigned'}})
  s:create_index('sk', {parts = {2, 'unsigned'}})
  s:replace{1, 1, 1}
  box.begin()
  s:update(1, {{'+', 3, 1}}) -- adds no-op REPLACE to sk write set
  s:delete(1)                -- produces deferred DELETE for sk
  box.commit()

results in

  vy_tx_handle_deferred_delete: Assertion `vy_stmt_type(stmt) == IPROTO_REPLACE' failed.

Fix this by making sure we set is_overwritten for all overwritten
statements in a secondary index write set.

Closes #4294

(cherry picked from commit 41219774)

Added to cmake environment CMAKE_DL_LIBS (The name of the library
that has dlopen and dlclose in it, usually -ldl) to openssl build
to add DL library, to fix the following fails:

Linking CXX executable crypto.test
/usr/local/lib64/libcrypto.a(dso_dlfcn.o): In function `dlfcn_globallookup':
dso_dlfcn.c:(.text+0x11): undefined reference to `dlopen'
dso_dlfcn.c:(.text+0x24): undefined reference to `dlsym'
dso_dlfcn.c:(.text+0x2f): undefined reference to `dlclose'
/usr/local/lib64/libcrypto.a(dso_dlfcn.o): In function `dlfcn_bind_func':
dso_dlfcn.c:(.text+0x334): undefined reference to `dlsym'
dso_dlfcn.c:(.text+0x3f2): undefined reference to `dlerror'
/usr/local/lib64/libcrypto.a(dso_dlfcn.o): In function `dlfcn_load':
dso_dlfcn.c:(.text+0x459): undefined reference to `dlopen'
dso_dlfcn.c:(.text+0x4c9): undefined reference to `dlclose'
dso_dlfcn.c:(.text+0x502): undefined reference to `dlerror'
/usr/local/lib64/libcrypto.a(dso_dlfcn.o): In function `dlfcn_pathbyaddr':
dso_dlfcn.c:(.text+0x5a1): undefined reference to `dladdr'
dso_dlfcn.c:(.text+0x601): undefined reference to `dlerror'
/usr/local/lib64/libcrypto.a(dso_dlfcn.o): In function `dlfcn_unload':
dso_dlfcn.c:(.text+0x662): undefined reference to `dlclose'
collect2: error: ld returned 1 exit status
make[2]: *** [test/unit/crypto.test] Error 1
make[1]: *** [test/unit/CMakeFiles/crypto.test.dir/all] Error 2

Closes #4245

(cherry picked from commit 31ab6ea9)

Run collectgarbage() between tests to ensure that there are no dangling
iterators and so on. Such objects can affect statistic counters that may
be important for a test.

Fixes #4287.

(cherry picked from commit 0cbcf265)

Currently to determine current source path, user has to do it
by getting current source info from the stack and parse it manually

This commit adds couple little helpers for this task called
`debug.sourcefile()` and `debug.sourcedir()`,
which returns a path to the file being executed.

The path is relative to the current working directory.
If such path cannot be determined, '.' is returned (i.e. interactive
mode). There is an exception if you are running script with an absolute
path. In that case, absolute path will be returned too.

```bash
tarantool /absolute/path/to/the/script.lua
```

There are also handy shortcuts for that functions: `debug.__file__` and
`debug.__dir__`

@TarantoolBot document
Title: Document new debug helper methods

There are two more helpers in debug module `sourcefile()` and
`sourcedir()`. This two helpers returns relative paths to source file
and directory respectively.

There are also `debug.__file__` and `debug.__dir__` which are just handy
ways to call `sourcefile()` and `sourcedir()`.

It should be mentioned that it is only possible to determine real path
to the file, if the function was defined in a some lua file.
`loadstring` may be exception here, since lua will store whole string
in debug info.

There is also a possible pitfall because of tail call optimization, so
if the function `fn` is defined inside `myapp/init.lua` like this:

```lua
function fn()
    return debug.sourcefile()
end
```

It would not be possible to determine file path correctly because that
function would not appear in a stacktrace. Even worse, it may return
wrong result, i.e. actual path where "parent" function was defined.

To force the interpreter to avoit this kind of optimizations you may use
parenthesis like this:

```lua
function fn()
    return (debug.sourcefile())
end
```

Also both `sourcefile` and `sourcedir` functions have optional level
argument, which allows set the level of the call stack function should
examine for source path. By default, 2 is used.

(cherry picked from commit 70b8583c)

After compacting runs, we first mark them as dropped (VY_LOG_DROP_RUN),
then try to delete their files unless they are needed for recovery from
the checkpoint, and finally mark them as not needed in the vylog
(VY_LOG_FORGET_RUN). There's a potential race sitting here: the problem
is the garbage collector might kick in after files are dropped, but
before they are marked as not needed. If this happens, there will be
runs that have two VY_LOG_FORGET_RUN records, which will break recovery:

  Run XX is forgotten, but not registered

The following patches make the race more likely to happen so let's
eliminate it by making the garbage collector the only one who can mark
runs as not needed (i.e. write VY_LOG_FORGET_RUN record). There will
be no warnings, because the garbage collector silently ignores ENOENT
errors, see vy_gc().

Another good thing about this patch is that now we never yield inside
a vylog transaction, which makes it easier to remove the vylog latch
blocking implementation of transactional DDL.

(cherry picked from commit 30c9c7c0)

Example:
```
[001] box/on_replace.test.lua                                         [ pass ]
[001] box/on_replace.test.lua                                         [ fail ]
[001]
[001] Test failed! Result content mismatch:
[001] --- box/on_replace.result	Tue Jun  4 10:44:39 2019
[001] +++ box/on_replace.reject	Tue Jun  4 10:44:50 2019
[001] @@ -4,7 +4,7 @@
[001]  -- test c and lua triggers: must return only lua triggers
[001]  #box.space._space:on_replace()
[001]  ---
[001] -- 0
[001] +- 2
[001]  ...
```
This happened because we forgot to remove triggers set on _space.
Clear them below the test case.

(cherry picked from commit d3552264)

This commit fixes a regression introduced by commit
5ab0763b
(pass operation type to lua triggers)

When passing operation type to the trigger, we relied on a corresponding
xrow_header, which would later be written to the log. When before_replace
triggers are fired, there is no row connected to the txn statement yet,
so we faked one to pass operations. We, however, didn't account for
temporary spaces, which never have a row associated with a corresponding
txn stmt. This lead to a segfault on an attemt to use on_replace triggers
with temporary spaces.

Add a fake row for temporary spaces to pass operation type in on_replace
triggers and add some tests.

Closes #4266

(cherry picked from commit 63ebc25a)

Since DDL is triggered by the admin, it can be deliberately initiated
when the workload is known to be low. Throttling it along with DML
requests would only cause exasperation in this case. So we don't apply
disk-based rate limit to DDL. This should be fine, because the
disk-based limit is set rather strictly to let the workload some space
to grow, see vy_regulator_update_rate_limit(), and in contrast to the
memory-based limit, exceeding the disk-based limit doesn't result in
abrupt stalls - it may only lead to a gradual accumulation of disk space
usage and read latency.

Closes #4238

(cherry picked from commit 6a44ac62)

If a key isn't found in the tuple cache, we fetch it from a run file. In
this case disk read and page decompression is done by a reader thread,
however key lookup in the fetched page is still performed by the tx
thread. Since pages are immutable, this could as well be done by the
reader thread, which would allow us to save some precious CPU cycles for
tx.

Close #4257

(cherry picked from commit 04b19ac1)

To handle fiber cancellation during page read we need to pin all objects
referenced by vy_page_read_task. Currently, there's the only such
object, vy_run. It has reference counting so pinning it is trivial.
However, to move page lookup to a reader thread, we need to also
reference key def, tuple format, and key. Format and key have reference
counting, but key def doesn't - we typically copy it. Copying it in this
case is too heavy.

Actually, cancelling a fiber manually or on timeout while it's reading
disk doesn't make much sense with PCIE attached flash drives. It used to
be reasonable with rotating disks, since a rotating disk controller
could retry reading a block indefinitely on read failure. It is still
relevant to Network Attached Storage. On the other hand, NAS has never
been tested, and what isn't tested, can and should be removed. For
complex SQL queries we'll be forced to rethink timeout handling anyway.

That being said, let's simply drop this functionality.

(cherry picked from commit bab04b25)

Page reading code is intermixed with the reader thread selection in the
same function, which makes it difficult to extend the former. So let's
introduce a helper function encapsulating a call on behalf of a reader
thread.

(cherry picked from commit d8a95a2a)

Since a page read task references the source run file, we don't need to
pass page info by value.

(cherry picked from commit 67d36ccc)

This function is a part of the run iterator API so we can't use it in
a reader thread. Let's make it an independent helper. As a good side
effect, we can now reuse it in the slice stream implementation.

(cherry picked from commit ac8ce023)

This warning breaks -Werror -O2 build on GCC 9.1.

(cherry picked from commit 2f1a9012)

(cherry picked from commit bfde5242)

We run testing in Travis-CI using docker, which by default enters into a
shell session as root.

This is follow up for e9c96a4c ('fio:
fix mktree error reporting').

(cherry picked from commit 8cd18ec6)

In case of, say, permission denied the function did try to concatenate a
string with a cdata<struct error> and fails.

Also unified error messages in case of a single part path and a multi
part one.

(cherry picked from commit e9c96a4c)

Tune secondary index content to make sure minor compaction expected by
the test does occur.

Fixes commit e2f5e1bc ("vinyl: don't produce deferred DELETE on
commit if key isn't updated").

Closes #4255

(cherry picked from commit a28a0d6a)

Backport of openrusty/luajit2-test-suite
commit 907c536c210ebe6a147861bb4433d28c0ebfc8cd

To test unsink 64 bit pointers

Part-of #4171

(cherry picked from commit 10142d83)

(cherry picked from commit e7b43e47)

Even if a statement isn't marked as VY_STMT_DEFERRED_DELETE, e.g. it's
a REPLACE produced by an UPDATE request, it may overwrite a statement in
the transaction write set that is marked so, for instance:

  s = box.schema.space.create('test', {engine = 'vinyl'})
  pk = s:create_index('pk')
  sk = s:create_index('sk', {parts = {2, 'unsigned'}})

  s:insert{1, 1}

  box.begin()
  s:replace{1, 2}
  s:update(1, {{'=', 2, 3}})
  box.commit()

If we don't mark REPLACE{3,1} produced by the update operatoin with
VY_STMT_DEFERRED_DELETE flag, we will never generate a DELETE statement
for INSERT{1,1}. That is, we must inherit the flag from the overwritten
statement when we insert a new one into a write set.

Closes #4248

(cherry picked from commit b54433d9)

Consider the following example:

  s = box.schema.space.create('test', {engine = 'vinyl'})
  s:create_index('primary')
  s:create_index('secondary', {parts = {2, 'unsigned'}})

  s:insert{1, 1, 1}
  s:replace{1, 1, 2}

When REPLACE{1,1} is committed to the secondary index, the overwritten
tuple, i.e. INSERT{1,1}, is found in the primary index memory, and so
deferred DELETE{1,1} is generated right away and committed along with
REPLACE{1,1}. However, there's no need to commit anything to the
secondary index in this case, because its key isn't updated. Apart from
eating memory and loading disk, this also breaks index stats, as vy_tx
implementation doesn't expect two statements committed for the same key
in a single transaction.

Fix this by checking if there's a statement in the log for the deleted
key and if there's skipping them both as we do in the regular case, see
the comment in vy_tx_set.

Closes #3693

(cherry picked from commit e2f5e1bc)

If an UPDATE request doesn't touch key parts of a secondary index, we
don't need to re-index it in the in-memory secondary index, as this
would only increase IO load. Historically, we use column mask set by the
UPDATE operation to skip secondary indexes that are not affected by the
operation on commit. However, there's a problem here: the column mask
isn't precise - it may have a bit set even if the corresponding column
value isn't changed by the update operation, e.g. consider {'+', 2, 0}.
Not taking this into account may result in appearance of phantom tuples
on disk as the write iterator assumes that statements that have no
effect aren't written to secondary indexes (this is needed to apply
INSERT+DELETE "annihilation" optimization). We fixed that by clearing
column mask bits in vy_tx_set in case we detect that the key isn't
changed, for more details see #3607 and commit e72867cb ("vinyl: fix
appearance of phantom tuple in secondary index after update"). It was
rather an ugly hack, but it worked.

However, it turned out that apart from looking hackish this code has
a nasty bug that may lead to tuples missing from secondary indexes.
Consider the following example:

  s = box.schema.space.create('test', {engine = 'vinyl'})
  s:create_index('pk')
  s:create_index('sk', {parts = {2, 'unsigned'}})
  s:insert{1, 1, 1}

  box.begin()
  s:update(1, {{'=', 2, 2}})
  s:update(1, {{'=', 3, 2}})
  box.commit()

The first update operation writes DELETE{1,1} and REPLACE{2,1} to the
secondary index write set. The second update replaces REPLACE{2,1} with
DELETE{2,1} and then with REPLACE{2,1}. When replacing DELETE{2,1} with
REPLACE{2,1} in the write set, we assume that the update doesn't modify
secondary index key parts and clear the column mask so as not to commit
a pointless request, see vy_tx_set. As a result, we skip the first
update too and get key {2,1} missing from the secondary index.

Actually, it was a dumb idea to use column mask to skip statements in
the first place, as there's a much easier way to filter out statements
that have no effect for secondary indexes. The thing is every DELETE
statement inserted into a secondary index write set acts as a "single
DELETE", i.e. there's exactly one older statement it is supposed to
purge. This is, because in contrast to the primary index we don't write
DELETE statements blindly - we always look up the tuple overwritten in
the primary index first. This means that REPLACE+DELETE for the same key
is basically a no-op and can be safely skip. Moreover, DELETE+REPLACE
can be treated as no-op, too, because secondary indexes don't store full
tuples hence all REPLACE statements for the same key are equivalent.
By marking both statements as no-op in vy_tx_set, we guarantee that
no-op statements don't make it to secondary index memory or disk levels.

Closes #4242

(cherry picked from commit 69aee6fc)

Backport of openresty/luajit2-test-suite
commit ce2c916d5582914edeb9499f487d9fa812632c5c

To test hash chain bug.

Part-of #4171

(cherry picked from commit 865de309f3b2ff3177a608af384839ce2fa36beb)

(cherry picked from commit 98aa418fdf74b29c90204f263866f2e64cc3acb4)

Crypto provides API to create stream objects. These streams
consume plain and return ecnrypted data. Steps:

    1 c = cipher.new([key, iv])
    2 c:init(key, iv)
    3 c:update(input)
    4 c:result()

Step 2 is optional, if key and iv are specified in new(), but if
it called without key or iv, then result() method crashes.

The commit allows to fill key and iv gradually, in several init()
calls, and remembers previous results.

Closes #4223

(cherry picked from commit 26333580)