In case an error occurs between relay_start() and cord_costart() in
relay_subscribe(), the relay status won't be reset to STOPPED. As a
result, any further attempt to re-subscribe will fail with ER_CFG:
duplicate connection with the same replica UUID. This may happen, for
example, if the WAL directory happens to be temporarily inaccessible on
the master.

Closes #4399

(cherry picked from commit 35ef3320)

There are two reasons to disable support of tables with __tostring
metamethods as opts.headers values:

* It seems this feature is not much needed: a type convertion can be
  easily done on a user's end with explicit tostring() call if needed.
* It never did work since introduction in 2.1.1-311-g85e1d78bc ('httpc:
  add checking of headers in httpc:request'): __tostring presence was
  verified, but a function in the field was not invoked.

Now http client accepts only a Lua string as a header key or a value.

Closes #3679 (again).

(cherry picked from commit 28e688294b8ae3868c138a9de965479e8de24c3e)

Added the following checks:

* opts.headers is a table.
* opts.headers keys are strings.

Clarified an error message re Lua type of opts.headers values.

Found and fixed a memory leak that appears in http client when invalid
opts.headers is passed.

Closes #4281

(cherry picked from commit fc355a2c)

The problem was that a tagged revision is not deployed, so after a
release we did an empty commit to trigger deployment. Now it is worked
around by adding extra deployment rules that deploys tagged revisions.
The workaround was suggested by Hiro Asari in [1].

[1]: https://github.com/travis-ci/travis-ci/issues/7780#issuecomment-302389370

Fixes #3745.

(cherry picked from commit 2fe7036d)

Due to the new 7.65.3 curl version released on 2019-07-19,
removed temporary workaround that downgraded the curl to
7.65.0.

This reverts commit 2e880af0.

Follows up #4288

(cherry picked from commit abd5ccee)

We may write to txn->sub_stmt_begin[TXN_SUB_STMT_MAX] so the array size
must be TXN_SUB_STMT_MAX+1 (see txn_begin_stmt). This didn't lead to any
problems, because we would only overwrite txn::signature, which wouldn't
break anything. However, should we change the txn struct, we could get
an unexpected error or even a crash.

(cherry picked from commit 4e72874a)

pretest_clean: extended a list of predefined functions with registered
SQL builtins and persistent Lua function 'LUA'.

Follows up #4182.

(cherry picked from commit e5e23ce2)

This last error
```
[035]  ...
[035]  disconnected_cnt
[035]  ---
[035] -- 1
[035] +- 2
[035]  ...
[035]  conn:close()
[035]  ---
[035]  ...
[035]  disconnected_cnt
[035]  ---
[035] -- 2
[035] +- 3
[035]  ...
[035]  test_run:cmd('stop server connecter')
[035]  ---
[035]
```
Happens because net.box is able to connect to tarantool before it has
finished bootstrap. When connecting, net.box tries to fetch schema
executing a couple of selects, but fails to pass access check since
grants aren't applied yet. This is described in detail in
https://github.com/tarantool/tarantool/issues/2763#issuecomment-499046998
So, alter the test so that it tolerates multiple connection failures.

Closes #4273

(cherry picked from commit 1a2addb8)

We are supposed to authenticate guest user without a password. This
used to work before commit 076a8420 ("Permit empty passwords in
net.box"), when guest didn't have any password. Now it has an empty
password and the check in authenticate turns out to be broken, which
breaks assumptions made by certain connectors. This patch fixes the
check.

Closes #4327

(cherry picked from commit c185a387)

box/net.box test flaky failed on grepping the log file
for 'ER_NO_SUCH_PROC' pattern on high load running hosts,
found that the issue can be resolved by updating the
grep_log to wait_log function to make able to wait the
needed message for some time.

[008] Test failed! Result content mismatch:
[008] --- box/net.box.result	Tue Jul  9 17:00:24 2019
[008] +++ box/net.box.reject	Tue Jul  9 17:03:34 2019
[008] @@ -1376,7 +1376,7 @@
[008]  ...
[008]  test_run:grep_log("default", "ER_NO_SUCH_PROC")
[008]  ---
[008] -- ER_NO_SUCH_PROC
[008] +- null
[008]  ...
[008]  box.schema.user.revoke('guest', 'execute', 'universe')
[008]  ---

Closes #4329

(cherry picked from commit 9bde3406)

Test box/net.box.test.lua checks state of the connection in case
of an error. It should be 'error_reconnect'. But, in cases where
testing was performed on a slow computer or in the case of a very
large load, it is possible that the connection status may change
from the 'error_reconnect' state to another state. This led to the
failure of the test. Since this check is not the main purpose of
the test, it is better to simply delete the check.

Closes #4335

(cherry picked from commit 77051a11)

On high loaded host the test vinyl/recover failed on waiting loop
for the dumper counter check. It expected that the value should be
equal to "2" exactly, while on the high loaded host the dump could
be already run more times and the counter value found to be "3" or
even bigger values. To fix it the counter value check was changed
from the exact value to the range bigger or equal of the expecting
value. The error message before the fix was:

[003] --- vinyl/recover.result	Mon Jul 15 10:46:00 2019
[003] +++ vinyl/recover.reject	Mon Jul 15 10:58:10 2019
[003] @@ -517,7 +517,7 @@
[003]  ...
[003]  test_run:wait_cond(function() return pk:stat().disk.dump.count == 2 end)
[003]  ---
[003] -- true
[003] +- false
[003]  ...
[003]  sk:stat().disk.dump.count -- 1
[003]  ---

Closes #4345

(cherry picked from commit 2d3f299e)

The test regarding logging corrupted rows failed occasionally with
```
[016]  test_run:grep_log('default', 'Got a corrupted row.*')
[016]  ---
[016] -- 'Got a corrupted row:'
[016] +- null
[016]  ...
```
The logs then had
```
[010] 2019-07-06 19:36:16.857 [13046] iproto sio.c:261 !> SystemError writev(1),
called on fd 23, aka unix/:(socket), peer of unix/:(socket): Broken pipe
```
instead of the expected message.

This happened, because we closed a socket before tarantool could write a
greeting to the client, the connection was then closed, and execution
never got to processing the malformed request and thus printing the
desired message to the log.

To fix this, actually read the greeting prior to writing new data and
closing the socket.

Follow-up #4273

(cherry picked from commit eb0cc50c)

The test case has two problems that appear from time to time and lead to
flaky fails. Those fails are look as shown below in a test-run output.

 | Test failed! Result content mismatch:
 | --- box/net.box.result	Mon Jun 24 17:23:49 2019
 | +++ box/net.box.reject	Mon Jun 24 17:51:52 2019
 | @@ -1404,7 +1404,7 @@
 |  ...
 |  test_run:grep_log('default', 'ER_INVALID_MSGPACK.*')
 | ---
 | -- 'ER_INVALID_MSGPACK: Invalid MsgPack - packet body'
 | +- 'ER_INVALID_MSGPACK: Invalid MsgPack - packet length'
 | ...
 | -- gh-983 selecting a lot of data crashes the server or hangs the
 | -- connection

'ER_INVALID_MSGPACK.*' regexp should match 'ER_INVALID_MSGPACK: Invalid
MsgPack - packet body' log message, but if it is not in a log file at a
time of grep_log() call (just don't flushed to the file yet) a message
produced by another test case can be matched ('ER_INVALID_MSGPACK:
Invalid MsgPack - packet length'). The fix here is to match the entire
message and check for the message periodically during several seconds
(use wait_log() instead of grep_log()).

Another problem is the race between writing a response to an iproto
socket on a server side and closing the socket on a client end. If
tarantool is unable to write a response, it does not produce the warning
re invalid msgpack, but shows 'broken pipe' message instead. We need
first grep for the message in logs and only then close the socket on a
client. The similar problem (with another test case) is described in
[1].

[1]: https://github.com/tarantool/tarantool/issues/4273#issuecomment-508939695

Closes: #4311

(cherry picked from commit 0f9fdd72)

Under certain circumstances vy_slice_new() may create an empty slice,
e.g. on range split:

   |------------------ Slice ---------------|
                         |---- Run -----|
                     +
                  split key
   |---- Slice 1 ----||------ Slice 2 ------|
         ^^^^^^^
          Empty

vy_range_update_compaction_priority() uses the size of the last slice in
a range as a base for LSM tree level sizing. If the slice size happens
to be 0, it will simply hang in an infinite loop. Fix this potential
hang by using 1 if the last slice size is 0.

(cherry picked from commit 75dc3e64)

Enabled use_unix_sockets_iproto option to use unix sockets
iproto provides the new way to handle the problem with
'Address already in use' error. It lets test-run appoint
unix sockets for LISTEN environment variable values.

Before this change the TcpPortDispatcher was used to
eliminate the race condition when two workers trying to use
the same port: the idea was that each worker used its own
ports range. Really these ports could race with client ports
(from, say, net.box or replication), which typically didn't
use bind() and so bound to a random available port (despite
any dispatched ranges) and could produce 'Address already in
use' error.

Closes: #4008

(cherry picked from commit 60f84cbf)

- Box configuration parameter `memtx_memory` is increased, because the
  test `lua` after `tuple` failed with the error:
  `Failed to allocate 368569 bytes in slab allocator for memtx_tuple`
  despite `collectgarbage('collect')` calls after cases with huge/many
  tuples.
  The statistics before the allocation fail gives the following values:
  ```
  box.slab.info()
  ---
  - items_size: 72786472
    items_used_ratio: 4.43%
    quota_size: 107374592
    quota_used_ratio: 93.75%
    arena_used_ratio: 6.1%
    items_used: 3222376
    quota_used: 100663296
    arena_size: 100663296
    arena_used: 6105960
  ```
  The reason of the fail seems to be a slab memory fragmentation. It is
  not clear for now whether we should consider this as a tarantool
  issue.

- Test `snapshot_stress` counts snapshot files present in the
  working directory and can reach the default 'checkpoint_count' value
  `2` if a previous test write its snapshots before.

- Restarting the default server w/o cleaning a working directory
  can leave a snapshot that holds a state saved at the middle of a test,
  before dropping of the space 'tweedledum' (because WAL is disabled),
  that can cause the error `Space 'tweedledum' already exists` for a
  following test.

- Use unix sockets because of errors `Address already in use`.

Part of #2436

(cherry picked from commit d837c94b)

Current results have some tests with flaky results, which
blocks the deploy stage - decided to merge deploy stage into
test stage temporary to fix it.

Follows up #4156

(cherry picked from commit b37959be)

Homebrew now contains curl-7.65.1 which affected by curl/curl#3995 (this
problem leads to segfaults). The next version is not released yet. The
current commit downgrades the curl version to 7.65.0.

Close #4288

(cherry picked from commit 2e880af0)

Implemented GitLab CI testing process additionally to existing Travis
CI. The new testing process is added to run tests faster. It requires to
control a load of machines to avoid flaky fails on timeouts. GitLab CI
allows us to run testing on our machines.

Created 2 stages for testing and deploying packages.

The testing stage contains the following jobs that are run for all
branches:

* Debian 9 (Stretch): release/debug gcc.
* Debian 10 (Buster): release clang8 + lto.
* OSX 14 (Mojave): release.
* FreeBSD 12: release gcc.

And the following jobs that are run of long-term branches (release
branches: for now it is 1.10, 2.1 and master):

* OSX 13 (Sierra): release clang.
* OSX 14 (Mojave): release clang + lto.

The deployment stage contains the same jobs as we have in Travis CI.
They however just build tarballs and packages: don't push them to S3 and
packagecloud.

In order to run full testing on a short-term branch one can name it with
'-full-ci' suffix.

The additional manual work is needed when dependencies are changed in
.travis.mk file ('deps_debian' or 'deps_buster_clang_8' goals):

 | make GITLAB_USER=foo -f .gitlab.mk docker_bootstrap

This command pushes docker images into GitLab Registry and then they are
used in testing. Pre-built images speed up testing.

Fixes #4156

(cherry picked from commit ce623a23)

Travis-ci failed on brew install python2 due to it was
already installed on OSX 10.13 Sierra, so it needs
to be installed without fails on its exists.

Follow up #4254

(cherry picked from commit ceefd0c8)

Removed skip flag file to switch on the testing of the
http_client test. Enabled http_client test on OSX,
fixed missing of the python2 symlink. Removed the subtest
on '595 error return' from 'error' suite, due to it may
hang forever. To enable test on travis-ci reverted commit:

1d7285c4 ('Disable flaky http_client.test.lua')

Closes #4254

(cherry picked from commit 33254bd6)

Issue pending, gh-4254.

(cherry picked from commit 1d7285c4)

Disabled tarballs and packages building on short-term branches.

Removed 'allow_failures' on coverage / debug build.

Replaced matrix expansion with the list of jobs (because Travis-CI
documentation says it does not support condition jobs with matrix
expansion).

Fixes #3755.

(backported from 28fcdaa0)

Fixed app/strict.test.lua fail after a test that sets 'a' global
variable. It was due to the way how 'strict' module tracks declared
global variables and was fixed in pretest_clean.lua test-run's module.

(cherry picked from commit 1df9c3d2)

* Implement SQL driver (#4123).
* Support line carrying with backslash.
* Support new result file format (for new tests).

See the link below for a full description of the new features.

https://github.com/tarantool/test-run/commit/a04b5b096c607172ce4fc86a84e3531c9f3a7304

Fixes #4123.

(cherry picked from commit d81d7b1c)

Fixed issues:

- box-py/iproto.test.py
  1) Fixed receive_response() to wait for whole response.
  2) Clean up _cluster space.

- replication-py/cluster.test.py
  1) Clean up _cluster space.

- replication-py/multi.test.py
  1) Removed vclock checking because it fails if previous test make some
     DML and vclock was incremented. Looks like it was used for debug
     and is not part of this test case.
  2) Fixed typo in 'Synchronize' block.

The following test sequences did fail due to unexpected IDs in _cluster
space:

- [box-py/iproto.test.py, null]
- [box-py/bootstrap.test.py, null]

- [replication-py/cluster.test.py, null]
- [replication-py/multi.test.py, null]

Part of #3232

(cherry picked from commit 4ea7d729)

Closes #3824

(cherry picked from commit 2aa25ba5)

region_reset() only frees memory from the last slab. As a result, if
a vylog transaction happens to use more than one slab, memory used by
vy_log.pool won't be freed, neither will it be reused, i.e. we'll get
a memory leak. Fix it by using region_free() instead of region_reset().
It's okay from performance point of view, because vylog transactions
are rare events.

Note, the master branch isn't affected to this issue, because the vylog
memory management was completely reworked there. 2.1 isn't affected
either, because there region_reset() was modified to free all slabs.
However, rather than backporting any of those commits, I think it's
more appropriate to simply use region_free().

vy_stmt_new_surrogate_from_key() and vy_stmt_new_surrogate_delete_raw()
allocate temporary objects on the region, but don't clean up after
themselves. Those functions may be called by a vinyl reader threads:

  vy_page_read_cb
    vy_page_find_key
      vy_page_stmt
        vy_stmt_decode

In this case the region will grow infinitely, because reader threads
never call fiber_gc().

The leak was introduced to 1.10 by commit b9072317 ("vinyl: lookup
key in reader thread"), which moved vy_page_find_key() invocation to
reader threads for the sake of performance. The fix is trivial - call
region_truncate() from those functions.

Note, neither the master branch nor 2.1 is affected to this issue,
because region_truncate() was added there in the scope of the multikey
index feature.

Add 'decimal' to a built-in modules list to preserve
package.loaded.decimal when pretest_clean is set in suite.ini.

Needed for #692.

(cherry picked from commit 0b7cc526)

Built-in modules are bundled into tarantool in the following way. A lua
file from src/lua or src/box/lua is stored as a string literal in a C
file, then built and linked into tarantool. During startup tarantool
calls luaL_loadbuffer() on this string.

When a Lua source is converted to a C literal, proper escaping is
performed. However there is one case, which was not covered: trigraphs.
The patch adds escaping of question mark symbols to avoid matching ??X
sequences as trigraphs by C preprocessor.

The most simple way to check that it works is to apply the following
patch:

 | diff --git a/src/lua/string.lua b/src/lua/string.lua
 | index 6e12c59ae..2da2dbf4d 100644
 | --- a/src/lua/string.lua
 | +++ b/src/lua/string.lua
 | @@ -425,3 +425,6 @@ string.fromhex    = string_fromhex
 |  string.strip      = string_strip
 |  string.lstrip      = string_lstrip
 |  string.rstrip      = string_rstrip
 | +string.foo = function()
 | +    return '??('
 | +end

And call the function like so:

 | ./src/tarantool -e 'print(string.foo()) os.exit()'

If it printfs `??(`, then everything is okay. If it prints `[`, then
`??(` was preprocessed as the trigraph.

We hit this problem when tried to bundle luarocks-3: it contains
"^(.-)(%??)$" regexp, where `??)` was interpreted as `]`. Debug build or
a build with -DENABLE_WERROR reports an error in the case, but usual
RelWithDebInfo build passes (with -Wtrigraphs warnings) and can show
this unexpected behaviour.

Fixes #4291.

(cherry picked from commit 177a1713)

vy_tx_handle_deferred_delete() expects (righteously) that in case a
deferred DELETE overwrites a statement in a secondary index write set
and the overwritten statement wasn't skipped on commit (i.e. doesn't
have txv->is_overwritten flag set), both the old and the new statement
must be REPLACEs (see the comment to the corresponding assertions for
more details).

The problem is we don't set is_overwritten flag in case a statement
doesn't have any effect (txv->is_nop is set), even if it was, in fact,
overwritten in the primary index write set (see vy_tx_prepare). As
a result, we get an assertion failure when we delete such statement
in the same transaction, e.g.

  s = box.schema.space.create('test', {engine = 'vinyl'})
  s:create_index('pk', {parts = {1, 'unsigned'}})
  s:create_index('sk', {parts = {2, 'unsigned'}})
  s:replace{1, 1, 1}
  box.begin()
  s:update(1, {{'+', 3, 1}}) -- adds no-op REPLACE to sk write set
  s:delete(1)                -- produces deferred DELETE for sk
  box.commit()

results in

  vy_tx_handle_deferred_delete: Assertion `vy_stmt_type(stmt) == IPROTO_REPLACE' failed.

Fix this by making sure we set is_overwritten for all overwritten
statements in a secondary index write set.

Closes #4294

(cherry picked from commit 41219774)

Added to cmake environment CMAKE_DL_LIBS (The name of the library
that has dlopen and dlclose in it, usually -ldl) to openssl build
to add DL library, to fix the following fails:

Linking CXX executable crypto.test
/usr/local/lib64/libcrypto.a(dso_dlfcn.o): In function `dlfcn_globallookup':
dso_dlfcn.c:(.text+0x11): undefined reference to `dlopen'
dso_dlfcn.c:(.text+0x24): undefined reference to `dlsym'
dso_dlfcn.c:(.text+0x2f): undefined reference to `dlclose'
/usr/local/lib64/libcrypto.a(dso_dlfcn.o): In function `dlfcn_bind_func':
dso_dlfcn.c:(.text+0x334): undefined reference to `dlsym'
dso_dlfcn.c:(.text+0x3f2): undefined reference to `dlerror'
/usr/local/lib64/libcrypto.a(dso_dlfcn.o): In function `dlfcn_load':
dso_dlfcn.c:(.text+0x459): undefined reference to `dlopen'
dso_dlfcn.c:(.text+0x4c9): undefined reference to `dlclose'
dso_dlfcn.c:(.text+0x502): undefined reference to `dlerror'
/usr/local/lib64/libcrypto.a(dso_dlfcn.o): In function `dlfcn_pathbyaddr':
dso_dlfcn.c:(.text+0x5a1): undefined reference to `dladdr'
dso_dlfcn.c:(.text+0x601): undefined reference to `dlerror'
/usr/local/lib64/libcrypto.a(dso_dlfcn.o): In function `dlfcn_unload':
dso_dlfcn.c:(.text+0x662): undefined reference to `dlclose'
collect2: error: ld returned 1 exit status
make[2]: *** [test/unit/crypto.test] Error 1
make[1]: *** [test/unit/CMakeFiles/crypto.test.dir/all] Error 2

Closes #4245

(cherry picked from commit 31ab6ea9)

Run collectgarbage() between tests to ensure that there are no dangling
iterators and so on. Such objects can affect statistic counters that may
be important for a test.

Fixes #4287.

(cherry picked from commit 0cbcf265)

Currently to determine current source path, user has to do it
by getting current source info from the stack and parse it manually

This commit adds couple little helpers for this task called
`debug.sourcefile()` and `debug.sourcedir()`,
which returns a path to the file being executed.

The path is relative to the current working directory.
If such path cannot be determined, '.' is returned (i.e. interactive
mode). There is an exception if you are running script with an absolute
path. In that case, absolute path will be returned too.

```bash
tarantool /absolute/path/to/the/script.lua
```

There are also handy shortcuts for that functions: `debug.__file__` and
`debug.__dir__`

@TarantoolBot document
Title: Document new debug helper methods

There are two more helpers in debug module `sourcefile()` and
`sourcedir()`. This two helpers returns relative paths to source file
and directory respectively.

There are also `debug.__file__` and `debug.__dir__` which are just handy
ways to call `sourcefile()` and `sourcedir()`.

It should be mentioned that it is only possible to determine real path
to the file, if the function was defined in a some lua file.
`loadstring` may be exception here, since lua will store whole string
in debug info.

There is also a possible pitfall because of tail call optimization, so
if the function `fn` is defined inside `myapp/init.lua` like this:

```lua
function fn()
    return debug.sourcefile()
end
```

It would not be possible to determine file path correctly because that
function would not appear in a stacktrace. Even worse, it may return
wrong result, i.e. actual path where "parent" function was defined.

To force the interpreter to avoit this kind of optimizations you may use
parenthesis like this:

```lua
function fn()
    return (debug.sourcefile())
end
```

Also both `sourcefile` and `sourcedir` functions have optional level
argument, which allows set the level of the call stack function should
examine for source path. By default, 2 is used.

(cherry picked from commit 70b8583c)

After compacting runs, we first mark them as dropped (VY_LOG_DROP_RUN),
then try to delete their files unless they are needed for recovery from
the checkpoint, and finally mark them as not needed in the vylog
(VY_LOG_FORGET_RUN). There's a potential race sitting here: the problem
is the garbage collector might kick in after files are dropped, but
before they are marked as not needed. If this happens, there will be
runs that have two VY_LOG_FORGET_RUN records, which will break recovery:

  Run XX is forgotten, but not registered

The following patches make the race more likely to happen so let's
eliminate it by making the garbage collector the only one who can mark
runs as not needed (i.e. write VY_LOG_FORGET_RUN record). There will
be no warnings, because the garbage collector silently ignores ENOENT
errors, see vy_gc().

Another good thing about this patch is that now we never yield inside
a vylog transaction, which makes it easier to remove the vylog latch
blocking implementation of transactional DDL.

(cherry picked from commit 30c9c7c0)

Example:
```
[001] box/on_replace.test.lua                                         [ pass ]
[001] box/on_replace.test.lua                                         [ fail ]
[001]
[001] Test failed! Result content mismatch:
[001] --- box/on_replace.result	Tue Jun  4 10:44:39 2019
[001] +++ box/on_replace.reject	Tue Jun  4 10:44:50 2019
[001] @@ -4,7 +4,7 @@
[001]  -- test c and lua triggers: must return only lua triggers
[001]  #box.space._space:on_replace()
[001]  ---
[001] -- 0
[001] +- 2
[001]  ...
```
This happened because we forgot to remove triggers set on _space.
Clear them below the test case.

(cherry picked from commit d3552264)

This commit fixes a regression introduced by commit
5ab0763b
(pass operation type to lua triggers)

When passing operation type to the trigger, we relied on a corresponding
xrow_header, which would later be written to the log. When before_replace
triggers are fired, there is no row connected to the txn statement yet,
so we faked one to pass operations. We, however, didn't account for
temporary spaces, which never have a row associated with a corresponding
txn stmt. This lead to a segfault on an attemt to use on_replace triggers
with temporary spaces.

Add a fake row for temporary spaces to pass operation type in on_replace
triggers and add some tests.

Closes #4266

(cherry picked from commit 63ebc25a)

Since DDL is triggered by the admin, it can be deliberately initiated
when the workload is known to be low. Throttling it along with DML
requests would only cause exasperation in this case. So we don't apply
disk-based rate limit to DDL. This should be fine, because the
disk-based limit is set rather strictly to let the workload some space
to grow, see vy_regulator_update_rate_limit(), and in contrast to the
memory-based limit, exceeding the disk-based limit doesn't result in
abrupt stalls - it may only lead to a gradual accumulation of disk space
usage and read latency.

Closes #4238

(cherry picked from commit 6a44ac62)