Code to run main script (passed via command line args, or
interactive console) has a footer where it notifies systemd,
logs a happened error, and panics.

Before the patch that code was unreachable in case of any
exception in a main script, because panic happened earlier. Now a
happened exception is correctly carried to the footer with proper
error processing.

A first and obvious solution was replace all panics with diag_set
and use fiber_join on the script runner fiber. But appeared, that
the fiber running a main script can't be joined. This is because
normally it exits via os.exit() which never returns and therefore
its caller never dies = can't be joined.

The patch solves this problem by passing main fiber diag to the
script runner by pointer, eliminating fiber_join necessity.

Closes #4382

(cherry picked from commit 157a2d88)

Closes #4434
Follow-up #4366

@TarantoolBot document
Title: json/msgpack.cfg.encode_deep_as_nil option

Tarantool has several so called serializers to convert data
between Lua and another format: YAML, JSON, msgpack.

YAML is a crazy serializer without depth restrictions. But for
JSON, msgpack, and msgpackffi a user could set encode_max_depth
option. That option led to crop of a table when it had too many
nested levels. Sometimes such behaviour is undesirable.

Now an error is raised instead of data corruption:

    t = nil
    for i = 1, 100 do t = {t} end
    msgpack.encode(t) -- Here an exception is thrown.

To disable it and return the old behaviour back here is a new
option:

    <serializer>.cfg({encode_deep_as_nil = true})

Option encode_deep_as_nil works for JSON, msgpack, and msgpackffi
modules, and is false by default. It means, that now if some
existing users have cropping, even intentional, they will get the
exception.

(cherry picked from commit d7a8942a)

Tuple is a C library exposed to Lua. In Lua to translate Lua
objects into tuples and back luaL_serializer structure is used.

In Tarantool we have several global serializers, one of which is
for msgpack. Tuples store data in msgpack, and in theory should
have used that global msgpack serializer. But in fact the tuple
module had its own private serializer because of tuples encoding
specifics such as never encode sparse arrays as maps.

This patch makes tuple Lua module use global msgpack serializer
always. But how does tuple handle sparse arrays now? In fact,
the tuple module still has its own serializer, but it is updated
each time when the msgpack serializer is changed.

Part of #4434

(cherry picked from commit 676369b1)

Msgpack Lua module is not a simple set of functions. It is a
global serializer object used by plenty of other Lua and C
modules. Msgpack as a serializer can be configured, and in theory
its configuration updates should affect all other modules. For
example, a user could change encode_max_depth:

    require('msgpack').cfg({encode_max_depth = <new_value>})

And that would make tuple:update() accept tables with <new_value>
depth without a crop.

But in fact msgpack configuration didn't affect some places, such
as this one. And all the others who use msgpackffi.

This patch fixes it, for encode_max_depth option. Other options
are still ignored.

Part of #4434

(cherry picked from commit 4bb253f7)

There are some objects called serializers - msgpack, cjson, yaml,
maybe more. They are global objects affecting both Lua and C
modules.

A serializer have settings which can be updated. But before the
patch an update changed only C structure of the serializer. It
made impossible to use settings of the serializers from Lua.

Now any update of any serializer is reflected both in its C and
Lua structures.

Part of #4434

(cherry picked from commit fe4a8047)

The submodule was on 7.65.3 version. This update closes two security
problems:

https://curl.haxx.se/docs/CVE-2019-5482.html
https://curl.haxx.se/docs/CVE-2019-5481.html

See also curl-7.66.0 release notes:

https://curl.haxx.se/changes.html#7_66_0

Fixes #4502.

(cherry picked from commit 4c2d1eff)

Found that the curl failed to build on FreeBSD with errors:
gmake[2]: Entering directory '/home/vagrant/tarantool/third_party/curl/src'
  CCLD     curl
/usr/local/bin/ld: ../lib/.libs/libcurl.so: undefined reference to `SSLv23_client_method'
/usr/local/bin/ld: ../lib/.libs/libcurl.so: undefined reference to `CONF_modules_free'
/usr/local/bin/ld: ../lib/.libs/libcurl.so: undefined reference to `ERR_free_strings'
/usr/local/bin/ld: ../lib/.libs/libcurl.so: undefined reference to `sk_value'
/usr/local/bin/ld: ../lib/.libs/libcurl.so: undefined reference to `ENGINE_cleanup'
/usr/local/bin/ld: ../lib/.libs/libcurl.so: undefined reference to `SSL_library_init'
/usr/local/bin/ld: ../lib/.libs/libcurl.so: undefined reference to `EVP_MD_CTX_destroy'
/usr/local/bin/ld: ../lib/.libs/libcurl.so: undefined reference to `sk_pop_free'
/usr/local/bin/ld: ../lib/.libs/libcurl.so: undefined reference to `SSLeay'
/usr/local/bin/ld: ../lib/.libs/libcurl.so: undefined reference to `SSL_get_ex_new_index'
/usr/local/bin/ld: ../lib/.libs/libcurl.so: undefined reference to `OPENSSL_add_all_algorithms_noconf'
/usr/local/bin/ld: ../lib/.libs/libcurl.so: undefined reference to `SSL_COMP_free_compression_methods'
/usr/local/bin/ld: ../lib/.libs/libcurl.so: undefined reference to `EVP_MD_CTX_create'
/usr/local/bin/ld: ../lib/.libs/libcurl.so: undefined reference to `EVP_cleanup'
/usr/local/bin/ld: ../lib/.libs/libcurl.so: undefined reference to `sk_num'
/usr/local/bin/ld: ../lib/.libs/libcurl.so: undefined reference to `sk_pop'
/usr/local/bin/ld: ../lib/.libs/libcurl.so: undefined reference to `SSL_load_error_strings'
collect2: error: ld returned 1 exit status
gmake[2]: *** [Makefile:921: curl] Error 1

Found root cause of the issue at the `./configure <...>` output:
 | checking for OpenSSL headers version... 1.0.2 - 0x1000214fL
 | checking for OpenSSL library version... 1.1.1
 | configure: WARNING: OpenSSL headers and library versions do not match.
It is seen that the Tarantool bootstrap installed pkg 'openssl'
of the version '1.0.2', while the currently default FreeBSD 'openssl'
version was '1.1.1'.

Anyway we don't need any special openssl version installed against
default one, so the fix is just to remove the openssl package from
bootstrap installation.
Also found that some installing packages are not needed too,
removed it from FreeBSD bootstrap.

Additionally added libiconv library into bootstrap which is needed
as workaround to avoid of the issue described in:
	https://github.com/tarantool/tarantool/issues/3791

Closed #4490

(cherry picked from commit 1f2338bd)

Since ARRAY and MAP cannot be converted to SCALAR type, this
operation should throw an error. But when the error is raised in
SQL, it is displayed in unreadable form. The reason for this is
that the given array or map is not correctly converted to a
string. This patch fixes the problem by converting ARRAY or MAP to
their string representation.
For example:

box.execute('CREATE TABLE t1(i INT PRIMARY KEY, a SCALAR);')
format = {}
format[1] = {type = 'integer', name = 'I'}
format[2] = {type = 'array', name = 'A'}
s = box.schema.space.create('T2', {format=format})
i = s:create_index('ii')
s:insert({1, {1,2,3}})
box.execute('INSERT INTO t1 SELECT * FROM t2;')

Should return:
- error: 'Type mismatch: can not convert [1, 2, 3] to scalar'

Follow-up #4189

(cherry picked from commit 736cdd81)

Function mem_apply_type() implements implicit type conversion. As
a rule, tuple to be inserted to the space is exposed to this
conversion which is invoked during execution of OP_MakeRecord
opcode (which in turn forms tuple). This function was not adjusted
to operate on ARRAY, MAP and ANY field types since they are poorly
supported in current SQL implementation. Hence, when tuple to be
inserted in space having mentioned field types reaches this
function, it results in error. Note that we can't set ARRAY or MAP
types in SQL, but such situation may appear during UPDATE
operation on space created via Lua interface. This problem is
solved by extending implicit type conversions with obvious casts:
array field can be casted to array, map to map and any to any.

Closes #4189

(cherry picked from commit de79b714)

Method fio.mktree is used to create given path unconditionally -
without checking if it was a directory or something else. This
led to inappropriate error messages or even inconsistent behavior.
Now check the type of a given path.

Closes #4439

(cherry picked from commit 8ccfc691)

If the rock was packed with luarocks then the time in it is set to zeros
and this is not the case with the unix format can be removed when date given
is added to zipwriter_open_new_file_in_zip (luarocks/luarocks#1056)

Closes #4481

(cherry picked from commit 3658cb00)

It was forgotten to swap old and new mask (holding fields involved into
foreign key relation) during space alteration (lists of object
representing FK metadata are swapped successfully). Since mask is vital
and depending on its value different byte-codes implementing SQL query
can be produced, this mistake resulted in assertion fault in debug build
and wrong constraint check in release build. Let's fix this bug and swap
masks as well as foreign key lists.

Closes #4495

(cherry picked from commit 33236ecc)

This commit does not change a user visible behaviour. It refactors pwd
module to explicitly divide code that fetches data from passwd / group
databases from code that performs deserialization of the data to Lua
tables.

The idea of splitting of those actions appears when it was observed that
a call of getpw() / getgr() leads to problems on some systems when it is
invoked during passwd / group database traveral.

Now it is more obvious that we don't call getpw() during passwd
traversal and getgr() during group traveral.

Follows up #4428 and #4447.

(cherry picked from commit 6c5d3f06)

LTO build fails on warning message:
In file included from /tarantool/src/lib/core/diag.h:33:0,
                 from /tarantool/src/box/engine.h:36,
                 from /tarantool/src/box/memtx_engine.h:40,
                 from /tarantool/src/box/memtx_engine.c:31:
/tarantool/src/box/memtx_engine.c: In function
'metmx_tuple_chunk_delete':
/tarantool/src/trivia/util.h:201:49: error: initialization from
incompatible pointer type [-Werror]
  const typeof( ((type *)0)->member  ) *__mptr = (ptr); \
                                                 ^
/tarantool/src/box/memtx_engine.c:1115:3: note: in expansion of macro
'container_of'
   container_of((typeof(tuple_chunk->data) *)data,
   ^
/tarantool/src/trivia/util.h:201:49: error: (near initialization for
'tuple_chunk') [-Werror]
  const typeof( ((type *)0)->member  ) *__mptr = (ptr); \
                                                 ^
/tarantool/src/box/memtx_engine.c:1115:3: note: in expansion of macro
'container_of'
   container_of((typeof(tuple_chunk->data) *)data,

Closes #4438

(cherry picked from commit 97012004)

This patch provides a test suite that allows us to make sure that
the SQL BOOLEAN type works as intended.

Part of #4228

(cherry picked from commit 7cf84a54)

In a configuration with several read-only and read-write instances, if
replication_connect_quorum is not greater than the amount of read-only
instances and replication_connect_timeout happens to be small enough
for some read-only instances to form a quorum and exceed the timeout
before any of the read-write instaces start, all these read-only
instances will choose themselves a read-only bootstrap leader.
This 'leader' will successfully bootstrap itself, but will fail to
register any of the other instances in _cluster table, since it isn't
writeable. As a result, some of the read-only instances will just die
unable to bootstrap from a read-only bootstrap leader, and when the
read-write instances are finally up, they'll see a single read-only
instance which managed to bootstrap itself and now gets a
REPLICASET_UUID_MISMATCH error, since no read-write instance will
choose it as bootstrap leader, and will rather bootstrap from one of
its read-write mates.

The described situation is clearly not what user has hoped for, so
throw an error, when a read-only instance tries to initiate the
bootstrap. The error will give the user a cue that he should increase
replication_connect_timeout.

Closes #4321

@TarantoolBot document
Title: replication: forbid to bootstrap read-only masters.

It is no longer possible to bootstrap a read-only instance in an emply
data directory as a master. You will see the following error trying to
do so:
```
ER_BOOTSTRAP_READONLY: Trying to bootstrap a local read-only instance as master
```
Now if you have a fresh instance, which has
`read_only=true` in an initial `box.cfg` call, you need to set up
replication from an instance which is either read-write, or has your
local instance's uuid in its `_cluster` table.

In case you have multiple read-only and read-write instances with
replication set up, and you still see the aforementioned error message,
this means that none of your read-write instances managed to start
listening on their port before read_only instances have exceeded the
`replication_connect_timeout`. In this case you should raise
`replication_connect_timeout` to a greater value.

(cherry picked from commit 037bd58c)

* All test chunks related to luajit were moved from tarantool source
tree to the luajit repo
* Adjusted CMakeLists via creating a symlink to luajit test directory
to fix out-of-source tests

Closed #4478

(cherry picked from commit 43575303)

rows_per_wal does not allow to properly limit size of WAL logs.
A user would need to estimate size of each WAL row to do that.
Now WAL supports option wal_max_size allowing to limit WAL size
much more precise.

Part of #3762

(cherry picked from commit c7a382af)

The json.encode() used to cause a segfault in case of recursive
table:
  tbl = {}
  tbl[1] = tbl
  json.encode(tbl)

Library doesn't test whether given object on Lua stack parsed
earlier, because it performs a lightweight in-depth traverse
of Lua stack. However it must stop when encode_max_depth is
reached (by design).

Tarantool's lua_cjson implementation has a bug introduced during
porting original library: it doesn't handle some corner cases:
entering into a map correctly increases a current depth, while
entering into an array didn't. This patch adopts author's
approach to check encode_max_depth limit. Thanks to handling this
constraint correctly the segfault no longer occurs.

Closes #4366

(cherry picked from commit d2641e66e65ab57317b8e70bb7115517ec81238e)

The test sql-tap/gh-3083-ephemeral-unref-tuples.test.lua took
too long runtime and failed to finish in timeout limit of 2
minutes when it runs near the end of the running queue:

No output during 120 seconds. Will abort after 120 seconds without output. List of workers not reporting the status:
- 012_sql-tap [sql-tap/gh-3083-ephemeral-unref-tuples.test.lua, vinyl] at var/012_sql-tap/gh-3083-ephemeral-unref-tuples.result:0
Test hung! Result content mismatch:
[File does not exists: sql-tap/gh-3083-ephemeral-unref-tuples.result]
[Main process] No output from workers. It seems that we hang. Send SIGKILL to workers; exiting...

Due to suggestion from Konstantin Osipov at gh-3845:
https://github.com/tarantool/tarantool/issues/3845#issue-385735959

The following tests:
  gh-3083-ephemeral-unref-tuples.test.lua
  gh-3332-tuple-format-leak.test.lua
where set to run only on memtx configuration.

Also the test:
  gh-3083-ephemeral-unref-tuples.test.lua
was removed from the test-run 'fragile' option flaky list.

Closed #4128

(cherry picked from commit 40c94af7)

We know that madvise(MADV_DONTDUMP) is present
on linux based platforms only (macos doesn't
support it at all, freebsd requires MADV_NOCORE
or similar which is unsupported by now) thus
we should not print a warning on other systems
to not spam users.

Fixes #4464

(cherry picked from commit 51d8e4c3)

Added "fragile" option to the flaky tests that are
not intended to be run in parallel with others.
Option set at the suite.ini file at the appropriate
suites with comments including the issue that stores
the fail.

(cherry picked from commit 165f8ee6)

Removed the jobs that is interesting to check on gitlab-ci
instead of travis-ci:
  - osx 13 "Sierra"
  - all LTO jobs
  - ASAN job

Part of #4410

(cherry picked from commit 05477787)

This is a followup for 7691154a

(cherry picked from commit ff14626d)

In case of throwing client error because of inactive function
we did not destroy used port. It could possibly cause huge
memory leaks as could be seen with top or its analogues when
performing net.box test run in a loop.

Closes #4388

(cherry picked from commit 7691154a)

It turned out that patch d0e38d59 wasn't accompanied
w/ a test. Add proper check.

(cherry picked from commit c80e9416)

Currently we only enter orphan mode when instance fails to connect to
replication_connect_quorum remote instances during local recovery.
On bootstrap and manual replication configuration change an error is
thrown. We better enter orphan mode on manual config change, and leave
it only in case we managed to sync with replication_connect_quorum
instances.

Closes #4424

@TarantoolBot document
Title: document reaction on error in replication configuration change.

Now when issuing `box.cfg{replication={uri1, uri2, ...}}` and failing to
sync with replication_connect_quorum remote instances, the server will
throw an error, if it is bootstrapping, and just set its state to orphan
in all other cases (recovering from existing xlog/snap files or manually
changing box.cfg.replication on the fly). To leave orphan mode, you may
wait until the server manages to sync with replication_connect_quorum
instances.
In order to leave orphan mode you need to make the server sync with
enough instances. To do so, you may either:
1) set replication_connect_quorum to a lower value
2) reset box.cfg.replication to exclude instances that cannot
   be reached or synced with
3) just set box.cfg.replication to "" (empty string)

(cherry picked from commit 5a0cfe02)

In 3-node replica-set, registering with the leader node
does not guarantee the registration record arrives to the second
peer immediately. The joining node may bootstrap faster than
the registration record arrives to the second peer, in which
case replication will fail to create a full mesh.

(cherry picked from commit a850acfd)

A follow up on gh-4305, fix failing args.test.py

(cherry picked from commit 83ef5a17)

(cherry picked from commit ef14050f)

To make it possible to develop and test related features on
systems without systemd.

WITH_SYSTEMD cmake flag is used to generate systemd related files:
unit, generator script, etc. To keep this behavior and make it possible
to use NOTIFY_SOCKET without other systemd-related stuff,
I added WITH_NOTIFY_SOCKET cmake flag.

It also required some changes to support other OS:

SOCK_CLOEXEC (not available on macOS) flag for socket()
is replaced with `fcntl(fd, F_SETFD, FD_CLOEXEC)` which has the same effect.

MSG_NOSIGNAL flag for sendmsg is also not available on macOS.
However it has SO_NOSIGPIPE flag for setsockopt which disables SIGPIPE.
So it requires different solution for different OS. Inspired by
https://nwat.xyz/blog/2014/01/16/porting-msg_more-and-msg_nosigpipe-to-osx/

Have to reduce send-buffer size to 4MB because larger values
are not supported on macOS by default. This value should be enough
for all systems because notification messages are usually less than 1KB.

Fixes #4436

(cherry picked from commit 1e509dde)

System-wide dynamic libraries usually (always?) have NEEDED and RUNPATH
tags in a dynamic section (as `readelf -d /usr/lib/lib<...>.so` shows),
so when we link, say, with libssl.so, which depends on libz.so, a linker
does not complain against unresolved symbols that can be found in Z
library (if it is installed within a system).

Things are different when we linking with a static library. Say, when we
linking with libssl.a, which contains an unresolved symbol from Z
library, a linker reports an error. It is not possible to store an
information where to find unresolved symbols (NEEDED / RUNPATH) in a
static library (AFAIK).

We depend on three libraries that are depend on Z library: libcurl,
libssl and libcrypto (two latter are part of OpenSSL). When one of those
libraries is linked statically we should link with libz.so or libz.a
(depending on BUILD_STATIC flag). The patch doing exactly this.

The patch changes OPENSSL_LIBRARIES variable to fix the issue with
static linking of OpenSSL libraries. It also changes CURL_LIBRARIES in
the same way, however this does not alter any visible behaviour, because
OPENSSL_LIBRARIES is added to CURL_LIBRARIES. The latter change was made
to unify the way to choose libraries to link with: it is pure
refactoring part.

Fixes #4437.

(cherry picked from commit 2cdfaf3b)

A manual action is needed after pulling of this commit to actually use
the downstream repository instead of the upstream one:

sed -i -e 's@https://github.com/curl/curl.git@https://github.com/tarantool/curl.git@' .git/config

It is part of our processes to use forked repositories for submodules.

Suggested by Konstantin Osipov.

(cherry picked from commit 5a46639f)

Cleaned up the .gitlab-ci.yml file from duplicating
code - added templates that storres the needed
configuration values for different jobs.
Also moved the static_build from 'deploy' tag to
'deploy_test' tag to be sure that tests will not
be run under high load.

* Added "fragile" option with a list of tests that are not intended to
  be run in parallel with others. The format of the list is the same as
  for "disabled" option (#187, PR #188).
* Ensured that non-parallel test suites and fragile tests will be run
  only when all parallel ones will be finished (PR #188).
* Fixed reporting in case of lack of temporary result (PR #172).

(cherry picked from commit 418ab172)

CentOS 6 and FreeBSD 12 implementations of getpwuid() rewind
setpwent()-{getpwent()}-endpwent() loop to a start that leads to a hang
during pwd.getpwall() invoke. The same is true for a getgrgid() call
during setgrent()-{getgrent()}-endgrent() loop.

The commit modifies pwd module to avoid getpwuid() calls during passwd
database traversal and to avoid getgrgid() calls when traversing groups.

The commit also fixes the important regression on CentOS 6 after
f5d8331e ('lua: workaround
pwd.getpwall() issue on Fedora 29'): tarantool hungs during startup due
to added getpwall() call. This made tarantool unusable on CentOS 6 at
all.

Aside of that the commit fixes another pwd.getgrall() problem: the
function gaves password entries instead of group entries.

Fixes #4428.
Fixes #4447.
Part of #4271.

(cherry picked from commit 7753d350)

app-tap.tarantoolctl.test.lua fails after
17df9edf ('tarantoolctl: allow to start
instances with delayed box.cfg{}').

The commit fixes the test case that did check that an error is reported
if box.cfg() was not called in an instance script.

Follows up #4435.
Fixes #4448.

(cherry picked from commit 6c627af3)

This change is related to #4391. The objective was to collect additional
information about modules, but it's hard to do without changing API.

This patch will allow to monkey-patch report generation and achieve the
same results without interfering the daemon behavior.

(cherry picked from commit 5b9f207d)

There is a problem with calculating .msg_namelen field
of msghdr struct. Instead of

    .msg_name   = &sa,
    .msg_namelen = sizeof(sa.sun_family) + strlen(sd_unix_path),

it must set as

    .msg_namelen = sizeof(sa) // larger value than current invalid one

It works on linux but when I tried to enable this feature for macOS
it didn't (maybe because of different order of fields in the struct).

Instead of fixing calculation, I've replaced original sendmsg call
with sendto, because it's a convenient shortcut which
simplifies code and can prevent such mistakes.

Required for #4436

(cherry picked from commit 89aae30c)