Make sure we're allowed to setup json formatter before
box.cfg() call, ie that named boot-time logger.

Part-of #5121

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

There is no reason to not allow for json formatter
on early logging stage.

We add verification that

	box.cfg{log="syslog:", log_format="json"}
or
	require('log').cfg{log="syslog:", format="json"}

is triggering error since syslog output requires
predefined structure and can't use json.

Fixes #5121

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

For some reason in commit 09832455 we've disabled
to use json format in boot time logger. There is
no reason to do so.

Only syslog output format is predefined and must not
be changed, in turn json format is just a decoration
over output stream so we can use it whenever requested.

Part-of #5121

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

Accidentally assignment is used in assertions instead of comparison
operation. Let's fix this mistake and use comparison.

We never use this method so no need to waste space.

In-scope-of #4842

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

Introduced in 157beda5
and never used since.

In-scope-of #4842

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

Last time used in 1d979029

In-scope-of #4842

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

Introduced in 157beda5
but never used since.

In-scope-of #4842

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

Found issue running test on FreeBSD VBox host:

 [011] --- box/net.box_wait_connected_gh-3856.result	Mon Jun 15 09:39:49 2020
 [011] +++ box/net.box_wait_connected_gh-3856.reject	Fri May  8 08:23:30 2020
 [011] @@ -12,7 +12,8 @@
 [011]  - opts:
 [011]      wait_connected: false
 [011]    host: 8.8.8.8
 [011] -  state: initial
 [011] +  state: error
 [011] +  error: Invalid argument
 [011]    port: '123456'
 [011]  ...
 [011]  c:close()

A. Turenko made deep investigation and found that the reason of the
fail was that getaddrinfo() returned EIA_SERVICE for an incorrect
TCP/IP port on FreeBSD, but crops it as modulo of 65536 on Linux/glibc.
Checked with his local script './getaddrinfo':

  (Linux/glibc) $ ./getaddrinfo 8.8.8.8 123456
  ----
  family: AF_INET
  socktype: SOCK_STREAM
  protocol: IPPROTO_TCP
  host: 8.8.8.8
  serv: 57920

  (FreeBSD) $ ./getaddrinfo 8.8.8.8 123456
  getaddrinfo: Service was not recognized for socket type

So obvious fix is to change 123456 to something less or equal to
65535. Say, 1234.

The test depended on an order in which fibers were scheduled
(net_box.connect() creates a separate fiber for connecting in background
using fiber.create(), which yields). Unlikely our fiber were not get
execution time during the connection attempt, so it was more like a
formal thing.

But we can decrease probability of this situation even more if we'll
grab all connection fields just when net_box.connect() returns, not
after yield in console (which is due to waiting a next command from
test-run).

Closes #5083

Co-authored-by: Alexander Turenko <alexander.turenko@tarantool.org>
Co-authored-by: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>

Found issue (reproduced on VBox FreeBSD machine):

 [016] --- replication/wal_rw_stress.result	Fri Feb 21 11:53:21 2020
 [016] +++ replication/wal_rw_stress.reject	Fri May  8 08:23:56 2020
 [016] @@ -73,7 +73,42 @@
 [016]  ...
 [016]  box.info.replication[1].downstream.status ~= 'stopped' or box.info
 [016]  ---
 [016] -- true
 [016] +- version: 2.5.0-27-g32f59756a
 [016] +  id: 2
 [016] +  ro: false
 [016] +  uuid: 41cbebcc-9105-11ea-96ab-08002739cbd6
 [016] +  package: Tarantool
 [016] +  cluster:
 [016] +    uuid: 397c196f-9105-11ea-96ab-08002739cbd6
 [016] +  listen: unix/:/home/vagrant/tarantool/test/var/016_replication/replica.socket-iproto
 [016] +  replication:
 [016] +    1:
 [016] +      id: 1
 [016] +      uuid: 397a1886-9105-11ea-96ab-08002739cbd6
 [016] +      lsn: 10005
 [016] +      upstream:
 [016] +        status: follow
 [016] +        idle: 0.46353673400017
 [016] +        peer: unix/:/home/vagrant/tarantool/test/var/016_replication/master.socket-iproto
 [016] +        lag: -0.45732522010803
 [016] +      downstream:
 [016] +        status: stopped
 [016] +        message: writev(1), called on fd 24, aka unix/:/home/vagrant/tarantool/test/var/016_replicati
 [016] +        system_message: Broken pipe
 [016] +    2:
 [016] +      id: 2
 [016] +      uuid: 41cbebcc-9105-11ea-96ab-08002739cbd6
 [016] +      lsn: 0
 [016] +  signature: 10005
 [016] +  status: running
 [016] +  vinyl: []
 [016] +  uptime: 2
 [016] +  lsn: 0
 [016] +  sql: []
 [016] +  gc: []
 [016] +  pid: 41231
 [016] +  memory: []
 [016] +  vclock: {1: 10005}
 [016]  ...
 [016]  test_run:cmd("switch default")
 [016]  ---

To check the downstream status and it's message need to wait until an
downstream appears. This prevents an attempt to index a nil value when
one of those functions are called before a record about a peer appears
in box.info.replication. It was observed on test:
  replication/show_error_on_disconnect
after commit
  c6bea65f ('replication: recfg with 0
quorum returns immediately').

Checked that test still checks the error for which it was created at
b9db91e1 ('xlog: fix fallocate vs
read race') patch and successfully got the needed error "tx checksum
mismatch":

[153] --- replication/wal_rw_stress.result      Fri Jun 19 15:01:49 2020
[153] +++ replication/wal_rw_stress.reject      Fri Jun 19 15:04:02 2020
[153] @@ -73,7 +73,43 @@
[153]  ...
[153]  test_run:wait_cond(function() return box.info.replication[1].downstream.status ~= 'stopped' end) or box.info
...
[153] +      downstream:
[153] +        status: stopped
[153] +        message: tx checksum mismatch

Note that wait_cond() allows to overcome a transient network
connectivity errors, but 'tx checksum mismatch' is persistent
one and will be catched.

Closes #4977

Found issue:

[003] --- replication/wal_off.result	Thu Apr 25 13:10:18 2019
[003] +++ replication/wal_off.reject	Tue Jul 16 17:10:31 2019
[003] @@ -95,6 +95,8 @@
[003]  ...
[003]  while string.find(box.info.replication[wal_off_id].upstream.message, check) == nil do fiber.sleep(0.01) end
[003]  ---
[003] +- error: '[string "while string.find(box.info.replication[wal_of..."]:1: bad argument
[003] +    #1 to ''find'' (string expected, got nil)'
[003]  ...
[003]  box.cfg { replication = "" }
[003]  ---

To check the upstream status and it's message need to wait until an
upstream appears. This prevents an attempt to index a nil value when
one of those functions are called before a record about a peer appears
in box.info.replication. It was observed on test:
  replication/show_error_on_disconnect
after commit
  c6bea65f ('replication: recfg with 0
quorum returns immediately').

Closes #4355

<box_process_lua> function created a new GCfunc object for a handler
having no upvalues depending on the request context on each call.

The change introduces the following mapping:
| <handler id> -> <handler GCfunc object>
Initializing this mapping on Tarantool startup is aimed to reduce Lua GC
memory usage.

Reviewed-by: Sergey Ostanevich <sergos@tarantool.org>
Reviewed-by: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Signed-off-by: Igor Munkin <imun@tarantool.org>

JIT compiler can generate an invalid trace for <fun.chain> iterator
(i.e. chain_gen_r1) breaking its semantics (see LuaJIT/LuaJIT#584).
Since interpreter works fine and produces the right results, disabling
JIT for this function stops execution failures.

As a result box-tap/key_def.test.lua is removed from box-tap suite
fragile tests list.

Relates to LuaJIT/LuaJIT#584
Fixes #4252

Reviewed-by: Alexander V. Tikhonov <avtikhon@tarantool.org>
Reviewed-by: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Signed-off-by: Igor Munkin <imun@tarantool.org>

Data read in vinyl is known to yield in case of disc access. So it opens
a window for modifications of in-memory level. Imagine following scenario:
right before data selection tuple is inserted into space. It passes first
stage of commit procedure, i.e. it is prepared to be committed but still
is not yet reached WAL.  Meanwhile iterator is starting to read the same key.
At this moment prepared statement is already inserted to in-memory tree
ergo visible to read iterator. So, read iterator fetches this statement
and proceeds to disk scan.  In turn, disk scan yields and in this moment
WAL fails to write statement on disk. Next, two cases are possible:
1. WAL thread has enough time to complete rollback procedure.
2. WAL thread fails to finish rollback in this time gap.

In the first case read iterator should skip statement: version of
in-memory tree has diverged from iterator's one, so we fall back into
iterator restoration procedure. Mem iterator might become invalid so
the only choice is to restart whole 'advance' routine.
Let's don't try to restore it and always restart iteration cycle if
L0 level has changed during yield.

In the second case nothing is changed to read iterator, so it simply
returns prepared statement (and it is considered to be OK).

Closes #3395

vy_page_find_key() assumes that equal_key parameter is initialized since
it is used unconditionally. Originally, function was designed with
assumption that parameter is initialized by caller. Since then it has
been used in several other places, but some callers doesn't initialize
this parameter to 'false' value. Let's fix it and inside
vy_page_find_key() set this output parameter to false by default.

Closes #5078

Calling box.cfg{} more than once does not normally cause any errors
(even though it might not have any effect). In contrast, assigning
it to some variable and then using it after the box was configured
caused an error since the method was overwritten by the initial call
of <load_cfg>.

The patch fixes this issue making box.cfg behave consistently in both
scenarios.

Follow-up #4231

Co-developed-by: Alexander Turenko <alexander.turenko@tarantool.org>

<box_load_and_execute> checks whether box is configured with appropriate
locking and configures it when necessary. However it is not so for
<lbox_execute>. We should replace the former with the latter only when
box is fully loaded.

Follow-up #4231

box.execute() initializes box if it is not initialized. For this sake,
box.execute() is another function (so called <box_load_and_execute>)
when box is not loaded: it loads box and calls 'real' box.execute().
However it is not enough: <box_load_and_execute> may be saved by a user
before box initialization and called after box loading or during box
loading from a separate fiber.

Note: calling <box_load_and_execute> during box loading is safe now, but
calling of box.execute() is not: the 'real' box.execute() does not
verify whether box is configured. It will be fixed in a further commit.

This commit changes <box_load_and_execute> to verify whether box is
initialized and to load box only when it is not loaded. Also it adds
appropriate locking around load_cfg() invocation from
<box_load_and_execute>.

While we're here, clarified contracts of functions that set box
configuration options.

Closes #4231

Co-developed-by: Alexander Turenko <alexander.turenko@tarantool.org>

- test: don't use not aligned size for mempool

No need for +1 byte here, PATH_MAX already implies
end of string.

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

This is more consistent than relying that array size
will remain PATH_MAX forever.

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

Similar to dirname there is no need for +1 byte.
Same time make sure xlog_open never end up without
trailing zero.

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

The PATH_MAX is the longest path including end
of string, no need for +1 byte.

Same time use sizeof(dirname) to not bound how
exactly dirname is declared.

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

This makes structure less in size and eliminates useless
padding (both enum and fd are integers 4 bytes long).

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

There is serious "inline disease" in the code:
it spread left and right without a serious reason.

The box_cfg_xc function is a pretty big one and
doesn't require being inlined anyhow.

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

Since currently query planner is unable to use HASH indexes
and attempt to use it will likely lead to SEGFAULT, this
patch raises an error on attempt to open VDBE cursor
against HASH index.

@TarantoolBot document
Title: Doceument allowed index type for SQL
Before the change, Tarantool query planner segfaulted on
try of using non-tree index. It is blocked now w/ appropriate
error message. Need to document the behaviour.
It should be noted, that this restriction might be relaxed in future.

Closes #4659

Writing less bytes than requested is perfectly fine. In turn out
that fio.write/pwrite api simply returns 'true' even if only some
part of a buffer has been written.

Thus make coio_write and coio_pwrite to write the whole data in
a cycle. Note in most situations there will be only one pass,
partial writes are really the rare cases.

Note that we're not handling nonblocking writes here (which
could return EAGAIN) simply because we need an other api
which would accept timeouts.

Fixes #4651

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

Since 527b02a2 (memtx: add yields
during index build) memtx_build_on_replace was introduced to handle
concurrent updates. The problem here was that the tuples being handled
with this trigger did not get reference counter promotion, leading to a
number of wrong behavior cases. Now this problem is solved.
This problem was found through primary index altering with updates in
background fiber. Corresponding test is introduced.

Closes #4973

Clang undefined behaviour sanitizer was turned on using
-fsanitize=undefined flag, which is supposed to turn on all the
sanitizations, except a few ones. Not needed sanitations were
turned off explicitly, using -fno-sanitize=<type> flags. However
appeared it does not work with some flags. For example,
nullability sanitations can't be turned off when
-fsanitize=undefined is used.

Nullability sanitations lead to lots of false-positive fails
such as typeof(*obj) where obj is NULL, or memcpy() with NULL
destination but 0 size.

The patch splits -fsanitize=undefined into separate flags and
never turns on nullability checks.

Part of #4609

SQL heavily depends on box, and box on SQL. So they can't be
separate libraries. The build started failing with undefined box
symbols in SQL, when code of the latter has slightly changed in
one of the recent commits.

The build failed only with UB sanitizer enabled, but
'VERBOSE=1 make' showed that both with UB and without UB the build
command was the same (not counting -fsanitize flags). So the
sanitizer has nothing to do with it.

The patch makes SQL sources being built as a part of box library.

Closes #5067

Fix bug with the display of collation for scalar fields in
<SELECT> result, when sql_full_metadata is enabled.

Closes #4755

Found issue:

 [009] --- replication/status.result	Wed May  6 09:03:18 2020
 [009] +++ replication/status.reject	Tue May 12 15:55:09 2020
 [009] @@ -307,11 +307,12 @@
 [009]  ...
 [009]  r.upstream.status == "stopped"
 [009]  ---
 [009] -- true
 [009] +- false
 [009]  ...
 [009]  r.upstream.message:match('Duplicate') ~= nil
 [009]  ---
 [009] -- true
 [009] +- error: '[string "return r.upstream.message:match(''Duplicate'') ..."]:1: attempt
 [009] +    to index field ''message'' (a nil value)'
 [009]  ...
 [009]  test_run:cmd('switch default')
 [009]  ---

To check the upstream status and it's message need to wait until an
upstream appears. This prevents an attempt to index a nil value when
one of those functions are called before a record about a peer appears
in box.info.replication. It was observed on test:
  replication/show_error_on_disconnect
after commit
  c6bea65f ('replication: recfg with 0
quorum returns immediately').

Closes #4969

Found the issue on regular testing hosts: [1]

  Fetching changes...
  00:04
   Reinitialized existing Git repository in
     /home/gitlab-runner/builds/zzyC6hh5/0/tarantool/tarantool/.git/
   Checking out 8ff7f32c as ...
   warning: failed to remove CMakeFiles/Makefile.cmake

Found the job that saved the directories with root permissions: [2]

The issue appeared because the job that saved directories with root
permissions used the 'shell' runner to run docker container inside.
It caused the gitlab-runner to run the default workspace cleanup
outside the docker container. In opposite to it, when 'docker' runner
is used, the cleanup routine runs inside the docker container and no
fails ever exist, because the root permissions are used in the docker
container and this is the same root permissions for the host. As the
result using 'shell' runner, cleanup routine failed to remove files
created by root inside the docker container and which were shared to
global host with the same permissions, because gitlab-runner runs the
'shell' runner by regular 'gitlab-runner' user, but not by root.

To fix the issue need to run docker containers using the gitlab-runner
only in RO mode with Out-Of-Source builds in it. Either use the
'docker' runners when docker containers are needed. Anyway 'shell'
runner jobs with additional calls to docker containers can't be
control for the branches of developers, to avoid of it need to make
local cleanup routine instead of default to the working paths for each
job with 'shell' runners use.

Decided to setup gitlab-runner configuration as described in: [3]

Also got the issue with left data from previous builds at the
submodule pathes, like here: [4]

   Undefined symbols for architecture x86_64:
     "_u_isprint_66", referenced from:
         _yaml_emitter_is_printable in libyaml_static.a(emitter.c.o)
   ld: symbol(s) not found for architecture x86_64
   clang: error: linker command failed with exit code 1 (...)

Also got issues with left files from previously tested branches,
like here: [5]

 [087] small/rlist.test
 [087] TAP13 parse failed (Missing plan in the TAP source).
 [087]
 [087] No result file (small/rlist.result) found.
 [087] Run the test with --update-result option to write the new result file.
 [087] [ fail ]

 [087] small/static.test
 [087] TAP13 parse failed (Missing plan in the TAP source).
 [087]
 [087] No result file (small/static.result) found.
 [087] Run the test with --update-result option to write the new result file.
 [087] [ fail ]

To fix it was added the command to clean all available git submodules:
  git submodule foreach git clean -ffdx

Closes #5036

1. https://gitlab.com/tarantool/tarantool/-/jobs/577884238#L7
2. https://gitlab.com/tarantool/tarantool/-/jobs/577768553
3. https://docs.gitlab.com/ce/ci/yaml/README.html#git-clean-flags
4. https://gitlab.com/tarantool/tarantool/-/jobs/574199256#L3141
5. https://gitlab.com/tarantool/tarantool/-/jobs/590573606#L3718

To fix deprecation warning, CMP0037 policy was changed to NEW for cmake
3.11 and above.

CMP0037 old behavior (cmake 2.8.12) allowed target names such as test.
In cmake 3.10 and below names test, help and etc. were reserved.

Starting from cmake 3.11 these names are only reserved when the
corresponding feature is enabled (e.g. by including the CTest or CPack
modules). Tarantool does not use CTest so the name test can be used.

Closes #3587

Set full testing with deploy builds and tests for all branches.

box/net.box_bad_argument_gh-594.test.lua
box/net.box_call_blocks_gh-946.test.lua
box/net.box_collectgarbage_gh-3107.test.lua
box/net.box_connect_timeout_gh-2054.test.lua
box/net.box_connect_triggers.test.lua
box/net.box_console_connections_gh-2677.test.lua
box/net.box_count_inconsistent_gh-3262.test.lua
box/net.box_discard_gh-3107.test.lua
box/net.box_disconnect_gh-3859.test.lua
box/net.box_fiber-async_gh-3107.test.lua
box/net.box_field_names_gh-2978.test.lua
box/net.box_get_connection_object.test.lua
box/net.box_gibberish_gh-3900.test.lua
box/net.box_huge_data_gh-983.test.lua
box/net.box_incompatible_index-gh-1729.test.lua
box/net.box_incorrect_iterator_gh-841.test.lua
box/net.box_index_unique_flag_gh-4091.test.lua
box/net.box_iproto_hangs_gh-3464.test.lua
box/net.box_is_nullable_gh-3256.test.lua
box/net.box_leaks_gh-3629.test.lua
box/net.box_log_corrupted_rows_gh-4040.test.lua
box/net.box_long-poll_input_gh-3400.test.lua
box/net.box_methods_gh-3107.test.lua
box/net.box_msgpack_gh-2195.test.lua
box/net.box_on_schema_reload-gh-1904.test.lua
box/net.box_password_gh-1545.test.lua
box/net.box_permissions.test.lua
box/net.box_pseudo_objects_gh-2401.test.lua
box/net.box_raw_response_gh-3107.test.lua
box/net.box_readahead_gh-3958.test.lua
box/net.box_reconnect_after_gh-3164.test.lua
box/net.box_reconnect_after.test.lua
box/net.box_reload_schema_gh-636.test.lua
box/net.box_remote_method_gh-544.test.lua
box/net.box_roll_back_gh-822.test.lua
box/net.box_schema_change_gh-2666.test.lua
box/net.box_schema_change_gh-3107.test.lua
box/net.box_session_type_gh-2642.test.lua
box/net.box_space_format_gh-2402.test.lua
box/net.box_stack_diag_gh-1148.test.lua
box/net.box_timeout_gh-1533.test.lua
box/net.box_timeout-gh-3107.test.lua
box/net.box_upsert_gh-970.test.lua
box/net.box_uri_first_arg_gh-398.test.lua
box/net.box_wait_connected_gh-3856.test.lua

Also removed the skip condition file:
  box/net.box.skipcond
mentioned as temporary workaround for enabling testing
on FreeBSD within issue #4271, because no really issues
with the tests on it found.

Closes #4880

Explicitly notify about a missing newline at end of a .result file in
GNU Diff like style:

 |  ---
 | -...
 | \ No newline
 | +...

Closes https://github.com/tarantool/test-run/issues/212

It may turn out that dump_generation does not catch up with current
generation and no other dump tasks are in progress. This may happen
dump process is throttled due to errors. In this case generation is
bumped but dump_generation is not (since dump is not completed). In
turn, throttling opens a window for DDL operations. For instance, index
dropping and creation of new one results in mentioned situation:

box.snapshot() -- fails for some reason; next attempt at dumping will be
               -- taken in one second.
s:drop() -- drop index to be dumped
s = box.schema.space.create('test', {engine = 'vinyl'})
-- create new one (its mem generation is greater than scheduler's one)
i = s:create_index('pk')

Closes #4821