This patch fixes memory leak in lbox_tuple_format_new().

Closes #4588

(cherry picked from commit 96199855)

Box.session.su() worked like following: check user
existence, create its credentials on the stack, check
the function, call the function, destroy the
credentials, restore the old credentials.

After creating the credentials on the stack the
function check could raise a Lua error. It led to the
credentials object not being destroyed. As a result,
user.credentials_list was pointing at invalid memory.

Now there is no errors between creating the temporary
credentials and its destruction.

Closes #4597

(cherry picked from commit 2bb8d1ea)

This function is supposed to return NULL on an error.
For exceptions there is user_find_by_name_xc.

(cherry picked from commit 8b6bdb43)

Func_delete() called credentials_destroy() after
func->vtab->destroy(). But appeared, that vtab->destroy() is
actually delete, and it frees the func object. Now the func's
owner credentials are destroyed before the function is freed.

Closes #4597
Follow up #2763

(cherry picked from commit 330ea240)

Argparse module stores unspecified parameter values as boolean
true. It led to a problem, that a command line '--value' with
'value' defined as a number or a string, showed a strange error
message:

    Expected number/string, got "true"

Even though a user didn't pass any value. Now it shows 'nothing'
instead of '"true"'. That is clearer.

Follow up #4076

(cherry picked from commit c214d086)

There was a complaint that tarantoolctl --show-system option is
very hard to use. It incorrectly parsed passed values, and
provided strange errors.

    tarantoolctl cat --show-system true
    Bad input for parameter "show-system". Expected boolean, got "true"

    tarantoolctl cat --show-system 1
    Bad input for parameter "show-system". Expected boolean, got "1"

    tarantoolctl cat --show-system=true
    Bad input for parameter "show-system". Expected boolean, got "true"

First of all, appeared that the complaining people didn't read
documentation in 'tarantoolctl --help'. It explicitly says, that
'--show-system' should go after a file name, and does not have a value.

Secondly, even having taken the documentation into account, the
errors indeed look ridiculous. 'Expected boolean, got "true"'
looks especially weird.

The problem appeared to be with argparse module, how it parses
boolean parameters, and how stores parameter values not specified
in a command line.

All parameters were parsed into a dictionary: parameter name ->
value. If a name is alone (no value), then it is boolean true.
Otherwise it was always a string value. An attempt to specify
an explicit parameter value 'true' led to storing string 'true'
in that dictionary.

Consequential check for boolean parameters was trivial:
type(value) == 'boolean', which was obviously wrong, and didn't
pass for 'true' string, but passed for an empty value.

Closes #4076

(cherry picked from commit 03f85d4c)

Credentials is a cache of user universal privileges. And that
cache can become outdated in case user privs were changed after
creation of the cache.

The patch makes user update all its credentials caches with new
privileges, via a list of all creds.

That solves a couple of real life problems:

- If a user managed to connect after box.cfg started listening
port, but before access was granted, then he needed a reconnect;

- Even if access was granted, a user may connect after box.cfg
listen, but before access *is recovered* from _priv space. It
was not possible to fix without a reconnect. And this problem
affected replication.

Closes #2763
Part of #4535
Part of #4536

@TarantoolBot document
Title: User privileges update affects existing sessions and objects
Previously if user privileges were updated (via
`box.schema.user.grant/revoke`), it was not reflected in already
existing sessions and objects like functions. Now it is.

For example:
```
        box.cfg{listen = 3313}
        box.schema.user.create('test_user', {password = '1'})
        function test1() return 'success' end

        c = require('net.box').connect(box.cfg.listen, {
                user = 'test_user', password = '1'
        })
        -- Error, no access for this connection.
        c:call('test1')

        box.schema.user.grant('test_user', 'execute', 'universe')
        -- Now works, even though access was granted after
        -- connection.
        c:call('test1')
```

A similar thing happens now with `box.session.su` and functions
created via `box.schema.func.create` with `setuid` flag.

In other words, now user privileges update is reflected
everywhere immediately.

(cherry picked from commit 06dbcec597f14fae6b3a7fa2361f2ac513099662)
(cherry picked from commit 2b599c0efa9ae265fb7464af6abae3f6a192e30e)

Struct credentials is a cache of user's universal privileges. It
is static and is never changed after creation. That is a problem.
If a user privileges are updated, it is not reflected in his
existing credentials caches.

This patch reworks credentials API so as now this struct is not
just a container for several numbers. It is an object with
standard methods like create(), destroy(). A credentials object
still is not updated together with its source user, but now at
least the API allows to fix that.

Next patch will link all struct credentials of a user into a list
via which the user will be able to keep the credentials up to
date.

Part of #2763

(cherry picked from commit a8c3ebdbfc97b72832ebc5d87b681a310cce9589)
(cherry picked from commit 6b15dce614cfc3b14a12b66819737263a5089eaf)

Before the patch there was a race in replication
password configuration. It was possible that a replica
connects to a master with a custom password before
that password is actually set. The replica treated the
error as critical and exited.

But in fact it is not critical. Replica even can
withstand absence of a user and keeps reconnecting.
Wrong password situation arises from the same problem
of non atomic configuration and is fixed the same -
keep reconnect attempts if the password was wrong.

Closes #4550

(cherry picked from commit aa2e2c56)

The previous patch introduced a way to set box.cfg options
in a strict order, even on a reconfiguration. It was used to set
listen before replication.
The same order problem existed for replication settings. A user
could do

    box.cfg{
        replication_connect_quorum = 0,
        replication = {...}
    }

and expect, that due to quorum 0 the cfg() will return
immediately. But actually the behaviour was undefined - due to
arbitrary order of keys in a Lua table, replication could be
applied before quorum.

The patch makes all replication settings be applied before
replication.

Follow up #4433
Part of #3760

(cherry picked from commit 00c6c437)

Before the patch the nil UUID was ignored and a new random one
was generated. This was because internally box treats nil UUID
as its absence.

Now a user will see an explicit message that nil UUID is a
reserved value.

Closes #4282

(cherry picked from commit a8ebd334)

Is_nullable JSON-path indexes used to lose is_nullable property
being defined on a space having format. This patch fixes this
problem.

Closes #4520

(cherry picked from commit 3d01b3af)

Functional index extractor code used to raise an invalid
error message when the user-defined extractor function
returns a scalar instead of a table.

Closes #4553

(cherry picked from commit 260a3b3a)

Wrap throwing lua_newthread in luaT_newthread using luaT_cpcall
to process arising error properly.

Closes #4556

(cherry picked from commit 54e23d6d)

End recovery (which means building secondary indexes) just after
last known log file was read. This allows fast switch to hot standby
instance without any delay for secondary index to be built.
Due to engine_end_recovery carryover, xdir_collect_inprogress,
previously being called from it, is now moved to garbage collector.

Closes #4135

(cherry picked from commit 5aa243de)

Console client's eval() method in case of an error at
reading from a socket was trying to return a variable
declared in a different view scope. Instead, the
error should be raised to drop the connection.

Reviewed-by: Alexander Turenko <alexander.turenko@tarantool.org>
(cherry picked from commit 89ec1d97)

When we handle unix console connection we should setup
eos if only we meet "\set output" command in a stream.
For example there could be "\set language" which should
not affect eos settings.

Closes #4568

Reported-by: Alexander Turenko <alexander.turenko@tarantool.org>
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
Reviewed-by: Alexander Turenko <alexander.turenko@tarantool.org>
(cherry picked from commit 092cd675)

This will allow us to reuse this map in other routines
and same time optimize memory allocation a bit.

Part-of #4568

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
Reviewed-by: Alexander Turenko <alexander.turenko@tarantool.org>
(cherry picked from commit 44bfb4f1)

Rows_per_wal option was deprecated because it can be covered by
wal_max_size. In order not to complicate WAL code with that
option's support this commit drops it completely.

In some tests the option was used to create several small xlog
files. Now the same is done via wal_max_size. Where it was
needed, number of rows per wal is estimated as wal_max_size / 50.
Because struct xrow_header size ~= 50 not counting paddings and
body.

Note, file box/configuration.result was deleted here, because it
is a stray result file, and it contained the rows_per_wal option
mentioning. Its test was dropped much earlier in
fdc3d1dd.

Closes #3762

(cherry picked from commit c6012920)

Replication quorum 0 not only affects orphan status, but also,
according to documentation, makes box.cfg() return immediately
regardless of whether connections to upstreams are established.

It was not so before the patch. What is worse, even with non 0
quorum the instance was blocked on reconfiguration for connect
timeout seconds, if at least one node is not connected.

Now quorum is respected on reconfiguration. On a bootstrap it is
still impossible to return earlier than
replication_connect_timeout, because nodes need to choose some
cluster settings. Too early start would make it impossible -
cluster's participants will just start and choose different
cluster UUIDs.

Closes #3760

(cherry picked from commit c6bea65f)

A successfully fetched remote instance ballot isn't updated during
bootstrap procedure. This leads to a case when different instances
choose different masters as their bootstrap leaders.

Imagine such a situation.
You start instance A without replication set up. Instance A successfully
bootstraps.
You also have instances B and C both with replication set up to {A, B,
C} and replication_connect_quorum set to 3
You first start instance B. It doesn't proceed to choosing a leader
until one of the events happens: either the replication_connect_timeout
runs out, or instance C is up and starts listening on its port.
B has established connection to A and fetched its ballot, with some
vclock, say, {1: 1}.
B retries connection to C every replication_timeout seconds.
Then you start instance C. Instance C succeeds in connecting to A and B
right away and bootstraps from instance A. Instance A registers C in its
_cluster table. This registration is replicated to instance C.
Meanwhile, instance C is trying to sync with quorum instances (which is
3), and stays in orphan mode.
Now replication_timeout on instance B finally runs out. It retries a
previously unsuccessful connection to C and succeeds. C sends its ballot
to B with vclock = {1: 2, 2:0} (in our example), since it has already
incremented it after _cluster registration.
B sees that C has a greater vclock than A, and chooses to bootstrap from
C instead of A. C is orphan and rejects B's attempt to join. B dies.

To fix such ungentlemanlike behaviour of C, we should at least include
loading status in ballot and prefer fully bootstrapped instances to the
ones still syncing with other replicas.
We also need to use a separate flag instead of ballot's already existent
is_ro, since we still want to prefer loading instances over the ones
explicitly configured to be read-only.

Closes #4527

(cherry picked from commit dc1e4009)

During rework of the console lua mode series
the declaration of variable has been lost and
this cause test case for remote unix console
connection to fail.

Fixes issue from c358398c

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
(cherry picked from commit df821d0f)

To be consistent in naming scheme, for example
we already have parse_operators.

Part-of #3834

Reviewed-by: Konstantin Osipov <kostja.osipov@gmail.com>
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
(cherry picked from commit af9ef211)

The other modules would be able to find out which eos marker
is currently active. For example when reading replies from
remote server via text based console protocol.

@TarantoolBot document
Title: > require('console').eos()

Returns a string with currently active end of string marker.

Part-of #3834

Reviewed-by: Konstantin Osipov <kostja.osipov@gmail.com>
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
(cherry picked from commit 9d83b5f6)

If we change output mode on remote machine via
text-based session

 node a (server)
 ------
 require('console').listen('unix/:/tmp/X.sock')
 ...

 node b (client)
 ------
 require('console').connect('unix/:/tmp/X.sock')
 connected to unix/:/tmp/X.sock
 ...
 unix/:/tmp/X.sock> \set output lua

the client get stuck forever, it is because the text
wire protocol waits for yaml eos terminator which of
course never hit the peer, because lua mode uses own
one.

Thus to fix this problem we have to preprocess the text
we're passing to the server, just like we do in local
console. So we reuse command parsing code and remember
current output terminator in text_connection_mt instance.

Another problem is that named default output mode. There
could be a mixed environment where server operates in
default lua mode but client connects with yaml mode. To
break a tie we yield "\set output" command with current
output mode when establishing a connection. Since the
output format is per-session bound this doesn't affect
any other connections on a server.

Part-of #3834

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
(cherry picked from commit c358398c)

We will need to parse and verify "\set" commands
in remote console text-wire protocol thus to not
duplicate the code lets factor helper out for
reuse sake.

Part-of #3834

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
(cherry picked from commit 96dbc49d)

This will help us to distinguish end of string/stream
in text protocols (such as remote console connection).

Note that we start printing ";" terminator for every
lua output. Actually current yaml output does the
same but inside console.c module.

And since lua output is yet a new feature in stabilization
phase we're safe to make such changes without breaking api.

Part-of #3834

Reviewed-by: Konstantin Osipov <kostja.osipov@gmail.com>
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
(cherry picked from commit d67808fa)

This allows lua to not create them every call since
they are constant speeding up processing a bit.

Part-of #3834

Reviewed-by: Konstantin Osipov <kostja.osipov@gmail.com>
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
(cherry picked from commit 840a7646)

Sole symbols (such as box.NULL) are not processed
by serpent "custom" symbols feature, since they
are not in table.

Thus we should process them separately. Without it
we have

 > require('console').set_default_output("lua,block")
 > ;
 > box.NULL
 > "cdata<void %*>: NULL";

instead of

 > box.NULL
 > box.NULL;

as it should be.

Part-of #3834

Reviewed-by: Konstantin Osipov <kostja.osipov@gmail.com>
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
(cherry picked from commit 6847367f)

fiber_stack_destroy() doesn't depend on cord() helper but
rather accepts slab cache pointer in arguments, lets
make the same in fiber_stack_create() helper, otherwise
it looks quite imbalanced.

Signed-off-by: Kirill Yukhin <kyukhin@tarantool.org>
(cherry picked from commit fa6d78b397d375d4c18875ac72b3798b2e06a1e8)

The mprotect() syscall may fail due to heavy memory
pressure (say there is no enough resources to split
VMAs and etc).

In this case we must not proceed without guard page
but refuse to create a new fiber.

Closes #4541

Signed-off-by: Kirill Yukhin <kyukhin@tarantool.org>
(cherry picked from commit e58dfe07189134297ea956eb70cd7d96e20a6c6e)

This will allow us to call it from inside fiber_stack_create
as we need to clean up resources on error.

Part-of #4541

Signed-off-by: Kirill Yukhin <kyukhin@tarantool.org>
(cherry picked from commit 8f1251d3611aa30d5bb6098716e965fd8dc08551)

Code to run main script (passed via command line args, or
interactive console) has a footer where it notifies systemd,
logs a happened error, and panics.

Before the patch that code was unreachable in case of any
exception in a main script, because panic happened earlier. Now a
happened exception is correctly carried to the footer with proper
error processing.

A first and obvious solution was replace all panics with diag_set
and use fiber_join on the script runner fiber. But appeared, that
the fiber running a main script can't be joined. This is because
normally it exits via os.exit() which never returns and therefore
its caller never dies = can't be joined.

The patch solves this problem by passing main fiber diag to the
script runner by pointer, eliminating fiber_join necessity.

Closes #4382

(cherry picked from commit 157a2d88)

Closes #4434
Follow-up #4366

@TarantoolBot document
Title: json/msgpack.cfg.encode_deep_as_nil option

Tarantool has several so called serializers to convert data
between Lua and another format: YAML, JSON, msgpack.

YAML is a crazy serializer without depth restrictions. But for
JSON, msgpack, and msgpackffi a user could set encode_max_depth
option. That option led to crop of a table when it had too many
nested levels. Sometimes such behaviour is undesirable.

Now an error is raised instead of data corruption:

    t = nil
    for i = 1, 100 do t = {t} end
    msgpack.encode(t) -- Here an exception is thrown.

To disable it and return the old behaviour back here is a new
option:

    <serializer>.cfg({encode_deep_as_nil = true})

Option encode_deep_as_nil works for JSON, msgpack, and msgpackffi
modules, and is false by default. It means, that now if some
existing users have cropping, even intentional, they will get the
exception.

(cherry picked from commit d7a8942a)

Tuple is a C library exposed to Lua. In Lua to translate Lua
objects into tuples and back luaL_serializer structure is used.

In Tarantool we have several global serializers, one of which is
for msgpack. Tuples store data in msgpack, and in theory should
have used that global msgpack serializer. But in fact the tuple
module had its own private serializer because of tuples encoding
specifics such as never encode sparse arrays as maps.

This patch makes tuple Lua module use global msgpack serializer
always. But how does tuple handle sparse arrays now? In fact,
the tuple module still has its own serializer, but it is updated
each time when the msgpack serializer is changed.

Part of #4434

(cherry picked from commit 676369b1)

Msgpack Lua module is not a simple set of functions. It is a
global serializer object used by plenty of other Lua and C
modules. Msgpack as a serializer can be configured, and in theory
its configuration updates should affect all other modules. For
example, a user could change encode_max_depth:

    require('msgpack').cfg({encode_max_depth = <new_value>})

And that would make tuple:update() accept tables with <new_value>
depth without a crop.

But in fact msgpack configuration didn't affect some places, such
as this one. And all the others who use msgpackffi.

This patch fixes it, for encode_max_depth option. Other options
are still ignored.

Part of #4434

(cherry picked from commit 4bb253f7)

There are some objects called serializers - msgpack, cjson, yaml,
maybe more. They are global objects affecting both Lua and C
modules.

A serializer have settings which can be updated. But before the
patch an update changed only C structure of the serializer. It
made impossible to use settings of the serializers from Lua.

Now any update of any serializer is reflected both in its C and
Lua structures.

Part of #4434

(cherry picked from commit fe4a8047)

Since ARRAY and MAP cannot be converted to SCALAR type, this
operation should throw an error. But when the error is raised in
SQL, it is displayed in unreadable form. The reason for this is
that the given array or map is not correctly converted to a
string. This patch fixes the problem by converting ARRAY or MAP to
their string representation.
For example:

box.execute('CREATE TABLE t1(i INT PRIMARY KEY, a SCALAR);')
format = {}
format[1] = {type = 'integer', name = 'I'}
format[2] = {type = 'array', name = 'A'}
s = box.schema.space.create('T2', {format=format})
i = s:create_index('ii')
s:insert({1, {1,2,3}})
box.execute('INSERT INTO t1 SELECT * FROM t2;')

Should return:
- error: 'Type mismatch: can not convert [1, 2, 3] to scalar'

Follow-up #4189

(cherry picked from commit 736cdd81)

Function mem_apply_type() implements implicit type conversion. As
a rule, tuple to be inserted to the space is exposed to this
conversion which is invoked during execution of OP_MakeRecord
opcode (which in turn forms tuple). This function was not adjusted
to operate on ARRAY, MAP and ANY field types since they are poorly
supported in current SQL implementation. Hence, when tuple to be
inserted in space having mentioned field types reaches this
function, it results in error. Note that we can't set ARRAY or MAP
types in SQL, but such situation may appear during UPDATE
operation on space created via Lua interface. This problem is
solved by extending implicit type conversions with obvious casts:
array field can be casted to array, map to map and any to any.

Closes #4189

(cherry picked from commit de79b714)

Method fio.mktree is used to create given path unconditionally -
without checking if it was a directory or something else. This
led to inappropriate error messages or even inconsistent behavior.
Now check the type of a given path.

Closes #4439

(cherry picked from commit 8ccfc691)