To be consistent in naming scheme, for example
we already have parse_operators.

Part-of #3834

Reviewed-by: Konstantin Osipov <kostja.osipov@gmail.com>
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

The other modules would be able to find out which eos marker
is currently active. For example when reading replies from
remote server via text based console protocol.

@TarantoolBot document
Title: > require('console').eos()

Returns a string with currently active end of string marker.

Part-of #3834

Reviewed-by: Konstantin Osipov <kostja.osipov@gmail.com>
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

If we change output mode on remote machine via
text-based session

 node a (server)
 ------
 require('console').listen('unix/:/tmp/X.sock')
 ...

 node b (client)
 ------
 require('console').connect('unix/:/tmp/X.sock')
 connected to unix/:/tmp/X.sock
 ...
 unix/:/tmp/X.sock> \set output lua

the client get stuck forever, it is because the text
wire protocol waits for yaml eos terminator which of
course never hit the peer, because lua mode uses own
one.

Thus to fix this problem we have to preprocess the text
we're passing to the server, just like we do in local
console. So we reuse command parsing code and remember
current output terminator in text_connection_mt instance.

Another problem is that named default output mode. There
could be a mixed environment where server operates in
default lua mode but client connects with yaml mode. To
break a tie we yield "\set output" command with current
output mode when establishing a connection. Since the
output format is per-session bound this doesn't affect
any other connections on a server.

Part-of #3834

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

We will need to parse and verify "\set" commands
in remote console text-wire protocol thus to not
duplicate the code lets factor helper out for
reuse sake.

Part-of #3834

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

This will help us to distinguish end of string/stream
in text protocols (such as remote console connection).

Note that we start printing ";" terminator for every
lua output. Actually current yaml output does the
same but inside console.c module.

And since lua output is yet a new feature in stabilization
phase we're safe to make such changes without breaking api.

Part-of #3834

Reviewed-by: Konstantin Osipov <kostja.osipov@gmail.com>
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

This allows lua to not create them every call since
they are constant speeding up processing a bit.

Part-of #3834

Reviewed-by: Konstantin Osipov <kostja.osipov@gmail.com>
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

Sole symbols (such as box.NULL) are not processed
by serpent "custom" symbols feature, since they
are not in table.

Thus we should process them separately. Without it
we have

 > require('console').set_default_output("lua,block")
 > ;
 > box.NULL
 > "cdata<void %*>: NULL";

instead of

 > box.NULL
 > box.NULL;

as it should be.

Part-of #3834

Reviewed-by: Konstantin Osipov <kostja.osipov@gmail.com>
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

fiber_stack_destroy() doesn't depend on cord() helper but
rather accepts slab cache pointer in arguments, lets
make the same in fiber_stack_create() helper, otherwise
it looks quite imbalanced.

Signed-off-by: Kirill Yukhin <kyukhin@tarantool.org>

The mprotect() syscall may fail due to heavy memory
pressure (say there is no enough resources to split
VMAs and etc).

In this case we must not proceed without guard page
but refuse to create a new fiber.

Closes #4541

Signed-off-by: Kirill Yukhin <kyukhin@tarantool.org>

This will allow us to call it from inside fiber_stack_create
as we need to clean up resources on error.

Part-of #4541

Signed-off-by: Kirill Yukhin <kyukhin@tarantool.org>

Count lines in the json parsing structure. It is needed to print
the number of line and column where a mistake was made.

Closes #3316

(cherry picked from commit 9f9bd3eb2d064129ff6b1a764140ebef242d7ff7)

Code to run main script (passed via command line args, or
interactive console) has a footer where it notifies systemd,
logs a happened error, and panics.

Before the patch that code was unreachable in case of any
exception in a main script, because panic happened earlier. Now a
happened exception is correctly carried to the footer with proper
error processing.

A first and obvious solution was replace all panics with diag_set
and use fiber_join on the script runner fiber. But appeared, that
the fiber running a main script can't be joined. This is because
normally it exits via os.exit() which never returns and therefore
its caller never dies = can't be joined.

The patch solves this problem by passing main fiber diag to the
script runner by pointer, eliminating fiber_join necessity.

Closes #4382

xlog/panic_on_broken_lsn includes waiting for replica and then
box.info.vclock usage. Sometimes box.info.vclock was giving wrong
values. Now it is stable due to improved replica expectation
criterion.

Closes #4508

Closes #4434
Follow-up #4366

@TarantoolBot document
Title: json/msgpack.cfg.encode_deep_as_nil option

Tarantool has several so called serializers to convert data
between Lua and another format: YAML, JSON, msgpack.

YAML is a crazy serializer without depth restrictions. But for
JSON, msgpack, and msgpackffi a user could set encode_max_depth
option. That option led to crop of a table when it had too many
nested levels. Sometimes such behaviour is undesirable.

Now an error is raised instead of data corruption:

    t = nil
    for i = 1, 100 do t = {t} end
    msgpack.encode(t) -- Here an exception is thrown.

To disable it and return the old behaviour back here is a new
option:

    <serializer>.cfg({encode_deep_as_nil = true})

Option encode_deep_as_nil works for JSON, msgpack, and msgpackffi
modules, and is false by default. It means, that now if some
existing users have cropping, even intentional, they will get the
exception.

Tuple is a C library exposed to Lua. In Lua to translate Lua
objects into tuples and back luaL_serializer structure is used.

In Tarantool we have several global serializers, one of which is
for msgpack. Tuples store data in msgpack, and in theory should
have used that global msgpack serializer. But in fact the tuple
module had its own private serializer because of tuples encoding
specifics such as never encode sparse arrays as maps.

This patch makes tuple Lua module use global msgpack serializer
always. But how does tuple handle sparse arrays now? In fact,
the tuple module still has its own serializer, but it is updated
each time when the msgpack serializer is changed.

Part of #4434

Msgpack Lua module is not a simple set of functions. It is a
global serializer object used by plenty of other Lua and C
modules. Msgpack as a serializer can be configured, and in theory
its configuration updates should affect all other modules. For
example, a user could change encode_max_depth:

    require('msgpack').cfg({encode_max_depth = <new_value>})

And that would make tuple:update() accept tables with <new_value>
depth without a crop.

But in fact msgpack configuration didn't affect some places, such
as this one. And all the others who use msgpackffi.

This patch fixes it, for encode_max_depth option. Other options
are still ignored.

Part of #4434

There are some objects called serializers - msgpack, cjson, yaml,
maybe more. They are global objects affecting both Lua and C
modules.

A serializer have settings which can be updated. But before the
patch an update changed only C structure of the serializer. It
made impossible to use settings of the serializers from Lua.

Now any update of any serializer is reflected both in its C and
Lua structures.

Part of #4434

LTO build failed because of warnings on the uninitialized
variables treated as errors:

/mnt/src/box/tuple_update.c: In function ‘do_op_splice’:
/mnt/src/box/tuple_update.c:764:15: error: ‘str_len’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
   arg->offset = str_len;
               ^
/mnt/src/box/tuple_update.c:755:10: note: ‘str_len’ was declared here
  int32_t str_len;
          ^
/mnt/src/box/tuple_update.c: In function ‘do_op_bit’:
/mnt/src/box/tuple_update.c:725:12: error: ‘val’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
   arg->val &= val;
            ^
/mnt/src/box/tuple_update.c:720:11: note: ‘val’ was declared here
  uint64_t val;
           ^
/mnt/src/box/tuple_update.c: In function ‘update_read_ops’:
/mnt/src/box/tuple_update.c:1068:4: error: ‘field_no’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
    diag_set(ClientError, ER_NO_SUCH_FIELD_NO, field_no);
    ^
/mnt/src/box/tuple_update.c:1057:10: note: ‘field_no’ was declared here
  int32_t field_no;
          ^
lto1: all warnings being treated as errors

Fixed by setting variables to 0 value on initializations.

Closed #4512

The submodule was on 7.65.3 version. This update closes two security
problems:

https://curl.haxx.se/docs/CVE-2019-5482.html
https://curl.haxx.se/docs/CVE-2019-5481.html

See also curl-7.66.0 release notes:

https://curl.haxx.se/changes.html#7_66_0

Fixes #4502.

Found that the curl failed to build on FreeBSD with errors:
gmake[2]: Entering directory '/home/vagrant/tarantool/third_party/curl/src'
  CCLD     curl
/usr/local/bin/ld: ../lib/.libs/libcurl.so: undefined reference to `SSLv23_client_method'
/usr/local/bin/ld: ../lib/.libs/libcurl.so: undefined reference to `CONF_modules_free'
/usr/local/bin/ld: ../lib/.libs/libcurl.so: undefined reference to `ERR_free_strings'
/usr/local/bin/ld: ../lib/.libs/libcurl.so: undefined reference to `sk_value'
/usr/local/bin/ld: ../lib/.libs/libcurl.so: undefined reference to `ENGINE_cleanup'
/usr/local/bin/ld: ../lib/.libs/libcurl.so: undefined reference to `SSL_library_init'
/usr/local/bin/ld: ../lib/.libs/libcurl.so: undefined reference to `EVP_MD_CTX_destroy'
/usr/local/bin/ld: ../lib/.libs/libcurl.so: undefined reference to `sk_pop_free'
/usr/local/bin/ld: ../lib/.libs/libcurl.so: undefined reference to `SSLeay'
/usr/local/bin/ld: ../lib/.libs/libcurl.so: undefined reference to `SSL_get_ex_new_index'
/usr/local/bin/ld: ../lib/.libs/libcurl.so: undefined reference to `OPENSSL_add_all_algorithms_noconf'
/usr/local/bin/ld: ../lib/.libs/libcurl.so: undefined reference to `SSL_COMP_free_compression_methods'
/usr/local/bin/ld: ../lib/.libs/libcurl.so: undefined reference to `EVP_MD_CTX_create'
/usr/local/bin/ld: ../lib/.libs/libcurl.so: undefined reference to `EVP_cleanup'
/usr/local/bin/ld: ../lib/.libs/libcurl.so: undefined reference to `sk_num'
/usr/local/bin/ld: ../lib/.libs/libcurl.so: undefined reference to `sk_pop'
/usr/local/bin/ld: ../lib/.libs/libcurl.so: undefined reference to `SSL_load_error_strings'
collect2: error: ld returned 1 exit status
gmake[2]: *** [Makefile:921: curl] Error 1

Found root cause of the issue at the `./configure <...>` output:
 | checking for OpenSSL headers version... 1.0.2 - 0x1000214fL
 | checking for OpenSSL library version... 1.1.1
 | configure: WARNING: OpenSSL headers and library versions do not match.
It is seen that the Tarantool bootstrap installed pkg 'openssl'
of the version '1.0.2', while the currently default FreeBSD 'openssl'
version was '1.1.1'.

Anyway we don't need any special openssl version installed against
default one, so the fix is just to remove the openssl package from
bootstrap installation.
Also found that some installing packages are not needed too,
removed it from FreeBSD bootstrap.

Additionally added libiconv library into bootstrap which is needed
as workaround to avoid of the issue described in:
	https://github.com/tarantool/tarantool/issues/3791

Closed #4490

It was forgotten to set MEM_Real flag for VDBE memory containing result
of varbinary to number cast operation. This patch fixes that and sets
corresponding flag if the cast takes place.

Closes #4356

Print the data type instead of the data itself in diag_set() in the case
of binary data. The reason of this patch is that LibYAML converts whole
error message to base64 in the case of non-printable symbols.

Part of #4356

Function do_op_insert() was depending on struct tuple_update
because of 'region' field. In next patches struct tuple_update
will be on much more high level of API than do_op_insert and such
a dependency would be wrong. Drop it.

Functions update_read_ops(), update_finish(),
tuple_upsert_squash() were depending on update_alloc() function
intended to be used by rope only. Rope will migrate down to lower
API in next patches, so this dependency should not exist.

Part of #1261

Splice was the last operation which used index_base in its do_op()
function. Now none of the operations use it. This allows to drop
dependency of operation implementations on index_base and on the
whole tuple_update, in next patches.

Part of #1261

- Unify error code names, they all should start with ER_UPDATE_*;

- Use string field identifier in error messages, because soon they
  will support both field numbers and JSON paths;

- Report all field numbers 1-based. This simplifies error analysis
  because no need to think from where an update has come - Lua or
  C/iproto. Also it allows to drop index_base argument from many
  functions;

- Introduce helper functions to set commonly appearing errors.
  Currently it is not a problem, but next patches will add several
  new files in all of which the same errors can happen;

- Deletion checks that number of fields to delete is > 0 right
  after reading the argument, not during deletion appliance. It
  allows to make such check in only one place even when more
  delete implementations will appear;

- make_arith_operation now takes update_op as an argument
  explicitly, not teared into separate arguments. It allows to use
  error helpers. Also dead code is dropped from this function with
  incorrect usage of some of errors;

Part of #1261

They are needed in JSON path update for a case, when a final part
of the path may not exist, and go_to_path returns an error.

That case is '=' and '!' on not existing fields. For example,
in {'=', '[1][2][3]', 20} field [3] can be not existing.

For these cases JSON update will have its own implementation of
go_to_path allowing optional last field.

Needed for #1261

Since ARRAY and MAP cannot be converted to SCALAR type, this
operation should throw an error. But when the error is raised in
SQL, it is displayed in unreadable form. The reason for this is
that the given array or map is not correctly converted to a
string. This patch fixes the problem by converting ARRAY or MAP to
their string representation.
For example:

box.execute('CREATE TABLE t1(i INT PRIMARY KEY, a SCALAR);')
format = {}
format[1] = {type = 'integer', name = 'I'}
format[2] = {type = 'array', name = 'A'}
s = box.schema.space.create('T2', {format=format})
i = s:create_index('ii')
s:insert({1, {1,2,3}})
box.execute('INSERT INTO t1 SELECT * FROM t2;')

Should return:
- error: 'Type mismatch: can not convert [1, 2, 3] to scalar'

Follow-up #4189

Function mem_apply_type() implements implicit type conversion. As
a rule, tuple to be inserted to the space is exposed to this
conversion which is invoked during execution of OP_MakeRecord
opcode (which in turn forms tuple). This function was not adjusted
to operate on ARRAY, MAP and ANY field types since they are poorly
supported in current SQL implementation. Hence, when tuple to be
inserted in space having mentioned field types reaches this
function, it results in error. Note that we can't set ARRAY or MAP
types in SQL, but such situation may appear during UPDATE
operation on space created via Lua interface. This problem is
solved by extending implicit type conversions with obvious casts:
array field can be casted to array, map to map and any to any.

Closes #4189

Method fio.mktree is used to create given path unconditionally -
without checking if it was a directory or something else. This
led to inappropriate error messages or even inconsistent behavior.
Now check the type of a given path.

Closes #4439

Allow views to use CTEs, which can be in any (nested) select after
<AS>. Before this patch, during view creation all referenced
spaces were fetched by name from SELECT and their reference
counters were incremented to avoid dangling references. It
occurred in update_view_references(). Obviously, CTE tables
weren't held in space cache, ergo error "space doesn’t exist" was
raised. Add check if a space from FROM is CTE. If it is, don't
increment its reference counter and don't raise the error.

Closes #4149

If the rock was packed with luarocks then the time in it is set to zeros
and this is not the case with the unix format can be removed when date given
is added to zipwriter_open_new_file_in_zip (luarocks/luarocks#1056)

Closes #4481

It was forgotten to swap old and new mask (holding fields involved into
foreign key relation) during space alteration (lists of object
representing FK metadata are swapped successfully). Since mask is vital
and depending on its value different byte-codes implementing SQL query
can be produced, this mistake resulted in assertion fault in debug build
and wrong constraint check in release build. Let's fix this bug and swap
masks as well as foreign key lists.

Closes #4495

ENGINE became reserved keyword in 1013a744. There's no any actual
reason why ENGINE should be reserved keyword. What is more, we are going
to use this word as a name of some fields for tables forming
informational schema. Hence, until it is too late (it is not documented
yet), let's remove ENGINE from the list of reserved keywords and allow
identifiers be that word.

Historically, we join a new replica off the last checkpoint. As a
result, we must always keep the last memtx snapshot and all vinyl data
files corresponding to it. Actually, there's no need to use the last
checkpoint for joining a replica. Instead we can use the current read
view as both memtx and vinyl support it. This should speed up the
process of joining a new replica, because we don't need to replay all
xlogs written after the last checkpoint, only those that are accumulated
while we are relaying the current read view. This should also allow us
to avoid creating a snapshot file on bootstrap, because the only reason
why we need it is allowing joining replicas. Besides, this is a step
towards decoupling the vinyl metadata log from checkpointing in
particular and from xlogs in general.

Closes #1271

This commit does not change a user visible behaviour. It refactors pwd
module to explicitly divide code that fetches data from passwd / group
databases from code that performs deserialization of the data to Lua
tables.

The idea of splitting of those actions appears when it was observed that
a call of getpw() / getgr() leads to problems on some systems when it is
invoked during passwd / group database traveral.

Now it is more obvious that we don't call getpw() during passwd
traversal and getgr() during group traveral.

Follows up #4428 and #4447.

Aside of that fixed other badges URLs to point to master branch.

(cherry picked from commit 7965aaaa88e6eaf266d2e51776598ac05a4a6cde)

LTO build fails on warning message:
In file included from /tarantool/src/lib/core/diag.h:33:0,
                 from /tarantool/src/box/engine.h:36,
                 from /tarantool/src/box/memtx_engine.h:40,
                 from /tarantool/src/box/memtx_engine.c:31:
/tarantool/src/box/memtx_engine.c: In function
'metmx_tuple_chunk_delete':
/tarantool/src/trivia/util.h:201:49: error: initialization from
incompatible pointer type [-Werror]
  const typeof( ((type *)0)->member  ) *__mptr = (ptr); \
                                                 ^
/tarantool/src/box/memtx_engine.c:1115:3: note: in expansion of macro
'container_of'
   container_of((typeof(tuple_chunk->data) *)data,
   ^
/tarantool/src/trivia/util.h:201:49: error: (near initialization for
'tuple_chunk') [-Werror]
  const typeof( ((type *)0)->member  ) *__mptr = (ptr); \
                                                 ^
/tarantool/src/box/memtx_engine.c:1115:3: note: in expansion of macro
'container_of'
   container_of((typeof(tuple_chunk->data) *)data,

Closes #4438

This patch provides a test suite that allows us to make sure that
the SQL BOOLEAN type works as intended.

Part of #4228

In a configuration with several read-only and read-write instances, if
replication_connect_quorum is not greater than the amount of read-only
instances and replication_connect_timeout happens to be small enough
for some read-only instances to form a quorum and exceed the timeout
before any of the read-write instaces start, all these read-only
instances will choose themselves a read-only bootstrap leader.
This 'leader' will successfully bootstrap itself, but will fail to
register any of the other instances in _cluster table, since it isn't
writeable. As a result, some of the read-only instances will just die
unable to bootstrap from a read-only bootstrap leader, and when the
read-write instances are finally up, they'll see a single read-only
instance which managed to bootstrap itself and now gets a
REPLICASET_UUID_MISMATCH error, since no read-write instance will
choose it as bootstrap leader, and will rather bootstrap from one of
its read-write mates.

The described situation is clearly not what user has hoped for, so
throw an error, when a read-only instance tries to initiate the
bootstrap. The error will give the user a cue that he should increase
replication_connect_timeout.

Closes #4321

@TarantoolBot document
Title: replication: forbid to bootstrap read-only masters.

It is no longer possible to bootstrap a read-only instance in an emply
data directory as a master. You will see the following error trying to
do so:
```
ER_BOOTSTRAP_READONLY: Trying to bootstrap a local read-only instance as master
```
Now if you have a fresh instance, which has
`read_only=true` in an initial `box.cfg` call, you need to set up
replication from an instance which is either read-write, or has your
local instance's uuid in its `_cluster` table.

In case you have multiple read-only and read-write instances with
replication set up, and you still see the aforementioned error message,
this means that none of your read-write instances managed to start
listening on their port before read_only instances have exceeded the
`replication_connect_timeout`. In this case you should raise
`replication_connect_timeout` to a greater value.