Before this patch, if integer overflow occurred during casting to
integer, it was simply ignored. As a result, wrong results might take
place. Lets check possible overflows before CAST conversion, and if it
happens, raise an appropriate error.

Part of #3735

Before this patch, if integer was overflowed during math operations
(OP_Add, OP_Subtract, OP_Multiply, OP_Divide), it would be implicitly
converted and stored as floating point number. This is obviously wrong
way to handle integer overflow errors. Instead, let's raise
corresponding error.

Part of #3735

Before this patch, resulting type of concatenation operator always was
TEXT (i.e. type of memory cell containing result - MEM_Str). Lets fix it
and return type depending on type of concatenation arguments: if both
arguments are TEXT, then resulting type is TEXT; BLOB otherwise. Note
that other options of combining types of arguments are illegal.

Closes #3544

Original SQLite operator of concatenation accepts all types of
arguments. If type of parameter is not TEXT, it is implicitly converted
to TEXT (except for NULLs). That contradicts ANSI (it is regulated by
[1]), so lets allow only TEXT and BLOB as argument type for
concatenation.  Moreover, they both must be of the same type at the same
time (i.e. both TEXT or BLOB).

[1] SQL ANSI 2013, 9.5 Result of data type combination

Part of #3544

yaml.encode() now wraps a string literal whose content is equal to a
null or a boolean value representation in YAML into single quotes. Those
literals are: 'false', 'no', 'true', 'yes', '~', 'null'.

Reverted the a2d7643c commit to use single quotes instead of multiline
encoding for 'true' and 'false' string literals.

Follows up #3476
Closes #3662
Closes #3583

Use lua_is*() functions instead of explicit lua_gettop() checks in
yaml.encode() and yaml.decode() functions.

Behaviour changes:

* yaml.decode(object, nil) ignores nil (it is consistent with encode
  behaviour).
* yaml.encode() gives an usage error instead of "unsupported Lua type
  'thread'".
* yaml.encode('', {}, {}) ignores 3rd argument (it is consistent with
  decode behaviour).

Iproto connections keep old readahead values for input buffers even
after box.cfg.readahead reconfiguration. This means that for the changes
to take place for the old clients, they have to reconnect. Otherwise
tarantool log will be spammed with 'readahead limit is reached' errors.

To fix this, start updating input buffer size for iproto connections
if needed every time the buffer is empty.

Closes: #3958

Checks for fiber_name legth worked with FIBER_NAME_MAX,
while one byte in fact was occupied by terminating zero.

Closes #4011

vy_run_iterator_seek left the return statement uninitialized under
certain conditions. The branch in the code wasn't covered by any test,
because it could only be triggered by an EQ SELECT following range
coalescing, so we only saw it when a customer reported a crash.
Fix it and improve vinyl/select_consistency test to cover this branch.

Now heap API works with struct heap_node only, which forces a
user to constantly call container_of. Such a code looks really
awful. This commit makes heap taking and returning user defined
structures, and removes container_of clue.

It is worth noting, that the similar API rb-tree and b-tree
have. Even rlist has its rlist_*_entry() wrappers, and mhash
provides macroses to define your own value type.

According to ANSI, result of concatenation operation should derive
collation sequence from its operands. Now it is not true: result is
always comes with no ("none") collation.

In a nutshell[1], rules are quite simple:

a) If some data type has an explicit collation EC1, then every data type
that has an explicit collation shall have declared type collation that
is EC1.  The collation derivation is explicit and the collation is EC1.

b) If every data type has an implicit collation, then:

 - If every data type has the same declared type collation IC1, then
   the collation derivation is implicit and the collation is IC1.

 - Otherwise, the collation derivation is none.

c) Otherwise, the collation derivation is none.

[1] Read complete statement at chapter 9.5 Result of data type
combinations, ANSI 2013, Part 2: Foundations.

Closes #3937

Rename struct fkey and struct fkey_def to fk_constraint and
fk_constraint_def respectively.
Consistently use "fk" for foreign key variables, and fk_def for
foreign key definition variables.

Remove dependency of fkey.h on space.h.
Enfore subject-verb-object naming in a few fk-related methods.

Follow up on the patch adding transaction boundaries to xrow stream.
Use tsn as an abbreviation for transaction identifier (transaction
sequence number). It is an important enough concept to use a short and
a convenient tag name for.
Deploy the name across the code - in names and comments.
Clarify comments.
Still use box_txn_id() as API method since box_tsn() and box_txn() would
be too easy to mistype.

SQLite grammar implementing foreign keys parsing is quite compact, but
on the other hand it allows to define ON DELETE and ON UPDATE actions
multiple times. For instance:

... REFERENCES t ON DELETE UPDATE ON DELETE RESTRICT;

It makes no sense and contradicts original ANSI syntax. So, lets rework
it a bit. Firstly, MATCH clause must come first, so we place it in
independent rule. Then we remove ON INSERT clause, since there is no
such opportunity at all. Finally, we have only 4 options to expose
refargs (i.e. grammar rule to parse FK actions): ON UPDATE, ON DELETE,
ON UPDATE ON DELETE, ON DELETE ON UPDATE. That's it.

Closes #3475

Append txn_id and is_commit to xrow_header structure, txn_id identifies
transaction id on replica where transaction was started. As transaction id
a lsn of the first row in the transaction is used. is_commit is set to true
for the last row in a transaction.

As encoding/deconding rule assumed:
 * txn_id encoded using transaction sequence number iproto field
   as IPROTO_TSN = lsn - txn_id,
 * is_commit packed into IPROTO_FLAGS field with a bit mask,
 * txn_id and is_commit are encoded only for multi-row transactions.
   So if we do not have txn_id after row decoding then this means that it
   is a single row transaction.

These rules provide compatibility with previous xlog format as well
as good compaction level.

Needed for #2798

Add optional 'chars' parameter to string.strip, string.lstrip
and string.rstrip for specifying the unwanted characters.
Behavior modeled after the equivalent Python built-ins.

Closes: #2977

@TarantoolBot document
Title: string.strip(inp, chars)
Update the documentation for string.strip,
string.lstrip and string.rstrip to reflect
the addition of the optional param.

This patch adds space name to descriptions of some of new errors.
Also it fixes name and description of a few errors.

Part of #3965

When a connection is closed, it should not allow any
requests - async and not. But before this patch this error
from netbox.perform_async_request was ignored.

This patch reworks SQL errors of types "no such object" and
"object exists". After this patch, these error will be set as
Tarantool errors.

Part of #3965

The suppressErr field was used to indicate that most of the errors
during the parsing should be suppressed. There was only one
feature that used it. After deleting this feature, it became
obvious that this field had become unused and should be removed.

The feature in question is: allow to use names and aliases from
input in ORDER BY clause of UNION or INTERSECT. After deleting
of this feature, requests like the one below become invalid:

SELECT 1 AS a UNION ALL SELECT 2 AS b ORDER BY b;

Part of #3965

The pragmas "query_only" and "read_uncommitted" didn't affect anything
and were removed.
Fixed an error in pragma index_list which caused a segmantation fault.
pragma sql_default_engine accepts only strings.
Thus pragma sql_default_engine('memtx') is a well-formed command,
while pragma sql_default_engine(memtx) or
pragma sql_default_engine("memtx") are considered as an ill-formed and
raise an error.

Closes #3733

The box.tuple.new() used to call luamp_encode_tuple with
default LUA serializer config 'luaL_msgpack_default'. This
routine may consider an array to be excessively sparse when
  + encode_sparse_ratio > 0
  + max(table) > encode_sparse_safe
  + max(table) > count(table) * encode_sparse_ratio.
Sparse optimization save memory via representing excessively
sparse tuple as MP_MAP. But Tarantool tuple always must be
MP_ARRAY so it is not relevant for box.tuple.new semantics.
So it is disabled with encode_sparse_ratio = 0 in a new local
serializer config.

Closes #3882

The function parses string URI consisting of either IP and port,
or UNIX socket address, and stores the result into struct
sockaddr.

Set the reason "Unknown" when it is CURLE_OK and status is more than or equal
to 400.

Closes #3681

A bug existed because module_init was called during a call to box_cfg{}.
Modules were not initialized before calling box.cfg{}.

Closes #3770

Lets completely remove struct Table. Also the patch simplifies memory
management as in many cases struct space (which replaces struct Table)
is allocated on region and shouldn't be explicitly freed.  Some wrappers
fetching data from space (such as space_checks_expr_list) have been
removed since now we can get all properties right from space object
right from cache.

Closes #3235

They aren't needed there as we reset them anyway once the snapshot is
replayed on initial bootstrap. OTOH having them results in calling
replica_{set,clear}_id twice on initial bootstrap, which will look weird
when I patch them to log the ids. So let's remove them from the initial
snapshot. This makes the initial bootstrap impossible to recover from as
it is, but that shouldn't be an issue since one can always bootstrap a
new instance in a normal way. This also allows us to make cluster uuid
truly immutable (currently, one can update it with REPLACE).

Currently, vclock_to_string() allocates the formatted vclock string
using malloc() and hence the caller is responsible for freeing it, which
isn't very user-friendly. Let's use a static buffer as we do to format
other objects.

vy_run_iterator doesn't take a reference to the format it uses to decode
statements loaded from disk. As a result, the format may be deleted by
DDL, leading to a use-after-free bug. Fix this by taking a reference to
the format used by an iterator.

Closes #4000

Modify only needed part of a vclock structure instead of full structure
writing.

Follow-up: #2283

Applier writes NOP if it is not able to apply a masters row
because of conflict when the replication_skip_conflict option is set.
This prevents applier from reapplying already skipped rows after
restart.

Closes: #3977

Replace all usage of sqlite3_, sqlite, SQLite prefixes with simple sql_
All other occurrences of SQLite are substituted with SQL word.
SQL test suit is purified as well.

The reason why the limit is so is that default Apache / nginx maximum
header size is 8 KiB.

Added a check to raise an error when a header is bigger then the limit.

Fixes #3959.

Since all ranges constituting an LSM tree have the same configuration,
they tend to get compacted at approximately the same time. This entails
IO load spikes, which, in turn, lead to deviation of the LSM tree from
the target shape and hence increased read amplification. To prevent this
from happening, this patch implements compaction randomization: with 10%
probability we defer compaction at each LSM tree level, i.e. if the
number of runs at a level exceeds the configured run_count_per_level,
the level will be compacted with 90%-probability, but with 10%
probability it won't - compaction will be deferred until another run
is added to the level.

Our simulations show that such a simple algorithm performs fairly well:
it randomizes compaction pace among ranges, spreading IO load evenly in
time, while the write amplification is increased by not more than 5-10%,
which seems to be a reasonable price for elimination of IO load spikes.

Closes #3944

The key space of a vinyl index consists of multiple ranges that can be
compacted independently. This design was initially invented to enable
parallel compaction, so the range size is configured statically, by the
range_size index option, which equals 1 GB by default. However, it turns
out that ranges can also be useful for smoothing IO load: if we compact
approximately the same number of ranges after each dump, we will avoid
IO bursts, which is good, because IO bursts can distort the LSM tree
shape, resulting in increased read amplification.

To achieve that, we need to maintain at least as many ranges as the
number of dumps it takes to trigger major compaction of a range. With
the default range size, this condition will hold only if the index is
huge (tens to hundreds gigabytes). If the database isn't that big or
consists of many small indexes, the range count will never even approach
that number. So this patch makes the range size scale dynamically to
satisfy that condition.

The range size configuration options, both global and per index, aren't
removed though. The patch just changes box.cfg.vinyl_range_size default
value to nil, which enables automatic range sizing for all new indexes
created without passing range_size explicitly. All existing indexes will
still use the range size stored in index options (we don't want to alter
the behavior of an existing production setup). We are not planning to
drop range_size option altogether - it still can be useful for testing
and performance analysis.

The actual range size value is now reported in index.stat().

Needed for #3944

This patch adds dumps_per_compaction metric to per index statistics. It
shows the number of dumps it takes to trigger a major compaction of a
range in a given LSM tree. We need it to automatically choose the
optimal number of ranges that would smooth out the load generated by
range compaction.

To calculate this metric, we assign dump_count to each run. It shows how
many dumps it took to create the run. If a run was created by a memory
dump, it is set to 1. If a run was created by a minor compaction, it is
set to the sum of dump counts of compacted ranges. If a run was created
by a major compaction, it is set to the sum of dump counts of compacted
ranges minus dump count of the last level run. The dump_count is stored
in vylog.

This allows us to estimate the number of dumps that triggers compaction
in a range as dump_count of the last level run stored in the range.
Finally, we report dumps_per_compaction of an LSM tree as the average
dumps_per_compaction among all ranges constituting the tree.

Needed for #3944

Currently, vinyl won't shutdown until all reader and writer threads
gracefully complete all their pending requests, which may take a while,
especially for writer threads that may happen to be doing compaction at
the time. This is annoying - there's absolutely no reason to delay
termination in such a case. Let's forcefully cancel all threads, like we
do in case of relay threads.

This should fix sporadic vinyl/replica_quota test hang.

Closes #3949

When computing the number of runs that need to be compacted for a range
to conform to the target LSM tree shape, we use the newest run size for
the size of the first LSM tree level. This isn't quite correct for two
reasons.

First, the size of the newest run is unstable - it may vary in a
relatively wide range from dump to dump. This leads to frequent changes
in the target LSM tree shape and, as a result, unpredictable compaction
behavior. In particular this breaks compaction randomization, which is
supposed to smooth out IO load generated by compaction.

Second, this can increase space amplification. We trigger compaction at
the last level when there's more than one run, irrespective of the value
of run_count_per_level configuration option. We expect this to keep
space amplification below 2 provided run_count_per_level is not greater
than (run_size_ratio - 1). However, if the newest run happens to have
such a size that multiplying it by run_size_ratio several times gives us
a value only slightly less than the size of the oldest run, we can
accumulate up to run_count_per_level more runs that are approximately as
big as the last level run without triggering compaction, thus increasing
space amplification by up to run_count_per_level.

To fix these problems, let's use the oldest run size for computing the
size of the first LSM tree level - simply divide it by run_size_ratio
until it exceeds the size of the newest run.

Follow-up #3657

While a replica is bootstrapped from a remote master, vinyl engine
may need to perform compaction, which means that it may write to
the _vinyl_deferred_delete system space. Compaction proceeds fully
asynchronously, i.e. a write may occur after the join stage is
complete, but before the WAL is initialized, in which case the new
replica will crash. To make sure a race like that won't happen, let's
setup WAL before making the initial checkpoint. The WAL writer is now
initialized right before starting the WAL thread and so we don't need
to split WAL struct into the thread and the writer anymore.

Closes #3968