It is now suitable not only for handling JSON paths, but also for
building complex JSON structures.

Follow-up b56103f5 ("json: some renames").

Long time ago, when the checkpoint daemon was added to Tarantool, it was
responsible not only for making periodic checkpoints, but also for
maintaining the configured number of checkpoints and removing old snap
and xlog files, so it was much easier to implement it in Lua than in C.
However, over time, all its responsibilities have been reimplemented in
C and moved to the server code so that now it just calls box.snapshot()
periodically. Let's rewrite this simple procedure in C as well - this
will allow us to easily add more complex logic there, e.g. triggering
checkpoint when WAL files exceed a configured threshold.

Note, this patch removes a few cases from xlog/checkpoint_daemon test
that tested the internal state of the checkpoint daemon, which isn't
available in Lua anymore. This is OK as those cases are covered by
unit/checkpoint_schedule test.

This is a very simple module that incorporates the logic for calculating
the time of the next scheduled checkpoint given the configured interval
between checkpoints. It doesn't have any dependencies, which allows to
cover it with a unit test. It will be used by the checkpoint daemon once
we rewrite it in C. Rationale: in future we might want to introduce more
complex rules for scheduling checkpoints (cron-like may be) and it will
be really nice to have this logic neatly separated and tested.

Currently, the checkpoint iterator is in fact a wrapper around
memtx_engine::snap_dir while the garbage collector knows nothing about
checkpoints. This feels like encapsulation violation. Let's keep track
of all available checkpoints right in the garbage collector instead
and export gc_ API to iterate over checkpoints.

Turned out that throttling isn't going to be as simple as maintaining
the write rate below the estimated dump bandwidth, because we also need
to take into account whether compaction keeps up with dumps. Tracking
compaction progress isn't a trivial task and mixing it in a module
responsible for resource limiting, which vy_quota is, doesn't seem to be
a good idea. Let's factor out the related code into a separate module
and call it vy_regulator. Currently, the new module only keeps track of
the write rate and the dump bandwidth and sets the memory watermark
accordingly, but soon we will extend it to configure throttling as well.

Since write rate and dump bandwidth are now a part of the regulator
subsystem, this patch renames 'quota' entry of box.stat.vinyl() to
'regulator'. It also removes 'quota.usage' and 'quota.limit' altogether,
because memory usage is reported under 'memory.level0' while the limit
can be read from box.cfg.vinyl_memory, and renames 'use_rate' to
'write_rate', because the latter seems to be a more appropriate name.

Needed for #1862

None of vy_quota methods is called from a hot path - even the most
frequently called ones, vy_quota_try_use and vy_quota_commit_use, are
only invoked once per a transactions. So there's no need to clog the
header with the methods implementation.

This patch introduces new system space to persist foreign keys
constraints. Format of the space:

_fk_constraint (space id = 358)

[<constraint name> STR, <parent id> UINT, <child id> UINT,
 <is deferred> BOOL, <match> STR, <on delete action> STR,
 <on update action> STR, <child cols> ARRAY<UINT>,
 <parent cols> ARRAY<UINT>]

FK constraint is local to space, so every pair <FK name, child id>
is unique (and it is PK in _fk_constraint space).

After insertion into this space, new instance describing FK constraint
is created. FK constraints are held in data-dictionary as two lists
(for child and parent constraints) in struct space.

There is a list of FK restrictions:
 - At the time of FK creation parent and child spaces must exist;
 - VIEWs can't be involved into FK processing;
 - Child space must be empty;
 - Types of referencing and referenced fields must be comparable;
 - Collations of referencing and referenced fields must match;
 - Referenced fields must compose unique index;
 - Referenced fields can not contain duplicates.

Until space (child) features FK constraints it isn't allowed to be
dropped. Implicitly referenced index also can't be dropped
(and that is why parent space can't be dropped). But :drop() method
of child space firstly deletes all FK constraint (the same as SQL
triggers, indexes etc) and then removes entry from _space.

Part of #3271

Review fixes

Blackhole is a very simple engine that allows to create spaces that may
written to, but not read from. It only supports INSERT/REPLACE requests.
It doesn't support any indexes hence SELECT is impossible. It does check
space format though and supports on_replace and before_replace triggers.

The whole purpose of this new engine is writing arbitrary rows to WAL
without storing them anywhere. In particular, we need this engine to
write deferred DELETEs generated for vinyl spaces to WAL.

Needed for #2129

They are fairly small and closely related so let's merge them and call
the result sysview.[hc].

All box exceptions belong to box/error.h. Let's move XlogGapError there
as well. This will facilitate conversion of recovery.cc to C when we
finally get to it. While we are at it, let's also move BuildXlogError
function declaration from diag.h to box/error.h, closer to its
definition.

The two files are too closely related: memtx_arena is defined and
used in memtx_engine.c, but initialized in memtx_tuple.cc. Since
memtx_tuple.cc is small, let's fold it into memtx_engine.c.

In the issue #3290 the important problem appeared - Tarantool can
not create completely internal collations with no ID, name,
owner. Just for internal usage.

Original struct coll can not be used for this since
* it has fields that are not needed in internals;
* collation name is public thing, and the collation cache uses
  it, so it would be necessary to forbid to a user usage of some
  system names;
* when multiple collations has the same comparator and only their
  names/owners/IDs are different, the separate UCollator objects
  are created, but it would be good to be able to reference a
  single one.

This patch renames coll to coll_id, coll_def to call_id_def and
introduces coll - pure collation object with no any user defined
things.

Needed for #3290.

No other changes.

No semantical changees.

No semantical changes.

So that it can be reused by vy_read_iterator. No functional changes,
this patch just moves pieces of code.

Until this moment in Lua a tuple data could be accessed in 3 ways:
- get field by field number, decode into Lua,
- get field by field name, decode into Lua
- decode entire tuple into Lua.

It was impossible to decode into Lua only a part of the field to
avoid Lua garbage creating. For example, consider the tuple:
`{1, 2, {key1 = value1,
         key2 = {key21 = {key211 = {key2111, key2112}}}}}`
with field names `field1`, `field2`, `field3`.

To get `key2112` it is necessary to decode into Lua the entire
3-th field, including `key1`, `value1`, `key2`, `key21`,
`key211`, `key2111` using the syntax
`key2112 = tuple.field3.key2.key21.key211[2]`.

Now the one can use the following syntax:
`key2112 = tuple["field3.key2.key21.key211[2]"]`. The difference
is in placing all the path into quotes and brackets `["..."]`.
The Tarantool goes through this path and MessagePack tuple body,
gets the needed tuple part and decodes into Lua only it.

The path must be valid JSON path with one exception - in
Tarantool a path can start with `.`. For example:
`tuple[".field3..."]` - it is done to lay emphasis that the JSON
path here is just a suffix for the tuple.

At the same time tuple field names still work: `tuple["field1"]`,
`tuple.field2`.

If the field name looks like JSON path, for example:
`my.field.name`, then it works too. The path at first is checked
to be a real JSON path, and if nothing is found, then the entire
path is considered as a field name. To combine such names and
path elements the one can use `["..."]`. For example:
`tuple["['my.field.name'].key1.key2.['another.key.in.map']"]`.

Closes #1285

Rewrote TCL scripts as posix shell ones, enabled SQL_MAINTAINER_MODE
commands unconditionally.

Rewrote cmake targets to generate those files in a build directory
instead of the source directory.

Fixes #3183.

Currently, we store and use bloom only for full-key lookups. However,
there are use cases when we can also benefit from maintaining bloom
filters for partial keys as well - see #3177 for example. So this patch
replaces the current full-key bloom filter with a multipart one, which
is basically a set of bloom filters, one per each partial key. Old bloom
filters stored on disk will be recovered as is so users will see the
benefit of this patch only after major compaction takes place.

When a key or tuple is checked against a multipart bloom filter, we
check all its partial keys to reduce the false positive result.
Nevertheless there's no size optimization as per now. E.g. even if the
cardinality of a partial key is the same as of the full key, we will
still store two full-sized bloom filters although we could probably save
some space in this case by assuming that checking against the bloom
corresponding to a partial key would reduce the false positive rate of
full key lookups. This is addressed later in the series.

Before this patch we used a bloom spectrum object to construct a bloom
filter. A bloom spectrum is basically a set of bloom filters ranging in
size. The point of using a spectrum is that we don't know what the run
size will be while we are writing it so we create 10 bloom filters and
choose the best of them after we are done. With the default bloom fpr of
0.05 it is 10 byte overhead per record, which seems to be OK. However,
if we try to optimize other parameters as well, e.g. the number of hash
functions, the cost of a spectrum will become prohibitive. Funny thing
is a tuple hash is only 4 bytes long, which means if we stored all
hashes in an array and built a bloom filter after we'd written a run, we
would reduce the memory footprint by more than half! And that would only
slightly increase the run write time as scanning a memory map of hashes
and constructing a bloom filter is cheap in comparison to mering runs.
Putting it all together, we stop using bloom spectrum in this patch,
instead we stash all hashes in a new bloom builder object and use them
to build a perfect bloom filer after the run has been written and we
know the cardinality of each partial key.

Closes #3177

Vinyl assigns a unique id to each index so that it can be identified in
vylog, see vy_index->id, but outside Vinyl index id means something
completely different - it's the ordinal number of the index in a space.
This creates a lot of confusion. To resolve this, let's rename vy_index
to vy_lsm and refer to Vinyl indexes as LSM trees in comments so that
Vinyl's index_id turns into lsm_id.

Also, rename vy_log_record's index_def_id and space_def_id back to
conventional index_id and space_id as they doesn't conflict with Vinyl
index id anymore (which is now lsm_id).

Thanks to @Gerold103 for suggesting the new name.

* feedback daemon sends information about instance to the
  specified host.
* Add new options to box.cfg:
    - feedback_enabled - switch on/off daemon, default=true.
    - feedback_host - host to which feedbacks are sent,
      default="https://feedback.tarantool.io".
    - feedback_interval - time interval in seconds of feedbacks
    sending, default=3600.
* Add possibility to generate feedback file in json format with
function box.feedback.save

Closes #2762

Needed for #2048

This patch adds two new Lua function, box.ctl.wait_ro() and
box.ctl.wait_rw(), that block the current fiber until the
server switches to read-only or read-write mode, respectively.
Both functions take the timeout as an optional argument.

Needed for #2537

This patch moves helpers used to fix requests after certain DML
operations to a separate source file. Currently, there are only
two of them, but there are going to be more so it seems to be a
good idea to isolate them. No functional changes.

Suggested by @kostja

Struct tuple_dictionary stores hash of field names, defined in
a space format. The structure is refable and are going to be used
to share new field names with old space formats.

Part of #3011

Before this patch, we would only allow alphabetical characters
plus underscore in identifier names. And we did not treat
all identifiers the same way: column names were not checked
at all.

SQL ANSI ISO allow delimited identifiers cantain any character
from source language character set.

After this patch, checks for allowed characters in identifier
names follow the same ruls for all identifiers: column names,
function names, user names, space names, index names.

In other words, this patch makes tarantool itentifier rules closer
to ANSI ones.

Closes #2914

We can do it for free now as all functions used by call.cc have already
been converted to C and there's nothing in call.cc that really needs any
C++ features.

The point iterator is not actually an iterator: it doesn't have an
internal state and it acts as a function. Wrapping it into the iterator
protocol only complicates its usage. Let's turn it into a function.

The engine infrastructure was initially implemented in C++ so we
needed the wrappers to provide C++ API to Vinyl. Now everything is
in C so we don't need them any more. Let's fold them in vinyl.c.

Note, this patch does not touch vinyl_engine, vinyl_index, and
vinyl_iterator structures, they are still there, it just gets rid
of the intermediate layer of wrapper functions, which is not needed
any more.

It's a big independent entity, let's isolate its code in
a separate file.

While we are at it, add missing comments to vy_scheduler
struct members.

The primary reason for these methods to be implemented differently
for memtx and vinyl was the 'position' optimization exploited by
the memtx engine: since selects from memtx do not yield, we could
use a preallocated iterator there.

Now, as the 'position' optimization became redundant and was
removed due to the switch to memory pools for iterator allocations,
the only idiosyncrasy left in the memtx implementation is the count()
optimization: count() falls back on size() for ITER_ALL. Since this
optimization consists of just a few lines of code, we don't really
need memtx_index_count() co-used by all memtx index implementations:
we can implement it in each memtx index separately.

That being said, let us:
 - implement generic versions of min(), max(), and count();
 - make vinyl, memtx, and sysview engines use generic versions of
   the above-mentioned methods if appropriate;
 - Remove memtx_index.[hc]

As a side-effect, this patch enables min(), max(), and count() in
the sysview engine, but that is not bad considering that this engine
implements general-purpose iterator for its indexes.

Needed for #2776

Needed for #2776

Needed for #2776

Needed for #2776

Add a new space that contains collation definitions for future
index collation and more.

Add a collation cache for fast retrieving a collation object by
its name.

Needed for #2649

Introduce coll object that allows to compare strings.
Add coll_def structure and functions for its setup.
Add unit test.

Needed for #2649

Move opt_def functions from alter.cc to opt_def.c.
Make no logical changes.

Closes #2639

Since commit c0a30d17 ("sequence: speed up allocation of sequence
data tuple during checkpoint") sequence_data_iterator_next() does not
throw an exception, so the whole thing can be moved to sequence.c.