We are planning to link json_path_node objects in a tree and attach some
extra information to them so that they could be used to describe a json
document structure. Let's rename it to json_token as it sounds more
appropriate for the purpose.

Also, rename json_path_parser to json_lexer as it isn't a parser,
really, it's rather a tokenizer or lexer. Besides, the new name is
shorter.

Needed for #1012

The failing test case checks that modifications done to the space during
the final dump of a newly built index are recovered properly. It assumes
that a series of operations will complete in 0.1 seconds, but it may not
happen if the disk is slow (like on Travis CI). This results in spurious
failures. To fix this issue, let's replace ERRINJ_VY_RUN_WRITE_TIMEOUT
used by the test with ERRINJ_VY_RUN_WRITE_DELAY, which blocks index
creation until it is disabled instead of injecting a time delay as its
predecessor did.

Closes #3756

These are stress testing some of the parser/vdbe features, no point
in replaying them against vinyl. They could just as well run in
wal_mode="none"

Disable these tests in regular suite until they are sped up in scope
of gh-3845

Refactoring. Move lua error functions to a separate file.

A prerequisite for #677

Closes #3824

To make functions in execute.h more universal we should reduce
their dependence on IPROTO. This patch removes IPROTO functions
from execute.c.

Needed for #3505

New method dump_lua dumps saved in port tuples to Lua stack. It
will allow us to call this method without any other interaction
with port.

Needed for #3505

Refactored sql_request structure to store pointer to sql string
data and it's length instead of pointer to msgpack
representation.
This is required to use this structure in sql.c where the query
has a different semantics and can be obtained from stack as a C
string.

Needed for #3505.

tonumber64() doesn't understand strings with "ULL" like "123ULL". The
expected output for tonumber64("123ULL") is 123, since 123ULL is a
correct number notation in lua. However, our function returns null.
This happens because suffix isn't trimmed in tonumber64.

Trim ULL/LLU, LL suffixes, but only when no base is specified or
base is equal to either 2, 10 or 16.

Closes #3431

Part of #2436, #3232

- Box configuration parameter `memtx_memory` is increased, because the
  test `lua` after `tuple` failed with the error:
  `Failed to allocate 368569 bytes in slab allocator for memtx_tuple`
  despite `collectgarbage('collect')` calls after cases with huge/many
  tuples.
  The statistics before the allocation fail gives the following values:
  ```
  box.slab.info()
  ---
  - items_size: 72786472
    items_used_ratio: 4.43%
    quota_size: 107374592
    quota_used_ratio: 93.75%
    arena_used_ratio: 6.1%
    items_used: 3222376
    quota_used: 100663296
    arena_size: 100663296
    arena_used: 6105960
  ```
  The reason of the fail seems to be a slab memory fragmentation. It is
  not clear for now whether we should consider this as a tarantool
  issue.

- Test `snapshot_stress` counts snapshot files present in the
  working directory and can reach the default 'checkpoint_count' value
  `2` if a previous test write its snapshots before.

- Restarting the default server w/o cleaning a working directory
  can leave a snapshot that holds a state saved at the middle of a test,
  before dropping of the space 'tweedledum' (because WAL is disabled),
  that can cause the error `Space 'tweedledum' already exists` for a
  following test.

- Use unix sockets because of errors `Address already in use`.

Part of #2436

Too many autogenerated ids leads to SEGFAULT. This problem
appeared because region was cleaned twice: once in
sqlite3VdbeHalt() and once in sqlite3VdbeDelete() which was
executed during sqlite3_finalize(). Autogenerated ids that were
saved there, were fetched after sqlite3VdbeHalt() and before
sqlite3_finalize(). In this patch region cleaning in
sqlite3VdbeHalt() has been removed.

Follow up #2618
Follow up #3199

Before this patch MSGPACK received using SELECT statement through
net.box was unpacked. Fixed in this patch.

Previously if an error occured in box_index_len() called from
sql_analysis_load(), the return code (-1 on error) was cast to uint32_t
and used later as size of memory to be allocated. This lead to assertion
failures in slab_order() since allocation size was too big. This was
discovered during investigation of #3779.
Fix error handling and add some error logging.

Follow-up #3779

Again, this is something that was introduced by commit f2bccc18
("Use WAL vclock instead of TX vclock in most places") without any
justification.

TX has its own copy of the current vclock - there's absolutely no need
to inquire it from the WAL thread. Actually, we already use TX local
vclock in box_process_vote(). No reason to treat join/subscribe any
different. Moreover, it's even harmful - there may be a gap at the end
of a WAL file, in which case WAL vclock will be slightly ahead of TX
vclock so that should a replica try to subscribe it would never finish
syncing, see #3830.

Closes #3830

Because this is pointless and confusing. This "feature" was silently
introduced by commit f2bccc18 ("Use WAL vclock instead of TX vclock
in most places"). Let's revert this change. This will allow us to
clearly separate WAL checkpointing from WAL flushing, which will in turn
facilitate implementation of the checkpoint-on-WAL-threshold feature.

There are two problems here, however. First, not rotating the log breaks
expectations of replication/gc test: an xlog file doesn't get deleted in
time as a consequence. This happens, because we don't delete xlogs
relayed to a replica after join stage is complete - we only do it during
subscribe stage - and if we don't rotate WAL on subscribe the garbage
collector won't be invoked. This is actually a bug - we should advance
the WAL consumer associated with a replica once join stage is complete.
This patch fixes it, but it unveils another problem - this time in the
WAL garbage collection procedure.

Turns out, when passed a vclock, the WAL garbage collection procedure
removes all WAL files that were created before the vclock. Apparently,
this isn't quite correct - if a consumer is in the middle of a WAL file,
we must not delete the WAL file, but we do. This works as long as
consumers never track vlcocks inside WAL files - currently they are
advanced only when a WAL file is closed and naturally they are advanced
to the beginning of the next WAL file. However, if we want to advance
the consumer associated with a replica when join stage ends (this is
what the previous paragraph is about), it might occur that we will
advance it to the middle of a WAL file. If that happens the WAL garbage
collector might remove a file which is actually in use by a replica.
Fix this as well.

First, this is consistent with other engine callbacks, such as
checkpoint or backup.

Second, a vclock can be used as a search key in a vclock set,
which in turn can make code more straightforward, e.g. look how
this patch simplifies vy_log_prev_checkpoint().

In the updated version rb_proto/rb_gen use const qualifier for the key
argument, which allows to pass pointers to const objects to search
methods.

Removed INDEX_XINFO pragma and updated INDEX_INFO with its columns with
the following corrections: replaced 6th result column (whether an index
column is key/aux one) with type name of the index column.

Cleaned up pragma column names array (pragma.h::pragCName).

Fixes #3194.

Allow constraints to appear along with columns definitions. Disallow typing
a constraint name without specifying the constraint and after it.

Closes #3504

 - Before checking that old WAL files have been removed, wait for the
   garabe collection to remove them to avoid a spurious test failure.
   Currently, the test waits until old checkpoints are removed, but the
   garbage collector can now remove checkpoints, but keep WAL files.
   This is a follow-up for commit 9c5d851d ("replication: remove old
   snapshot files not needed by replicas").
 - Remove a few pointless box.info.gc().checkpoints checks.
 - Use test_run.wait_cond for waiting instead of while-do-sleep loops.

The test is very intricate and relies on timings too much so that should
we change anything in the SUBSCRIBE procedure, we could easily get a
spurious test failure. Let's rewrite it a little bit to make it more
robust. The details are in the comment in the code.

While we are at it, let's also fix comments and use test_run.wait_cond
helper for waiting for a condition to come true instead of repeat-sleep
loops.

When SQL DELETE statement comes in most primitive from without WHERE
clause and foreign key constraints, it is optimized and processed with
one VDBE instruction (instead of several OP_Delete). However, it was
forgotten to account affected tuples by row counter. Current patch fixes
this obvious defect.

Closes #3816

To fix the warning on bootstrap:

  xlog.c:1920 E> can't open tx: bootstrap: has some data after eof marker at 1906

It started to hang after the last merge.

Introduced a new tuple_format_use_key_part that makes format
initialization for specified key_part and configuration.
This decrease tuple_format_create routine complexity and would
be used to initialize structures in format for JSON path.

Need for #1012

The new tuple_field_go_to_path routine is used in function
tuple_field_raw_by_path to retrieve data by JSON path from field.
We need this routine exported in future to access data by JSON
path specified in key_part.

Need for #1012

Introduced a new key_def_parts_are_sequential routine that test,
does specified key_def have sequential fields. This would be
useful with introducing JSON path as there would be another
complex criteria as fields with JSONs can't be 'sequential' in
this meaning.

Need for #1012

Refactored key_def_find routine to use key_part as a second
argument. Introduced key_def_find_by_fieldno helper to use in
scenarios where no key_part exists.
New API is more convenient for complex key_part that will appear
with JSON paths introduction.

Need for #1012

After patch that introduced "none" collation (a953051e),
box.internal.bootstrap() started to fail due to inability to drop
mentioned collation. Lets turn off system triggers for _collation space
in order to process its complete purification during bootstrap.

box.cfg() updates only those options that have actually changed.
However, for replication it is not always true: box.cfg{replication = x}
and box.cfg{replication = {x}} are treated differently, and as
the result - replication is restarted. The patch fixes such behaviour.

Closes #3711

Currently, run_count_per_level index option is applied to each LSM tree
level. As a result, we may end up storing each key run_count_per_level
times at the last level alone, which would result in prohibitive space
amplification. To avoid that, let's ignore run_count_per_level for the
last level.

Note, we have to tweak quite a few vinyl tests, because they implicitly
relied on the fact that producing run_count_per_level dumps would never
trigger compaction.

Closes #3657

I. Fixes tarantoolctl rocks install hanging in resctricted networks
corner-case.

A customer configured two rocks servers:
1. offline (file:///path/to/rocks)
2. and default online (rocks.tarantool.org)

He tries to do `rocks install http 1.0.5-1`.

Online server is unavailable due to his local network policy, but
the rock is available offline. Despite anything, luarocks still
tries to fetch manifest online, which results in 30 sec hang since
network access is restricted.

This change aborts scanning when exact match is found

II. Remove cyclic dependencies

This is required to embed luarocks into tarantool, as
current tarantool preloader can't preload cyclic dependencies.
There should be a unidirectional dependency graph and
predictable order.

Note: as a consequence of this patch, operating systems other that
unix-compatible ones are no longer supported. This is because I
had to manually resolve dependency graph for predictable require()
order.

III. Use digest.md5_hex to compute md5 digests instead of openssl

luarocks has support for calculating md5 with 'md5' rock if it's
present, but we don't have it in tarantool, and instead have the
'digest' module. That's why luarocks falls back to 'openssl' binary
to calculate md5 digests.

This patch will allow luarocks to use our internal digests module.