Needed to check if the key definition loaded from vylog to send initial
data to a replica has the collation properly recovered.

It isn't stored currently, but this doesn't break anything, because the
primary key, which is the only key whose definition is used after having
been loaded from vylog, can't be nullable. Let's store it there just in
case. Update the vinyl/layout test to check that.

Collations were disabled in vinyl by commmit 2097908f ("Fix
collation test on some platforms and disable collation in vinyl"),
because a key_def referencing a collation could not be loaded from
vylog on recovery (collation objects are created after vylog is
recovered). Now, it isn't a problem anymore, because the decoding
procedure, key_def_decode_parts(), deals with struct key_part_def,
which references a collation by id and hence doesn't need a collation
object to be created. So we can enable collations in vinyl.

This patch partially reverts the aforementioned commit (it can't
do full revert, because that commit also fixed some tests along
the way).

Closes #2822

We can't use key_def_decode_parts() when recovering vylog if key_def has
a collation, because vylog is recovered before the snapshot, i.e. when
collation objects haven't been created yet, while key_def_decode_parts()
tries to look up the collation by id. As a result, we can't enable
collations for vinyl indexes.

To fix this, let's rework the decoding procedure so that it works with
struct key_part_def instead of key_part.  The only difference between
the two structures is that the former references the collation by id
while the latter by pointer.

Needed for #2822

Writer fiber should be stopped before re-connect to avoid
sending unwanted IPROTO_OK replication acknowledges.

Fixes #2726

SUBSCRIBE command is not multiplexed in the binary protocol.
When relay exits with an error during subscribe, remote replica
still continue to send IPROTO_OK replication acknowledges to
the master. These packets are unwanted by IPROTO decoder.

Close socket on errors during SUBSCRIBE.

Fixes #2726

The corresponding comparator is missing, which leads to a crash.
Fix it and add a test case checking that nullable indexes work fine
with all available types.

Symbol resolving can be expensive. Introduce an option for fiber.info():

    fiber.info({ backtrace = true })
    fiber.info({ bt = true })

Fixes #2878

- Change signature of function access_check_func.
  Now it returns status instead of function.

Close #2816

> src/tarantool/src/box/vinyl.c:2111:33: error: initializer element is not a compile-time constant
>         static const double weight = 1 - exp(-VY_QUOTA_UPDATE_INTERVAL /
>                                      ~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Remove the "static" qualifier. It is not really needed as any sane
compiler will pre-calculate the value of 'weight' at compile time
(checked on gcc 6.3.0 with -O0).

 - Remove vy_stat::rmean statistics, which were left from Sophia, as now
   we have per index statistics which are much more verbose than those.

 - Move vy_stat::dump_bw to vy_env and remove struct vy_stat as there's
   nothing left in it.

 - Move quota statistics from box.info.vinyl().performance.memory to
   box.info.vinyl().quota. Remove 'ratio', which equals used / limit, as
   this kind of calculations should be done by a script aggregating
   statistics. Report 'use_rate' and 'dump_bandwidth' there.

 - Report 'limit' in cache statistics to make them consistent with
   'quota' statistics, where 'limit' is reported. Rename 'cache.count'
   to 'cache.tuples'. Remove vy_cache_env::cache_count, use mempool
   stats instead.

 - Move 'tx_allocated', 'txv_allocated', 'read_interval', 'read_view'
   from box.info.vinyl().performance to box.info.vinyl().tx and name
   them 'transactions', 'statements', 'gap_locks', and 'read_views',
   respectively. Remove vy_tx_stat::active and 'tx.active' as the same
   value is shown by 'tx.transactions', extracted from the mempool.

 - Zap box.info.vinyl().performance - there's nothing left there.

Now global statistics look like:

  tarantool> box.info.vinyl()
  ---
  - cache:
      limit: 134217728
      tuples: 32344
      used: 34898794
    tx:
      conflict: 1
      commit: 324
      rollback: 13
      statements: 10
      transactions: 3
      gap_locks: 4
      read_views: 1
    quota:
      dump_bandwidth: 10000000
      watermark: 119488351
      use_rate: 1232703
      limit: 134217728
      used: 34014634
  ...

Closes #2861

We have a timer for updating watermark every second. Let's reuse it for
quota use rate calculation. This will allow us to get rid of legacy
vinyl statistics.

Also, let's use EWMA for calculating the average. It is a more efficient
and common method, which allows to easily tune the period over which the
value is averaged.

Follow up #1557

Collation was simply ignored for non-string parts, that could
confuse potential user.

Generate a readable error in this case.

Fix #2862 part 2

Now collation is silently ignored for type='scalar' parts.

Use collation for string scalar fields.

Fix #2862 part 1

Show collation name (if present) in space.index.name.parts[no].

Fix #2862 part 4

test:create_index('unicode_s1', {parts = {{1, 'STR', collation =
'UNICODE'}}}) will work now.

Fix #2862 part 3

If a field is not indexed and no more indexed or not nullable
fields after that, than allow to skip it in insertion. Such field
value looks like MP_NIL, but MP_NIL is not explicitly stored.
Named access to this field in lua returns nil.

Example:
format =
{{'field1'},
 {'field2'},
 {'field3', is_nullable = true},
 {'field4', is_nullable = true}}

t = space:insert{1, 2} -- ok.

t.field1 == 1, t.field2 == 2, t.field3 == nil, t.field4 == nil

Closes #2880

Some users store in format fields their custom keys. But current
opts parser does not allow to store any unknown keys. Lets allow it.

Example:
format = {}
format[1] = {name = 'field1', type = 'unsigned', custom_field = 'custom_value'}
s = box.schema.create_space('test', {format = format})
s:format()[1].custom_field == 'custom_value'

Closes #2839

Using DML/DDL on a Vinyl index with wal_mode = 'none' is likely to
result in unrecoverable errors like:

  F> can't initialize storage: Invalid VYLOG file: Index 512/0 created twice

To avoid data corruption in case the user tries to use an existing Vinyl
database in conjunction with wal_mode = 'none', let's explicitly forbid
it until we figure out how to fix it.

Workaround #2278

During initial join, a replica receives all data accumulated on the
master for its whole lifetime, which may be quota a lot. If the network
connection is fast enough, the replica might fail to keep up with dumps,
in which case replication fails with ER_VY_QUOTA_TIMEOUT. To avoid that,
let's ignore quota timeout until bootstrap is complete.

Note, replication may still fail during the 'subscribe' stage for the
same reason, but it's unlikely, because the rate at which the master
sends data is limited by the number of requests served by the master per
a unit of time, and it should become nearly impossible once throttling
is introduced (See #1862).

Closes #2873

If the user sets snap_dir to an empty directory by mistake while leaving
vinyl_dir the same, tarantool will still bootstrap, but there is likely
to be errors like:

  vinyl.c:835 E> 512/0: dump failed: file './512/0/00000000000000000001.run' already exists
  vy_log.c:1095 E> failed to rotate metadata log: file './00000000000000000005.vylog' already exists

Even worse, it may eventually fail to restart with:

  vy_log.c:886 E> ER_MISSING_SNAPSHOT: Can't find snapshot

To avoid that, let's check the vinyl_dir on bootstrap and abort if it
contains vylog files left from previous setups.

Closes #2872

The only reason why it was allocated is that struct vy_scheduler was
defined after struct vy_env, which is not a problem any more. Embedding
it allows us to drop the extra argument to vy_scheduler_need_dump_f().

It's a big independent entity, let's isolate its code in
a separate file.

While we are at it, add missing comments to vy_scheduler
struct members.

Instead of storing a pointer to vy_env in vy_scheduler, let's:

 - Add pointers to tx_manager::read_views and vy_env::run_env to
   vy_scheduler struct. They are needed to create a write iterator
   for a dump/compaction task.

 - Add a callback to struct vy_scheduler that is called upon dump
   completion to free memory. This allows us to eliminate accesses
   vy_env::quota and vy_env::allocator from vy_scheduler code.

 - Move the assert that assures that the scheduler isn't started during
   local recovery from vy_scheduler_f() to vy_env_quota_exceeded_cb()
   callback so that we don't need to access vy_env::status from the
   scheduler code. Note, after this change we have to set vy_env::status
   to VINYL_ONLINE before calling vy_quota_set_limit(), because the
   latter might schedule a dump.

 - Check if we have anything to dump from vy_begin_checkpoint() instead
   of vy_scheduler_begin_checkpoint().

This will allow us to isolate the scheduler code in a separate file.

Currently, dump is triggered (by bumping the memory generation) by the
scheduler fiber while quota consumers just wake it up. As a result, the
scheduler depends on the quota - it has to access the quota to check if
it needs to trigger dump. In order to move the scheduler to a separate
source file, we need to get rid of this dependency.

Let's rework this code as follows:

 - Remove vy_scheduler_trigger_dump() from vy_scheduler_peek_dump(). The
   scheduler fiber now just dumps all indexes eligible for dump and
   completes dump by bumping dump_generation. It doesn't trigger dump by
   bumping generation anymore. As a result, it doesn't need to access
   the quota.

 - Make quota consumers call vy_scheduler_trigger_dump() instead of just
   waking up the scheduler. This function will become a public one once
   the scheduler is moved out of vinyl.c. The function logic is changed
   a bit. First, besides bumping generation, it now also wakes up the
   scheduler fiber. Second, it does nothing if dump is already in
   progress or can't be scheduled because of concurrent checkpoint.
   In the latter case it sets a special flag though that will force the
   scheduler trigger dump upon checkpoint completion.

 - vy_scheduler_begin_checkpoint() can't use vy_scheduler_trigger_dump()
   anymore due to additional checks added to the function, so it bumps
   the generation directly. This looks fine.

 - Such a design has a subtlety regarding how quota consumers notify the
   scheduler and how they are notified back about available quota.
   In extreme cases, quota released by a dump may be not enough to
   satisfy all consumers, in which case we need to reschedule dump.
   Since the scheduler doesn't check the quota anymore and doesn't
   reschedule dump, it has to be done by the left consumers. So
   consumers has to call the quota_exceeded_cb (which triggers a dump
   now) callback every time they are woken up and see there's not enough
   quota. The vy_quota_use() is reworked accordingly.

   Also, since the quota usage may exceed the limit (because of
   vy_quota_force_use()), the quota usage may remain higher than the
   limit after a dump completion, in which case vy_quota_release()
   doesn't wake up consumers and again there's no one to trigger another
   dump. So we must wake up all consumers every time vy_quota_release()
   is called.

quota_cond, which is used for throttling quota consumers, doesn't really
belong to vy_scheduler. It would fit much better in vy_quota. Let's move
it there. This also allows us to remove the two callbacks from vy_quota
struct, quota_throttled_cb and quota_released_cb, and make the code more
straightforward.

While we are at it, let's also rename vy_scheduler_quota_exceeded_cb()
to vy_env_quota_exceeded_cb().

If xlog_flush() fails, box.snapshot() will still succeed, but
recovery from such an incomplete snapshot will fail. Fix it and
add the corresponding test case.

Update netbox.space objects in-place instead of re-creating them
on every schema change.

Closes #2401

* Add support of string.strip, string.lstrip, string.rstrip

Closes #2785

In 1.7 the join procedure consists of two phases: initial, during which
we send the last snapshot, and final, when we send xlogs written after
the snapshot. Between the two phases, the replica uuid is added to the
cluster table on the master, so by the time join is finished, the
replica should have received its id.

However, on 1.6 there's no final join phase, instead the master expects
the replica to receive xlogs upon subscription. As a result, the replica
doesn't receive its id until it sends the subscribe request.  This is
not expected by 1.7 clients - they fail with ER_UNKNOWN_REPLICA.

Fix this problem by making 1.7 replicas proceed to subscription and wait
until the id is received before completing bootstrap from 1.6 master.

Closes #2702

There was a bug in small garbage collection that resulted in tuple
leak in case box.snapshot() races with DML. The leak was indicated
by constantly growing box.slab.info().items_used. Update the small
library to fix it.

Closes #2842

This reverts commit 8b6cefd0.

This feature is so good to be pushed into 1.7. Sorry.

Improve usability.

Follow up 9297ec36 "chmod and chown control socket"

Fixes #2852.
Fixes #2849.

* Fix parsing in case of unexpected headers
* Fix duplicating headers in case of retransmitting responses

A workaround for #2836

Tomap() creates a lua table with both names and number indexes.
Each named field stored by its name AND by its index in a tuple.
For example, if a tuple is {'a', 'b', 'c'} and its format is
{'field1', 'field2'}, then t.field1 is the same as t[1], t.field2 is
the same as t[2].

Not named fields can be accessed only by their indexes. For the example
above 'c' can be accessed only as t[3].

Closes #2821

* Call box.cfg() instead of raising an error on the first access to box.XXX

Fixes #2559