- Remove extra scan of the WAL directory from local_recovery() - we
   scan the directory in recovery_end_vclock() hence we can skip scan in
   recover_remaining_wals() by passing scan_dir = false.

 - Rename recovery_end_vclock() to recovery_scan() to emphasize the fact
   that this function scans the WAL directory. Write a comment to this
   function.

 - Add comments to wal.c explaining why we scan the WAL directory there.

Follow-up 0695fbbb ("box: retrieve end vclock before starting local
recovery").

To bring crash_expected option of "start server" command.

There were no checks for granting and revoking a non-existent
privilege or a privilege to a non-existent entity.
Added the checks, and a test case.

Closes #3417

If is_first_insert flag is set and vy_stmt_type(rv->tuple) equals
IPROTO_DELETE, we free rv->tuple, but then we dereference it via
an on-stack variable to check if we need to turn a REPLACE into an
INSERT or vice versa. Fix this.

This is not necessary, as we can use fiber()->gc, as we usually do.

Move key_def creation to compare_write_iterator_results as it is the
same for all test cases. Performance is not an issue here, obviously, so
we can close our eyes to the fact that now we create a new key def for
each test cases.

The new function inserts a new item into the list at the specified
postion.

If a fiber pool reuses already canceled fiber then the fiber reports an
error for any next request. Now canceled fiber returns and fiber pool
creates a new one.

Fixes #3527

New commit in third_party/libyaml downgrades
required cmake version.

Need to update tests as with fixup in upstrem
commit baf636a74b4b6d055d93e2d01366d6097eb82d90
Author: Tina Müller <cpan2@tinita.de>
Date:   Thu Jun 14 19:27:04 2018 +0200

The closing single quote needs to be indented...
if it's on its own line.

Closes #3275.

Closes #3275.

Found by @ImeevMA

 - Factor out local_recovery() from box_cfg_xc(). Make it setup
   replication and handle local recovery and hot standby cases.
 - Move replication setup in case of initial bootstrap from box_cfg_xc()
   to bootstrap() to make bootstrap() consistent with local_recovery().
 - Move initial snapshot creation from bootstrap() to bootsrap_master()
   and bootstrap_from_master().

Needed for #461

box_sync_replication() can now be called before recovery, right after
box_listen(). This is a step toward detecting if the instance fell too
much behind its peers in the cluster and so needs to be rebootstrapped.

Needed for #461

In order to find out if the current instance fell too much behind its
peers in the cluster and so needs to be re-bootstrapped we need to
connect it to remote peers before proceeding to local recovery. The
problem is box.cfg.replication may have an entry corresponding to the
instance itself so before connecting we have to start listening to
incoming connections. So this patch moves the call to box_listen()
before recoery is started unless the instance in hot standby mode.
It also folds box_bind() into box_listen() as it is no longer needed
as a separate function.

Needed for #461

In order to find out if the current instance fell too much behind its
peers in the cluster and so needs to be rebootstrapped, we need to know
its vclock before we start local recovery. To do that, let's scan the
most recent xlog. In future, we can optimize that by either storing end
vclock in xlog eof marker or by making a new xlog on server stop.

Needed for #461

Introduce vclock_is_set() helper and use it on xlog_meta::prev_vclock
instead.

Follow-up ac90b498 ("xlog: store prev vclock in xlog header").

Store replication group id in vylog and ignore spaces whose group_id
equals GROUP_LOCAL when relaying initial join rows.

Folow-up #3443

Follow-up #3443

Replica local spaces can't cause replication conflicts so we don't
really need to respect box.cfg.read_only for them (they are similar
to temporary spaces in this regard).

Follow-up #3443

Next checkpoint time is set by the formula:
 period = self.checkpoint_interval + offset,
where offset is defined as follow:
 offset = random % self.checkpoint_interval

So offset must be calculated again if at least the new
interval is less than the old one.

Closes #3370

vy_mem_iterator_next is as effecient as the current implementation of
vy_point_lookup_scan_mem, because it doesn't copy statements anymore
(see commit 1e1c1fdb vinyl: make read iterator always return newest
tuple version). Let's use it instead of open-coding vy_mem tree lookup.

Now it is possible to specify a number in exponential
form via all formats allowed by json standard.
json.decode('{"remained_amount":2.0e+3}')
json.decode('{"remained_amount":2.0E+3}')
json.decode('{"remained_amount":2e+3}')
json.decode('{"remained_amount":2E+3}')     <-- fixed

Closes #3514.

This patch introduces a new space option, group_id, which defines how
the space is replicated. If it is 0 (default), the space is replicated
throughout the entire cluster. If it is 1, the space is replica local,
i.e. all changes made to it are invisible to other replicas in the
cluster. Currently, no other value is permitted, but in future we will
use this option for setting up arbitrary replication groups in a
cluster. The option can only be set on space creation and cannot be
altered.

Since the concept of replication groups hasn't been established yet,
group_id isn't exposed to Lua. Instead, we use is_local flag, both in
box.schema.space.create arguments and in box.space output.

Technically, to support this feature, we introduce a new header key,
IPROTO_GROUP_ID, which is set to the space group id for all rows
corresponding to a space, both in xlog and in snap. Relay won't send
snapshot rows whose group_id is 1. As for xlog rows, they are
transformed to IPROTO_NOP so as to promote vclock on replicas without
any actual data modification.

The feature is currently supported for memtx spaces only, but it should
be easy to implement it for vinyl spaces as well.

Closes #3443

@TarantoolBot document
Title: Document new space option - is_local
If a space is created with is_local flag set in options, changes made to
the space will be persisted, but won't be replicated.

If a vinyl iterator is passed to another fiber, it may trigger a
use-after-free bug, because the tx it's using may be destroyed
while it's reading the disk. So let's explicitly ban that.

Closes #3394

Fixes commit a09c04bf ("test: fix a sporadic failure of
replication/gc.test").

The test was failing due to a race condition between gc and relay
threads on the master: replica could have acknoledged the last xlog
before it was stopped, in which case gc would correctly delete it and
the total number of xlogs would be equal to 2, not 3.

We typically prefix all boolean variables with 'is_', so let's rename
space_opts::temporary to is_temporary for consistency.

While we are at it, let's also rename tuple_format::temporary to
is_temporary and use space_is_temporary() helper wherever we have
a space pointer.

A NOP request has no body, but since it is treated as DML, we still
encode a zero-size map for its body. This complicates conversion of
local requests to NOP in relay as we can't omit xrow_encode_dml (see
the next patch), so let's allow DML requests to be bodiless.

Needed for #3443

Currently, IPROTO_NOP can only be generated by a before_replace trigger,
when it returns the old tuple thus turning the original operation into a
NOP. In such a case we know the space id and we write it to the request
body. This allows us to dispatch NOP requests via DML route.

As a part of replica local spaces feature, we will substitute requests
operating on local spaces with NOP in relay in order to promote vclock
on replicas without actual data modification. Since space_id is stored
in request body, sending it to replicas would mean decoding the request
body in relay, which is an overkill. To avoid that, let's separate NOP
and DML paths and remove space_id from NOP requests.

Needed for #3443

If pk_def passed to index_def_new() is not NULL, the function will merge
it with the given key_def to create index cmp_def, no matter if the
index is primary or secondary. When an index is altered, we call
index_def_new() to create the new definition, passing the primary key
definition of the altered space for pk_def. If it is the primary index
that is altered, we will pass the definition of the old primary index
and index_def_new() will happily merge it with the new index definition,
resulting in invalid index_def::cmp_def. This doesn't affect memtx, as
memtx doesn't use cmp_def for unique indexes, but it does affect vinyl
in a peculiar way:

  tarantool> _ = box.schema.space.create('test', {engine = 'vinyl'})
  ---
  ...

  tarantool> _ = box.space.test:create_index('pk')
  ---
  ...

  tarantool> _ = box.space.test.index.pk:alter{parts = {2, 'unsigned'}}
  ---
  ...

  tarantool> _ = box.space.test:replace{1, 1}
  ---
  ...

  tarantool> _ = box.space.test:replace{2, 1}
  ---
  ...

  tarantool> box.space.test:select()
  ---
  - - [1, 1]
    - [2, 1]
  ...

(expected: [2, 1])

Fix this by making index_def_new() merge key_def with pk_def only for
secondary indexes.

Closes #3508