Because it's the minimal number of checkpoints that must not be deleted,
not the actual number of preserved checkpoints. Do it now, in a separate
patch so as to ease review of the next patch.

While we are at it, fix the comment to gc_set_(min_)checkpoint_count()
which got outdated by commit 5512053f ("box: gc: do not remove files
being backed up").

It's better than using tt_snprintf at call sites.

gc_consumer_new is used in gc_consumer_register. Let's fold it to make
the code flow more straightforward.

The length of a consumer name never exceeds 64 characters so no use to
allocate a string. This is a mere code simplification.

It's exasperating to write trivial external functions for each member of
an opaque struct (gc_consumer_vclock, gc_consumer_name, etc) while we
could simply access those fields directly if we made those structs
transparent. Since we usually define structs as transparent if we need
to use them outside a source file, let's do the same for gc_consumer and
gc_state and remove all those one-line wrappers.

If an instance is restarted while building a new vinyl index, there will
probably be some run files left. Currently, we won't delete such files
until box.snapshot() is called, even though there's no point in keeping
them around. Let's tweak vy_gc_lsm() so that it marks all runs that
belong to an unfinished index as incomplete to force vy_gc() to remove
them immediately after recovery is complete.

This also removes files left from a failed rebootstrap attempt so we can
remove a call to box.snapshot() from vinyl/replica_rejoin.test.lua.

This patch fixes a trivial error on vy_send_range() error path which
results in a master crash in case a file needed to join a replica is
missing or corrupted.

See #3708

The main fiber should have a lua state as any other lua fiber.

Needed for #3538

Added MAKE_BUILD_TYPE=RelWithDebInfoWError option, which means enabling
-DNDEBUG=1, -O2 and -Wall -Wextra -Werror. This ensures we have clean
release build without warnings.

Fixed found -Wunused-variable and -Wunused-parameter warnings.

Part of #3238.

Closes #3709

Patch fixes behavior when replica tries to connect to the same master
more than once. In case when it is initial configuration we raise the
exception. If it in not initial config we print the error and disconnect
the applier.

@locker: minor test cleanup.

Closes #3610

They are only used to set corresponding members of vy_quota, vy_run_env,
and vy_scheduler when vy_env is created. No point in keeping them around
all the time.

Currently, we create a quota object with the limit maximized, and only
set the configured limit when local recovery is complete, so as to make
sure that no dump is triggered during recovery. As a result, we have to
store the configured limit in vy_env::memory, which looks ugly, because
this member is never used afterwards. Let's introduce a new method
vy_quota_enable to enable quota so that we can set the limit right on
quota object construction. This implies that we add a boolean flag to
vy_quota and only check the limit if it is set.

There's another reason to add such a method. Soon we will implement
quota consumption rate limiting. Rate limiting requires a periodic timer
that would replenish quota. It only makes sense to start such a timer
upon recovery completion, which again leads us to an explicit method for
enabling quota.

vy_env::memory will be removed by the following patch along with a few
other pointless members of vy_env.

Needed for #1862

Using fiber_cond as a wait queue isn't very convenient, because:
 - It doesn't allow us to put a spuriously woken up fiber back to the
   same position in the queue where it was, thus violating fairness.
 - It doesn't allow us to check whether we actually need to wake up a
   fiber or it will have to go back to sleep anyway as it needs more
   memory than currently available.
 - It doesn't allow us to implement a multi-queue approach where fibers
   that have different priorities are put to different queues.

So let's rewrite the wait queue with plain rlist and fiber_yield.

Needed for #1862

There's a sanity check in vinyl_engine_prepare, which checks if the
transaction size is less than the configured limit and fails without
waiting for quota if it isn't. Let's move this check to vy_quota_use,
because it's really a business of the quota object. This implies that
vy_quota_use has to set diag to differentiate this error from timeout.

The refactoring is targeted at facilitating introduction of rate
limiting within the quota class. It moves code blocks around, factors
out some blocks in functions, and improves comments. No functional
changes.

Needed for #1862

Turned out that throttling isn't going to be as simple as maintaining
the write rate below the estimated dump bandwidth, because we also need
to take into account whether compaction keeps up with dumps. Tracking
compaction progress isn't a trivial task and mixing it in a module
responsible for resource limiting, which vy_quota is, doesn't seem to be
a good idea. Let's factor out the related code into a separate module
and call it vy_regulator. Currently, the new module only keeps track of
the write rate and the dump bandwidth and sets the memory watermark
accordingly, but soon we will extend it to configure throttling as well.

Since write rate and dump bandwidth are now a part of the regulator
subsystem, this patch renames 'quota' entry of box.stat.vinyl() to
'regulator'. It also removes 'quota.usage' and 'quota.limit' altogether,
because memory usage is reported under 'memory.level0' while the limit
can be read from box.cfg.vinyl_memory, and renames 'use_rate' to
'write_rate', because the latter seems to be a more appropriate name.

Needed for #1862

There are three places where we start the scheduler fiber and enable the
configured memory quota limit: local bootstrap, remote bootstrap, and
local recovery completion. I'm planning to add more code there so let's
factor it out now.

When replication is restarted with the same replica set configuration
(i.e. box.cfg{replication = box.cfg.replication}), there's a chance that
an old relay will be still running on the master at the time when a new
applier tries to subscribe. In this case the applier will get an error:

  main/152/applier/localhost:62649 I> can't join/subscribe
  main/152/applier/localhost:62649 xrow.c:891 E> ER_CFG: Incorrect value for
      option 'replication': duplicate connection with the same replica UUID

Such an error won't stop the applier - it will keep trying to reconnect:

  main/152/applier/localhost:62649 I> will retry every 1.00 second

However, it will stop synchronization so that box.cfg() will return
without an error, but leave the replica in the orphan mode:

  main/151/console/::1:42606 C> failed to synchronize with 1 out of 1 replicas
  main/151/console/::1:42606 C> entering orphan mode
  main/151/console/::1:42606 I> set 'replication' configuration option to
    "localhost:62649"

In a second, the stray relay on the master will probably exit and the
applier will manage to subscribe so that the replica will leave the
orphan mode:

  main/152/applier/localhost:62649 C> leaving orphan mode

This is very annoying, because there's no need to enter the orphan mode
in this case - we could as well keep trying to synchronize until the
applier finally succeeds to subscribe or replication_sync_timeout is
triggered.

So this patch makes appliers enter "loading" state on configuration
errors, the same state they enter if they detect that bootstrap hasn't
finished yet. This guarantees that configuration errors, like the one
above, won't break synchronization and leave the user gaping at the
unprovoked orphan mode.

Apart from the issue in question (#3636), this patch also fixes spurious
replication-py/multi test failures that happened for exactly the same
reason (#3692).

Closes #3636
Closes #3692

First, we print "will retry every XX second" to the log after an error
message only for socket and system errors although we keep trying to
establish a replication connection after configuration errors as well.
Let's print this message for those errors too to avoid confusion.

Second, in case we receive an error in reply to SUBSCRIBE command, we
log "can't read row" instead of "can't join/subscribe". This happens,
because we switch an applier to SYNC/FOLLOW state before receiving a
reply to SUBSCRIBE command. Fix this by updating an applier state only
after successfully subscribing.

Third, we detect duplicate connections coming from the same replica on
the master only after sending a reply to SUBSCRIBE command, that is in
relay_subscribe rather than in box_process_subscribe. This results in
"can't read row" being printed to the replica's log even though it's
actually a SUBSCRIBE error. Fix this by moving the check where it
actually belongs.

In some cases no-ops are written to xlog. They have no effect but are
needed to bump lsn.

Some time ago (see commit 89e5b784) such
ops were made bodiless, and empty body requests are not handled in
xrow_header_decode(). This leads to recovery errors in special case:
when we have a multi-statement transaction containing no-ops written to
xlog, upon recovering from such xlog, all data after the no-op end till
the start of new transaction will become no-op's body, so, effectively,
it will be ignored. Here's example `tarantoolctl cat` output showing
this (BODY contains next request data):

    ---
    HEADER:
      lsn: 5
      replica_id: 1
      type: NOP
      timestamp: 1536656270.5092
    BODY:
      type: 3
      timestamp: 1536656270.5092
      lsn: 6
      replica_id: 1
    ---
    HEADER:
      type: 0
    ...

This patch handles no-ops correctly in xrow_header_decode().

@locker: refactored the test case so as not to restart the server for
a second time.

Closes #3678

fio.rmtree should use lstat instead of stat, otherwise it won't be
able to remove a directory if there's a symbolic link pointing to a
non-existent file.

The test case will be added to app/fio.test.lua by the following commit,
which is aimed at cleaning up /tmp directory after running tests.

Currently, there are two ways of creating a new key definition object
apart from copying (key_def_dup): either use key_def_new_with_parts,
which takes definition of all key parts and returns a ready to use
key_def, or allocate an empty key_def with key_def_new and then fill it
up with key_def_set_part. The latter method is rather awkward: because
of its existence key_def_set_part has to detect if all parts have been
set and initialize comparators if so. It is only used in schema_init,
which could as well use key_def_new_with_parts without making the code
any more difficult to read than it is now.

That being said, let us:
 - Make schema_init use key_def_new_with_parts.
 - Delete key_def_new and bequeath its name to key_def_new_with_parts.
 - Simplify key_def_set_part: now it only initializes the given part
   while comparators are set by the caller once all parts have been set.

These changes should also make it easier to add json path to key_part.

This reverts commit ea3a2b5f.

Once we finally implement json path indexes, more fields that are
calculated at run time will have to be added to struct key_part, like
path hash or field offset. So this was actually a mistake to remove
key_part_def struct, as it will grow more and more different from
key_part. Conceptually having separate key_part_def and key_part is
consistent with other structures, e.g. tuple_field and field_def.
That said, let's bring key_part_def back. Sorry for the noise.

The only difference between struct key_part_def and struct key_part is
that the former stores only the id of a collation while the latter also
stores a pointer to speed up tuple comparisons. It isn't worth keeping a
separate struct just because of that. Let's use struct key_part
everywhere and assume that key_part->coll is NULL if the part is needed
solely for storing a decoded key part definition and isn't NULL if it is
used for tuple comparisons (i.e. is attached to a key_def).

Start use tuple_field_by_part(_raw) routine in *extract,
*compare, *hash functions. This new function use key_part to
retrieve field data mentioned in key_part. Now it is just a
wrapper for tuple_field_raw but with introducing JSON paths
it would work in other way.

Needed for #1012

To introduce JSON indexes we need changeable key_def containing
key_part definition that would store JSON path and offset slot
and slot epoch in following patches.

Needed for #1012

The problem is that clang does not support -Wno-cast-function-type flag.
It is the regression from 8c538963.

Follow up of #3685.
Fixes #3701.

This patch adds some essential disk statistics that are already
collected and reported on per index basis to box.stat.vinyl().
The new statistics are shown under the 'disk' section and currently
include the following fields:

 - data: size of data stored on disk.
 - index: size of index stored on disk.
 - dump.in: size of dump input.
 - dump.out: size of dump output.
 - compact.in: size of compaction input.
 - compact.out: size of compaction output.
 - compact.queue: size of compaction queue.

All the counters are given in bytes without taking into account
disk compression. Dump/compaction in/out counters can be reset with
box.stat.reset().

So that we can easily extend them to account the stats not only per LSM
tree, but also globally, in vy_lsm_env.

Currently, there's no way to figure out whether compaction keeps up
with dumps or not while this is essential for implementing transaction
throttling. This patch adds a metric that is supposed to help answer
this question. This is the compaction queue size. It is calculated per
range and per LSM tree as the total size of slices awaiting compaction.
We update the metric along with the compaction priority of a range, in
vy_range_update_compact_priority(), and account it to an LSM tree in
vy_lsm_acct_range(). For now, the new metric is reported only on per
index basis, in index.stat() under disk.compact.queue.

Currently, we call memset() on vy_stmt_counter and vy_disk_stmt_counter
directly, but that looks rather ugly, especially when a counter has a
long name. Let's introduce helper functions for that.

There's no reason not to report pages and bytes_compressed under
disk.stat.dump.out and disk.stat.compact.{in,out} apart from using
the same struct for dump and compaction statistics (vy_compact_stat).
The statistics are going to differ anyway once compaction queue size
is added to disk.stat.compact so let's zap struct vy_compact_stat
and report as much info as we can.

The code is difficult to follow when there are nested info tables,
because info_table_end() doesn't refer to the table name. Let's
annotate info_table_end() with a comment to make it easier to follow.
No functional changes.

When a few ranges are coalesced, we "force" compaction of the resulting
range by raising its compaction priority to max (slice count). There's
actually no point in that, because as long as the shape of the resulting
LSM tree is OK, we don't need to do extra compaction work. Moreover, it
actually doesn't work if a new slice is added to the resulting range by
dump before it gets compacted, which is fairly likely, because then its
compaction priority will be recalculated as usual. So let's simply call
vy_range_update_compact_priority() for the resulting range.

When a range is split, the produced ranges will inherit its compaction
priority. This is actually incorrect, because range split may change the
shape of the tree so let's recalculate priority for each part the usual
way, i.e. by calling vy_range_update_compact_priority().

After this patch, there's this only place where we can update compaction
priority of a range - it's vy_range_update_compact_priority().

This patch addresses a few problems index.compact() is suffering from,
namely:

 - When a range is split or coalesced, it should inherit the value of
   needs_compaction flag from the source ranges. Currently, the flag is
   cleared so that the resulting range may be not compacted.

 - If a range has no slices, we shouldn't set needs_compaction flag for
   it, because obviously it can't be compacted, but we do.

 - The needs_compaction flag should be cleared as soon as we schedule a
   range for compaction, not when all slices have been compacted into
   one, as we presently expect, because the latter may never happen
   under a write-intensive load.

If some error occured during execution of a function called from
box.session.su(), we assumed that fiber diagnostics area was not empty,
and tried to print an error message using data from the diagnostics.
However, this assumption is not true when some lua error happens.
Imagine such a case:

  box.session.su('admin', function(x) return #x end, 3)

A lua error would be pushed on the stack but the diagnostics would be
empty, and we would get an assertion failure when trying to print the
error message. Handle this by using lua_error() instead of luaT_error().

Closes #3659

Fixed false positive -Wimplicit-fallthrough in http_parser.c by adding a
break. The code jumps anyway, so the execution flow is not changed.

Fixed false positive -Wparenthesis in reflection.h by removing the
parentheses. The argument 'method' of the macro 'type_foreach_method' is
just name of the loop variable and is passed to the macro for
readability reasons.

Fixed false positive -Wcast-function-type triggered by reflection.h by
adding -Wno-cast-function-type for sources and unit tests. We cast a
pointer to a member function to an another pointer to member function to
store it in a structure, but we cast it back before made a call. It is
legal and does not lead to an undefined behaviour.

Fixes #3685.

Add an ability to pass options to json.encode()/decode().

Closes: #2888.

@TarantoolBot document
Title: json.encode() json.decode()
Add an ability to pass options to
json.encode() and json.decode().
These are the same options that
are used globally in json.cfg().

Since addition of -fopenmp to compiler also means
addition of -lgomp to the link stage, pass -fno-openmp
to the linking stage in case of static build. In that
case OMP functions are statically linked into libmisc.

Also, emit error if trying to perform static build using
clang.