In some cases luajit does not collect cdata objects which were
transformed with ffi.cast as tuple_bless does. In consequence, internal
table with gc callback overflows and then lua crashes. There might be an
internal luajit issue because it fires only for jitted code. But assigning
a gc callback before transformation fixes the problem.

Closes #3751

Back when bloom filters were introduced, neither box.info.memory() nor
box.stat.vinyl().memory didn't exist so bloom filters were accounted to
box.runtime.info().used for lack of a better place. Now, there's no
point to account them there. In fact, it's confusing, because bloom
filters are allocated with malloc(), not from the runtime arena, so
let's drop it.

If a tuple read from a run by a slice stream happens to be out of the
slice bounds, it will never be freed. Fix it.

The leak was introduced by commit c174c985 ("vinyl: implement new
simple write iterator").

After changing behavior of the `IS` operator (#b3a3ddb5),
`SET NULL` was rewritten to use `EQ` instead. Which doesn't respect
NULLs.

This commit fixes the null related behavior by emitting logical
constructions equivalent for this case to old `IS`.
The new expression works differently than old `IS` for nulls, however
the difference doesn't change anything, because matched rows are then
searched in a child table with `EQ` expression which do not match nulls.
Before:
`oldval` old_is `newval`
Now:
`oldval` is_null or (`newval` is_not_null and `oldval` eq `newval`)

Closes #3645

Upload tarballs of alpha and beta tarantool versions (*.0 and *.1
branches) into 2x (3x, 4x...) buckets. See more details about the
release process in the documentation: [1].

[1]: https://tarantool.io/en/doc/2.0/dev_guide/release_management/

This patch adds logging amount of rows received by applier during the
join stage, the same way that recovery has it.

Closes #3165

Remove function which deletes from cache, making replace more general:
it might be used for both insertions, deletions and replaces. Also, put
assert on equality of space pointer found in cache to old one into
replace routine.

The idea behind this change is to have 2x (and maybe later 3x, 4x, ...)
bucket for alpha and beta releases. See more details about the release
process in the documentation: [1].

[1]: https://tarantool.io/en/doc/2.0/dev_guide/release_management/

Now if the WAL thread fails to preallocate disk space needed to commit
a transaction, it will delete old WAL files until it succeeds or it
deletes all files that are not needed for local recovery from the oldest
checkpoint. After it deletes a file, it notifies the garbage collector
via the WAL watcher interface. The latter then deactivates consumers
that would need deleted files.

The user doesn't see a ENOSPC error if the WAL thread successfully
allocates disk space after deleting old files. Here's what's printed
to the log when this happens:

  wal/101/main C> ran out of disk space, try to delete old WAL files
  wal/101/main I> removed /home/vlad/src/tarantool/test/var/001_replication/master/00000000000000000005.xlog
  wal/101/main I> removed /home/vlad/src/tarantool/test/var/001_replication/master/00000000000000000006.xlog
  wal/101/main I> removed /home/vlad/src/tarantool/test/var/001_replication/master/00000000000000000007.xlog
  main/105/main C> deactivated WAL consumer replica 82d0fa3f-6881-4bc5-a2c0-a0f5dcf80120 at {1: 5}
  main/105/main C> deactivated WAL consumer replica 98dce0a8-1213-4824-b31e-c7e3c4eaf437 at {1: 7}

Closes #3397

In order to implement WAL auto-deletion, we need a notification channel
through which the WAL thread could notify TX that a WAL file was deleted
so that the latter can shoot off stale replicas. We will reuse existing
wal_watcher API for this. Currently, wal_watcher invokes the registered
callback on each WAL write so using it as is would be inefficient. To
avoid that, let's allow the caller to specify events of interest when
registering a wal_watcher.

Needed for #3397

We will add another event bitmap to wal_watcher. To avoid confusion
between them, let's rename wal_watcher->events.

This should make it easier to pass some extra information along with the
event mask. For example, we will use it to pass the vclock of the oldest
stored WAL, which is needed for WAL auto-deletion.

Needed for #3397

Added -DENABLE_LTO=ON/OFF cmake option, OFF by default.

LTO speeds up cpu-intensive workloads by up to 20% (see [1] and [2]).

Requirements to enable LTO:

- cmake >= 3.9;
- Linux: ld.bfd / ld.gold from binutils >= 2.31 (or later 2.30)
  (gold >= 1.15);
- Mac OS: xcode >= 8 (earlier versions are not tested).

The requirement of the recent ld version is due to bug with exporting
symbols from dynamic list when LTO is enabled, see [3].

Note: -Wlto-type-mismatch on GCC (enabled by default with -flto) gives
namy warnings. Filed [4] to investigate it.

Note: LuaJIT will be compiled w/o LTO despite the option set, filed [5].

[1]: https://github.com/tarantool/tarantool/wiki/performance-research
[2]: https://gist.github.com/Khatskevich/31a2da6ab46ce903120e7a03d65966db
[3]: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=84901
[4]: https://github.com/tarantool/tarantool/issues/3742
[5]: https://github.com/tarantool/tarantool/issues/3743

Closes #3117

f

In case of very aggressive optimizations the compiler can
optimize guard-breaker function away and the `unit/guard`
test would fail.

2.0 was renamed to 2.1.

Updated the test case for #2780 to check a last snapshot file
modification time instead of search log messages.

The test was flaky, because of small timeouts on Linux, but now we
spinning on a condition check to achieve both stable results and fast
execution.

Follows up #2780.
Fixes #3684.

* added more details about hung tests (#107);
* added show_reproduce_content option (#113);
* fixed inspector error reporting for a failed app test;
* expand action of use_unix_socket option to non-default servers;
* updated tarantool-python submodule (#126);
* added test_run:wait_cond() and test_run:wail_log().

Updated box-py/call.test.py result file, because tarantool-python now
uses CALL 1.7 convention by default and slightly changed yaml output
formatting. See [1] and [3] for more information.

Updated replication-py/cluster.test.py, because of changed
tarantool-python internals, see commit [2].

Updated box-py/iproto.test.py because it uses tarantool-python internals
that was rewritten in [2]. Updated its result file according to CALL 1.7
response format that was set as default with [1] and yaml output
formatting changed within [3].

Updated replication-py/swap.test.py result file, because of yaml output
formatting that was slightly changed within [3].

[1]: https://github.com/tarantool/tarantool-python/issues/82
[2]: https://github.com/tarantool/tarantool-python/commit/4639d9ae1c48f1608bd599c6d93ed6bfca48fbf9
[3]: https://github.com/tarantool/tarantool-python/issues/90

This function introduces a new xlog method xlog_fallocate() that makes
sure that the requested amount of disk space is available at the current
write position. It does that with posix_fallocate(). The new method is
called before writing anything to WAL, see wal_fallocate(). In order not
to invoke the system call too often, wal_fallocate() allocates disk
space in big chunks (1 MB).

The reason why I'm doing this is that I want to have a single and
clearly defined point in the code to handle ENOSPC errors, where
I could delete old WALs and retry (this is what #3397 is about).

Needed for #3397

Memory allocated for vy_write_iterator::src_heap is never freed. Fix it.
The leak was introduced by commit c174c985 ("vinyl: implement new
simple write iterator").

So that we can add more flags.

This patch adds a new entry to per index statistics reported by
index.stat():

  disk.statement
    inserts
    replaces
    deletes
    upserts

It shows the number of statements of each type stored in run files.
The new statistics are persisted in index files. We will need this
information so that we can force major compaction when there are too
many DELETE statements accumulated in run files.

Needed for #3225

Local variable total_size equals total_stmt_count.bytes_compressed so we
don't really need it.

tuple_extra() allows to store arbitrary metadata inside tuples.
To use it, one should set extra_size when creating a tuple_format.
It was introduced for storing UPSERT counter or column mask inside
vinyl statements. Turned out that it wasn't really needed as UPSERT
counter can be stored on lsregion while column mask doesn't need to
be stored at all.

Actually, the whole idea of tuple_extra() is rather crooked: why
would we need it if we can inherit struct tuple instead, as we do
in case of memtx_tuple and vy_stmt? Accessing an inherited struct
is much more convenient than using tuple_extra().

So this patch gets rid of tuple_extra(). To do that, it partially
reverts the following commits:

6c0842e0 vinyl: refactor vy_stmt_alloc()
74ff46d8 vinyl: add special format for tuples with column mask
11eb7816 Add extra size to tuple_format->field_map_size

This function was only used for creating a format for tuples with column
mask in vinyl. Not needed anymore and can be removed.

Anyway, it doesn't make much sense to duplciate a tuple format, because
it can be referenced instead. Besides, once JSON indexes are introcued,
duplicating a tuple format will be really painful. One more reason to
drop it now.

Finally, these atrocities are not used anywhere and can be removed.

This patch is a preparation for removing vy_stmt_column_mask.

This patch is a preparation for removing vy_stmt_column_mask.

If a REPLACE statement was generated by an UPDATE operation that updated
a column indexed by a secondary key, we can turn it into INSERT when the
secondary index is dumped, because there can't be an older statement
with the same key other than DELETE. Currently, we use the statement
column mask to detect such REPLACEs in the write iterator, but I'm
planning to get rid of vy_stmt_column_mask so let's instead introduce
a new statement flag to mark such REPLACEs.

This patch introduces a helper function vy_perform_update() that
performs operations common for UPDATE and UPSERT, namely replaces
a tuple in a transaction write set.

An UPDATE operation is written as DELETE + REPLACE to secondary indexes.
We write those statements to the memory level even if the UPDATE doesn't
actually update columns indexed by a secondary key. We filter them out
in the write iterator when the memory level is dumped. That's what we
use vy_stmt_column_mask for.

Actually, there's no point to keep those statements until dump - we
could as well filter them out when the transaction is committed. This
would even save some memory. This wouldn't hurt read operations, because
point lookup doesn't work for secondary indexes by design and so we have
to read all sources, including disk, on every read from a secondary
index.

That said, let's move update optimization from the write iterator to
vy_tx_commit. This is a step towards removing vy_stmt_column_mask.

The behaviour change was introduced in cda3cb55: sync_is_async option
was forgotten to be updated from xdir; sync_interval was forgotten too,
but was restored in 1900c58b.

The commit fixes the performance regression around 6-14% for average RPS
on default nosqlbench workload with 30 seconds duration. The additional
information about benchmarking can be found in #3747.

Thanks to Vladimir Davydov (@locker) for the investigation of the
cda3cb55 changes.

Closes #3747

(cherry picked from commit cd9cc4c5)

The behaviour change was introduced in cda3cb55: sync_is_async option
was forgotten to be updated from xdir; sync_interval was forgotten too,
but was restored in 1900c58b.

The commit fixes the performance regression around 6-14% for average RPS
on default nosqlbench workload with 30 seconds duration. The additional
information about benchmarking can be found in #3747.

Thanks to Vladimir Davydov (@locker) for the investigation of the
cda3cb55 changes.

Closes #3747