The test checks that files left after rebootstrap are removed by the
garbage collector. It does that by printing file names to the result
file. This is inherently unstable, because should timing change, and
we can easily get an extra dump or compaction resulting in a different
set of files and hence test failure. Let's rewrite the test so that
it checks that files are actually removed using fio.path.exists().

Timeout injections are unstable and difficult to use. Injecting a delay
is much more convenient.

ERROR_INJECT_YIELD yields the current fiber execution by calling
fiber_sleep(0.001) while the given error injection is set.

ERROR_INJECT_SLEEP suspends the current thread execution by calling
usleep(1000) while the given error injection is set.

When we implement transactional DDL, txn_last_stmt won't necessarily
point to the right statement on commit or rollback so we must avoid
using it.

Note, replicas are still registered/unregisterd in the on_commit
trigger, but that's okay, as we don't really need _cluster space to
be in sync with the replica set.

When we implement transactional DDL, txn_last_stmt won't necessarily
point to the right statement on commit or rollback so we must avoid
using it.

While we are at it, let's also make sure that changes are propagated
to Lua on replace, not on commit, by moving on_alter_space trigger
invocation appropriately.

When we implement transactional DDL, txn_last_stmt won't necessarily
point to the right statement on commit or rollback so we must avoid
using it.

When we implement transactional DDL, txn_last_stmt won't necessarily
point to the right statement on commit or rollback so we must avoid
using it.

A sequence isn't supposed to roll back to the old value if the
transaction it was used in is aborted for some reason. However,
if a sequence is dropped, we do want to restore the original
value on rollback so that we don't lose it on an unsuccessful
attempt to drop the sequence.

_space_sequence changes are not rolled back properly. Fix it keeping in
mind that our ultimate goal is to implement transactional DDL, which
implies that all changes to the schema should be done synchronously,
i.e. on_replace, not on_commit.

To implement transactional DDL, we must make sure that in-memory schema
is updated synchronously with system space updates, i.e. on_replace, not
on_commit.

Note, to do this in case of the sequence cache, we have to rework the
way sequences are exported to Lua - make on_alter_sequence similar to
how on_alter_space and on_alter_func triggers are implemented.

To implement transactional DDL, we must make sure that in-memory schema
is updated synchronously with system space updates, i.e. on_replace, not
on_commit.

To implement transactional DDL, we must make sure that in-memory schema
is updated synchronously with system space updates, i.e. on_replace, not
on_commit.

See also commit 22bedebe ("ddl: synchronize privileges cache with
actual data state").

Do it on_replace rather than on_commit. This is required to implement
transactional DDL.

Note, this is the only place where on_replace_dd_space() postpones
schema update until after commit. Other than that, space updates are
already transactional DDL friendly.

Replace prints that indicate on_shutdown trigger execution with
log.warn, which is more reliable. This eliminates occasional test
failures. Also instead of waiting for the server to start and executing
grep_log, wait for the desired log entries to appear with wait_log.

Closes #4134

Fixed app/strict.test.lua fail after a test that sets 'a' global
variable. It was due to the way how 'strict' module tracks declared
global variables and was fixed in pretest_clean.lua test-run's module.

A new VARBINARY field type would be useful for SQL type system.

Closes #4201
Needed for #4206

@TarantoolBot document
Title: new varbinary field type

Introduced a new field type varbinary to represent mp_bin values.
The new type varbinary may be used in format or index definition.

Example:
s = box.schema.space.create('withdata')
s:format({{"b", "varbinary"}})
pk = s:create_index('pk', {parts = {1, "varbinary"}})

Fixes commit abf5ef4f ("box: export registered functions in box.func
folder").

* Implement SQL driver (#4123).
* Support line carrying with backslash.
* Support new result file format (for new tests).

See the link below for a full description of the new features.

https://github.com/tarantool/test-run/commit/a04b5b096c607172ce4fc86a84e3531c9f3a7304

Fixes #4123.

Generation is supposed to be a persistent counter to distinguish
between different installations of the same SWIM instance. By
default it was set to 0, which was quite unsafe.

Kostja proposed an easy and bright solution - generation could be
set to timestamp by default. In such a case on each restart it is
almost 100% will be different.

Follow up #4280

swim.new() is declared as allowed to be called before swim:cfg().
But in fact swim.new({generation = ...}) didn't work because
after generation extraction the empty config {} was passed to
swim:cfg() and led to an error.

The patch allows to call swim.new() with generation only, as well
as without parameters at all.

Follow up #4280

FFI can't survive yields. A yield in ffi.C.func() leads to a
crash; yield in ffi.gc is not documented as allowed. Yield in any
GC function leads to garbage collector stuck until the yield is
finished.

This patch makes SWIM GC callback non-yielding. Now yielding
swim_delete() is called in a separate fiber created in GC
callback, but started at the end of event loop only.

Follow up #3234

SWIM wraps user triggers to prepare arguments. The wrapper
function kept a reference to SWIM object, and prevented its
automatic deletion at GC.

The patch makes this reference weak.

Follow up #4250

In SQL, it is allowed to use vector expressions, that is, an
operation that uses vectors as operands. For instance, vector
comparison:
SELECT (1,2,3) < (1,2,4);

Accidentally, routines handling IN operator contained a bug: in
cases where we used a vector as the left value in the IN operator,
we received an assertion in debug build or a segmentation fault in
release. This was due to some legacy code in which it was assumed
that the left value of the IN operator can have only one column in
case it is a vector. Let's fix this by allowing vectors of the
other sizes as the left value of the IN operator and providing
check which verifies that both sides of IN operator have the same
dimension.

Closes #4204

This test fails sporadically

This patch optimizes some netbox requests that were slowed down in
commit 665fc3a1.

Follow-up #2978

Applier use asynchronous transaction to batch journal writes. All
appliers share the replicaset.applier.tx_vclock which means the vclock
applied but not necessarily written to a journal. Appliers use a trigger
to coordinate in case of failure - when a transaction is going to
be rolled back.

Closes: #1254

This commit implements asynchronous transaction processing using
txn_write. The method prepares a transaction and sends it to an journal
without an yield until the transaction was finished. The transaction
status could be controlled via on_commit/on_rollback triggers.
In order to support asynchronous transaction journal_write method turned
to an asynchronous one and now a transaction engine controls journal status
using journal entry finalization callback.

Prerequisites: #1254

Finalize a transaction thorough a journal entry callback. So transaction
processing doesn't rely on fiber schedule.
Also allow to steal locked latch ownership for fiber which isn't owner
of the latch. This is required to process transaction triggers
asynchronously.

Prerequisites: #1254

Refactoring: don't touch a fiber gc storage on a transaction rollback
explicitly. This relaxes dependencies between fiber and transaction
life cycles.

Prerequisites: #1254

Move transaction auto start and auto commit behavior to the box level.
From now a transaction won't start and commit automatically without
txn_begin/txn_commit invocations. This is a part of a bigger transaction
refactoring in order to implement detachable transactions and a parallel
applier.

Prerequisites: #1254

Refactoring: put txn statement unref code into transaction free function.

Do not spread the space:truncate() to replicas if the
affected space is local or temporary.

Closes #4263

This reverts commit 4e3470ce.

The RFC did not pass review yet.

One of subtests was checking if crypto_decode returns an error
when fails to decode. But due to randomness of the test sometimes
it happened, that initial vector of encrypted data somehow didn't
lead to an error. Decryption was not correct, but only in terms
of result, not in terms of decryption algorithm. -1 was not
returned, and diag was not set.

This patch checks all the cases.

Closes #4306

Closes #4182
Needed for #1260

@TarantoolBot document
Title: Persistent Lua functions

Now Tarantool supports 'persistent' Lua functions.
Such functions are stored in snapshot and are available after
restart.
To create a persistent Lua function, specify a function body
in box.schema.func.create call:
e.g. body = "function(a, b) return a + b end"

A Lua persistent function may be 'sandboxed'. The 'sandboxed'
function is executed in isolated environment:
  a. only limited set of Lua functions and modules are available:
    -assert -error -pairs -ipairs -next -pcall -xpcall -type
    -print -select -string -tonumber -tostring -unpack -math -utf8;
  b. global variables are forbidden

Finally, the new 'is_deterministic' flag allows to mark a
registered function as deterministic, i.e. the function that
can produce only one result for a given list of parameters.

The new box.schema.func.create interface is:
box.schema.func.create('funcname', <setuid = true|FALSE>,
	<if_not_exists = true|FALSE>, <language = LUA|c>,
	<body = string ('')>, <is_deterministic = true|FALSE>,
	<is_sandboxed = true|FALSE>, <comment = string ('')>)

Example:
lua_code = [[function(a, b) return a + b end]]
box.schema.func.create('sum', {body = lua_code,
		is_deterministic = true, is_sandboxed = true})
box.func.sum
---
- is_sandboxed: true
  is_deterministic: true
  id: 2
  setuid: false
  body: function(a, b) return a + b end
  name: sum
  language: LUA
...
box.func.sum:call({1, 3})
---
- 4
...

The box_lua_find routine used to work with an empty stack only.
It is unacceptable in following patches because this helper
need to be reused in following patches for environment table
construction.

Needed for #4182, #1260

Needed for #4182, #1260

@TarantoolBot document
Title: Export registered functions to box.func folder

Now all registered with box.schema.func.create functions are
exported in box.func folder.
Each function have :call and :drop method. The :drop method
just a shortcut for box.schema.func.drop interface.
The :call method is similar to net.box connection:call method
and allows to call a registered function directly.
All access checks are performed on each function call.

Example:
function sum(a, b) return a + b end
box.schema.func.create('sum')
box.func.sum
---
- language: LUA
  setuid: false
  name: sum
  id: 2
...
box.func.sum:call({1, 3})
---
- 4
...
box.func.sum:drop()

The function func object used to provide a call method only for
C functions. In scope of this patch it reworked to be a uniform
function call frontend both for C and Lua functions.

Introduced classes func_c and func_lua, that provide own
constructors which produce implementation-specific object with
call and destroy methods.

Needed for #4182, #1260

Re-factor box_lua_call and box_lua_eval so that they don't take
call_request. This approach is more scalable: in case of a
functional index, the user expects to see a tuple with field
names so we should be able to pass not only raw msgpack, but
also a tuple to a Lua call so we need an universal way to pass
arguments to _call methods.

To pass a tuple msgpack introduced a new port_msgpack: the port
class with dump_lua method.
A new method get_msgpack returns a content of a port as a
msgpack data. The lifecycle of the returned value is
implementation-specific: it may either be returned directly from
the port, in which case the data will stay alive as long as the
port is alive, or it may be allocated on the fiber()->gc, in
which case the caller is responsible for cleaning up.

Needed for #4182, #1260

Tarantool used to assume that func_new call must not fail and
it used to build a new func object by given definition just on
func cache replace operation. We need to fix it to perform
user-dependent risky actions like Lua function assemble in
further patches.

The replace method is disallowed for _func space because it is
redundant and difficult to maintain in case of functions that
have pre-compiled runtime.

Needed for #4182, #1260