We tend to synchronize cached data with the actual data changes: apply
while on_replace and undo while on_rollback.

Fire transaction trigger after a transaction finalization. This allows
to not to view the transaction dismissed changes in case of rollback.

Fixes: #4276

SWIM as a monitoring module is hard to use without an ability to
subscribe on events. Otherwise a user should have polled a SWIM
member table for updates - it would be too inefficient.

This commit exposes an ability to set Lua triggers.

Closes #4250

@TarantoolBot document
Title: SWIM: swim:on_member_event

Now a user can set triggers on member table updates. There is a
function for that, which can be used in one of several ways:
```Lua
swim:on_member_event(new_trigger[, ctx])
```
Add a new trigger on member table update. The function
`new_trigger` will be called on each new member appearance, an
existing member drop, and update. It should take 3 arguments:
first is an updated SWIM member, second is an events object,
third is `ctx` passed as is.

Events object has methods to help a user to determine what event
has happened.
```Lua
local function on_event(member, event, ctx)
    if event:is_new() then
        ...
    elseif event:is_drop() then
        ...
    end

    if event:is_update() then
        -- All next conditions can be
        -- true simultaneously.
        if event:is_new_status() then
            ...
        end
        if event:is_new_uri() then
            ...
        end
        if event:is_new_incarnation() then
            ...
        end
        if event:is_new_payload() then
            ...
        end
    end
end

s:on_member_event(on_event, ctx)
```
Note, that multiple events can happen simultaneously. A user
should be ready to that. Additionally, 'new' and 'drop' never
happen together. But they can happen with 'update', easily.
Especially if there are lots of events, and triggers work too
slow. Then a member can be added and updated after a while, but
still does not reach a trigger.

A remarkable case is 'new' + 'new payload'. This case does not
correlate with triggers speed. The thing is that payload absence
and payload of size 0 are not the same. And sometimes is happens,
that a member is added without a payload. For example, a ping
was received - pings do not carry payload. In such a case the
missed payload is received later eventually. If that matters for
a user's application, it should be ready to that: 'new' does not
mean, that the member already has a payload, and payload size
says nothing about its presence or absence.

Second usage case:
```Lua
swim:on_member_event(nil, old_trigger)
```
Drop an old trigger.

Third usage case:
```Lua
swim:on_member_event(new_trigger, old_trigger[, ctx])
```
Replace an existing trigger in-place, with keeping its position
in the trigger list.

Fourth usage case:
```Lua
swim:on_member_event()
```
Get a list of triggers.

When drop or replace a trigger, a user should be attentive - the
following code does not work:
```Lua
tr = function() ... end
-- Add a trigger.
s:on_member_event(tr)
...
-- Drop a trigger.
s:on_member_event(nil, tr)
```
The last line, if be precise. This is because SWIM wraps user
triggers with an internal closure for parameters preprocessing.
To drop a trigger a user should save result of the first
on_member_event() call.

This code works:
```Lua
tr = function() ... end
-- Add a trigger.
tr_id = s:on_member_event(tr)
...
-- Drop a trigger.
s:on_member_event(nil, tr_id)
```

The triggers are executed one by one in a separate fiber. And
they can yield. These two facts mean that if one trigger sleeps
too long - other triggers wait. It does not block SWIM from doing
its routine operations, but block other triggers.

The last point to remember is that if a member was added and
dropped before its appearance has reached a trigger, then such
a member does not fire triggers at all. A user will not notice
that easy rider member.

These function are going to yield in scope of #4250, because
swim:new() will start a fiber, while swim:delete() cancels and
gives it a control.

Needed for #4250

SWIM is asynchronous by design. It means, that there are two
ways of getting updates of a member table and individual members:
polling and triggers. Polling is terrible - this is why libev
with select(), poll(), epoll(), kevent() have appeared. The only
acceptable solution is triggers.

This commit allows to set triggers on member updates.

Part of #4250

It is a miracle, but somehow it worked until I changed a couple
of places. Here objects stored in an rlist are freed, but not
deleted from the list. The list is reused after that.

The SWIM unit tests code with the fake events and time does lots
of forbidden things: it manually invokes pending watcher
callbacks; manages global time without a kernel; puts not
existing descriptors into the loop. These foul blows open the
gates to the full control over IO events, descriptors, virtual
time. Hundreds of virtual seconds pass in milliseconds in
reality, it makes SWIM unit tests fast despite complex logic.

All these actions does not affect the loop until yield. On yield
a scheduler fiber wakes up and

    1) infinitely generates EV_READ on not existing descriptors
       because a kernel considers them closed;

    2) manual pending callbacks invocation asserts, because it is
       not allowed for non-scheduler fibers.

To avoid these problems a new isolated loop is created, not
visible for the scheduler. Here the fake events library can rack
and ruin whatever it wants.

Needed for #4250

When a member is deleted, pointer at its struct tt_uuid becomes
invalid. This commit makes member:uuid() create a new tt_uuid
object, cached for subsequent calls.

Travis-ci failed on brew install python2 due to it was
already installed on OSX 10.13 Sierra, so it needs
to be installed without fails on its exists.

Follow up #4254

Up to this point, the is_temporary flag has been set for surrogate space
definitions used to transfer meta-information during compilation stage
of DML queries (CREATE TABLE/INDEX etc).

At some point, it became possible to use spaces created in Lua in
SQL statements. Since it is allowed to create temporary spaces in
Lua, not all temporary spaces in SQL are surrogate wrappers. To separate
real temporary spaces (i.e. which came from space cache) from surrogate
ones, is_ephemeral flag is now set in such spaces instead of
is_temporary. Note that there can't be mess between these flags since
ephemeral spaces can exist only during execution of VDBE bytecode.
Also note that flag is required only for debugging facilities.

Close #4139

Fixed swim headers in addition to commit:
88892f13 ('swim: fix build on FreeBSD')

Included unistd.h header to fix the following errors:

In file included from src/lib/swim/swim_transport_udp.c:31:
src/lib/swim/swim_transport.h:61:1:
    error: unknown type name 'ssize_t'; did you mean 'size_t'?
ssize_t
^~~~~~~
size_t
/usr/include/netinet6/in6.h:698:18: note: 'size_t' declared here
typedef __size_t        size_t;
                        ^
In file included from src/lib/swim/swim_transport_udp.c:31:
src/lib/swim/swim_transport.h:66:1:
    error: unknown type name 'ssize_t'; did you mean 'size_t'?
ssize_t
^~~~~~~
size_t
/usr/include/netinet6/in6.h:698:18: note: 'size_t' declared here
typedef __size_t        size_t;

Closes #4050

Enabled use_unix_sockets_iproto option to use unix sockets
iproto provides the new way to handle the problem with
'Address already in use' error. It lets test-run appoint
unix sockets for LISTEN environment variable values.

Before this change the TcpPortDispatcher was used to
eliminate the race condition when two workers trying to use
the same port: the idea was that each worker used its own
ports range. Really these ports could race with client ports
(from, say, net.box or replication), which typically didn't
use bind() and so bound to a random available port (despite
any dispatched ranges) and could produce 'Address already in
use' error.

Closes: #4008

Removed skip flag file to switch on the testing of the
http_client test. Enabled http_client test on OSX,
fixed missing of the python2 symlink. Removed the subtest
on '595 error return' from 'error' suite, due to it may
hang forever. To enable test on travis-ci reverted commit:

1d7285c4 ('Disable flaky http_client.test.lua')

Closes #4254

This is the regression from a7c855e5
(net.box: fetch '_vcollation' sysview into the module). The result file
was corrupted, because the test sometimes produces incorrect result in
the test case for on_connect / on_disconnect triggers. This issue is
tracked by #4273.

Follow up #3941.

Fetch "_vcollation" sysview to show collation name instead collation id.

Closes #3941

Add "_vcollation" sysview to read it from net.box. This
sysview is always readable, except when a user doesn't have
"public" role.

Needed for #3941

@TarantoolBot document
Title: box.space._vcollation

_vcollation is a system space that represents a virtual view.
The structure of its tuples is identical to that of _collation.
Tuples of this sysview is always readable, except when the user
doesn't have "public" role.

Follow-up #4196

It is legacy flag and isn't used anymore.

Before this patch, the existence of space was checked for the
CREATE TABLE or CREATE VIEW statements during the parsing. If the
space already exists, an error has been set and cleanup is
performed. But, if the statement contained 'IF NOT EXIST', the
cleanup was performed, but the error was not set. Meanwhile,
create_foreign_key() assumes that if create_table_def->new_space
is NULL, then we are dealing with ALTER TABLE statement. This in
turn false, since ctd->new_space is nullified also in case of
already existing table. This causes an assertion or a segmentation
fault when creating a foreign key during space creation.

This patch moves this check to VDBE. Parsing now is always
processed till the end as in case space doesn’t exist.

Closes #4196

After compacting runs, we first mark them as dropped (VY_LOG_DROP_RUN),
then try to delete their files unless they are needed for recovery from
the checkpoint, and finally mark them as not needed in the vylog
(VY_LOG_FORGET_RUN). There's a potential race sitting here: the problem
is the garbage collector might kick in after files are dropped, but
before they are marked as not needed. If this happens, there will be
runs that have two VY_LOG_FORGET_RUN records, which will break recovery:

  Run XX is forgotten, but not registered

The following patches make the race more likely to happen so let's
eliminate it by making the garbage collector the only one who can mark
runs as not needed (i.e. write VY_LOG_FORGET_RUN record). There will
be no warnings, because the garbage collector silently ignores ENOENT
errors, see vy_gc().

Another good thing about this patch is that now we never yield inside
a vylog transaction, which makes it easier to remove the vylog latch
blocking implementation of transactional DDL.

Closes #3691

@TarantoolBot document
Title: check constraint for Lua space

The check constraint is a type of integrity constraint which
specifies a requirement that must be met by tuple before it
is inserted into space. The constraint result must be predictable.
Expression in check constraint must be <boolean value expression>
I.e. return boolean result.

Now it is possible to create ck constraints only for empty space
having format. Constraint expression is a string that defines
relations between top-level tuple fields.
Take into account that all names are converted to an uppercase
before resolve(like SQL does), use \" sign for names of fields
that were created not with SQL.

The check constraints are fired on insertion to the Lua space
together with Lua space triggers. The execution order of
ck constraints checks and space triggers follows their creation
sequence.

Note: this patch changes the CK constraints execution order for
SQL. Previously check of CK constraints integrity was fired before
tuple is formed; meanwhile now they are implemented as NoSQL before
replace triggers, which are fired right before tuple insertion.
In turn, type casts are performed earlier than msgpack
serialization. You should be careful with functions that use
field types in your check constrains (like typeof()).

Consider following situation:
```
 box.execute("CREATE TABLE t2(id  INT primary key,
                              x INTEGER CHECK (x > 1));")
 box.execute("INSERT INTO t2 VALUES(3, 1.1)")
```
the last operation would fail because 1.1 is silently
cast to integer 1 which is not greater than 1.

To create a new CK constraint for a space, use
space:create_check_constraint method. All space constraints are
shown in space.ck_constraint table. To drop ck constraint,
use :drop method.

Example:
```
s1 = box.schema.create_space('test1')
pk = s1:create_index('pk')
ck = s1:create_check_constraint('physics', 'X < Y')
s1:insert({2, 1}) -- fail
ck:drop()
```

To perform ck constraints tests before insert or update space
operation, we use precompiled VDBE machine associated with
each ck constraint, that is executed in on_replace trigger.
Each ck constraint VDBE code consists of
1) prologue code that maps new(or updated) tuple via binding,
2) ck constraint code generated by CK constraint AST.

In case of ck constraint error the tuple insert/replace operation
is aborted and ck constraint error is handled as diag message.

Needed for #3691

This patch introduces a new system space to persist check
constraints. The format of the new system space is

_ck_constraint (space id = 364)
[<space id> UINT, <constraint name> STR,
 <is_deferred>BOOL, <language>STR, <code>STR]

A CK constraint is local for a space, so every pair
<space id, CK name> is unique
(it is also the PK in the _ck_constraint space).

After insertion into this space, a new instance describing check
constraint is created. Check constraint holds an exspression AST.
While space features some check constraints, it isn't allowed to
be dropped. The :drop() space method firstly deletes all check
constraints and then removes an entry from the _space.

Because the space alter, the index alter and the space truncate
operations cause space recreation process, a new
RebuildCkConstrains object is introduced. This alter object
compiles a new ck constraint object, replaces and removes
an existent instances atomically (but if the assembly of some
ck constraint object fails, nothing is changed).
In fact, in scope of this patch we don't really need to recreate
a ck_constraint object in such situations (it is enough to patch
space_def pointer in AST tree like we did it before, but we are
going to recompile a VDBE that represents ck constraint in
further patches, and that operation is not safe).

The main motivation for these changes is an ability to support
ADD CHECK CONSTRAINT operation in the future. CK constraints are
easier to manage as self-sustained objects: such change is
managed with atomic insertion(unlike the current architecture).

Finally, the xfer optimization is disabled now if some space have
ck constraints. In following patches this xfer optimisation
becomes impossible, so there is no reason to rewrite this code
now.

Needed for #3691

Refactored OP_Column instruction with a new vdbe_field_ref class.
The vdbe_field_ref is a reusable object that speeds-up field
access for given tuple or tuple data.
Introduced OP_Fetch opcode that uses vdbe_field_ref given as a
first argument. This opcode makes possible to perform binding
of a new tuple to an existent VDBE without decoding it's fields.

Needed for #3691

This preparatory refactoring is necessary to simplify the process
of introducing a new OP_Fetch statement in the next patch.
- got rid of useless sMem local variable
- got rid of useless payloadSize in VdbeCursor structure

Needed for #3691

A new sql_bind_ptr routine allows to bing a generic pointer to
VDBE variable. This change is required to pass a tuple_fetcher
representing a new tuple to the check constraint VDBE.

Needed for #3961

The sql_flags is a parser parameter that describes how to parse
the SQL request, determines general behaviour: like whether
foreign keys are handled as deferred or not etc. But now this
information is taken from the global user session object.

When we need to run the parser with some other parameters, it is
necessary to change global session object, which may lead to
unpredictable consequences in general case.
Introduced a new parser and vdbe field sql_flags which is
responsible for SQL parsing results.

Needed for #3691

The SQL_NullCallback flag is never set now so it redundant.
Let's get red if it because we are going to pass
user_session->sql_flags to parser and use it instead of
session instance.

Needed for #3691

After making memtx space format check non-blocking, move the appropriate
vinyl test case to engine suite. Introduce a new errinj,
ERRINJ_CHECK_FORMAT_DELAY, to unify the test case for both engines.

Follow-up #3976

Just like index build, space check in memtx stalls event loop for the
whole check time. Add occasional yields, and an on_replace trigger,
which checks format for tuples inserted while space format check is in
progress.

Follow-up #3976

@TarantoolBot document
Title: memtx now checks space format in background

There is no event loop stall when memtx engine checks space format
anymore. You may insert tuples into a space, while its new format is
being checked. If the tuples don't match the new format, format change
will be aborted.

Currently to determine current source path, user has to do it
by getting current source info from the stack and parse it manually

This commit adds couple little helpers for this task called
`debug.sourcefile()` and `debug.sourcedir()`,
which returns a path to the file being executed.

The path is relative to the current working directory.
If such path cannot be determined, '.' is returned (i.e. interactive
mode). There is an exception if you are running script with an absolute
path. In that case, absolute path will be returned too.

```bash
tarantool /absolute/path/to/the/script.lua
```

There are also handy shortcuts for that functions: `debug.__file__` and
`debug.__dir__`

@TarantoolBot document
Title: Document new debug helper methods

There are two more helpers in debug module `sourcefile()` and
`sourcedir()`. This two helpers returns relative paths to source file
and directory respectively.

There are also `debug.__file__` and `debug.__dir__` which are just handy
ways to call `sourcefile()` and `sourcedir()`.

It should be mentioned that it is only possible to determine real path
to the file, if the function was defined in a some lua file.
`loadstring` may be exception here, since lua will store whole string
in debug info.

There is also a possible pitfall because of tail call optimization, so
if the function `fn` is defined inside `myapp/init.lua` like this:

```lua
function fn()
    return debug.sourcefile()
end
```

It would not be possible to determine file path correctly because that
function would not appear in a stacktrace. Even worse, it may return
wrong result, i.e. actual path where "parent" function was defined.

To force the interpreter to avoit this kind of optimizations you may use
parenthesis like this:

```lua
function fn()
    return (debug.sourcefile())
end
```

Also both `sourcefile` and `sourcedir` functions have optional level
argument, which allows set the level of the call stack function should
examine for source path. By default, 2 is used.

Example:
```
[001] box/on_replace.test.lua                                         [ pass ]
[001] box/on_replace.test.lua                                         [ fail ]
[001]
[001] Test failed! Result content mismatch:
[001] --- box/on_replace.result	Tue Jun  4 10:44:39 2019
[001] +++ box/on_replace.reject	Tue Jun  4 10:44:50 2019
[001] @@ -4,7 +4,7 @@
[001]  -- test c and lua triggers: must return only lua triggers
[001]  #box.space._space:on_replace()
[001]  ---
[001] -- 0
[001] +- 2
[001]  ...
```
This happened because we forgot to remove triggers set on _space.
Clear them below the test case.

This commit fixes a regression introduced by commit
5ab0763b
(pass operation type to lua triggers)

When passing operation type to the trigger, we relied on a corresponding
xrow_header, which would later be written to the log. When before_replace
triggers are fired, there is no row connected to the txn statement yet,
so we faked one to pass operations. We, however, didn't account for
temporary spaces, which never have a row associated with a corresponding
txn stmt. This lead to a segfault on an attemt to use on_replace triggers
with temporary spaces.

Add a fake row for temporary spaces to pass operation type in on_replace
triggers and add some tests.

Closes #4266

This reverts commit 6600dc1b.

Pushed by mistake before all changes.

Since DDL is triggered by the admin, it can be deliberately initiated
when the workload is known to be low. Throttling it along with DML
requests would only cause exasperation in this case. So we don't apply
disk-based rate limit to DDL. This should be fine, because the
disk-based limit is set rather strictly to let the workload some space
to grow, see vy_regulator_update_rate_limit(), and in contrast to the
memory-based limit, exceeding the disk-based limit doesn't result in
abrupt stalls - it may only lead to a gradual accumulation of disk space
usage and read latency.

Closes #4238

Fixed swim headers in addition to commit

Closes #4050

Since we have implemented memtx background index build, the
corresponding vinyl test cases are now also suitable for memtx,
so move them to engine suite so that both engines are tested.
Also add some tests to check that an ongoing index build is aborted in
case a tuple violating unique constraint or format of the new index is
inserted.
Add some error injections to unify appropriate memtx/vinyl tests.

Closes #3976

Memtx index build used to stall event loop for all the build period.
Add occasional yields so that the loop is not blocked for too long.
Also make index build set on_replace triggers so that concurrent
replaces are also correctly handled during the build.

Part of #3976

@TarantoolBot document
Title: memtx indices are now built in background

Memtx engine no longer stalls the event loop during an index build.
You may insert tuples into a space while an index build is in progress:
the tuples will be correctly added to the new index. If such tuple
violates the new indexes unique constraint or doesn't match the new
index format, the index build will be aborted.

If a key isn't found in the tuple cache, we fetch it from a run file. In
this case disk read and page decompression is done by a reader thread,
however key lookup in the fetched page is still performed by the tx
thread. Since pages are immutable, this could as well be done by the
reader thread, which would allow us to save some precious CPU cycles for
tx.

Close #4257

To handle fiber cancellation during page read we need to pin all objects
referenced by vy_page_read_task. Currently, there's the only such
object, vy_run. It has reference counting so pinning it is trivial.
However, to move page lookup to a reader thread, we need to also
reference key def, tuple format, and key. Format and key have reference
counting, but key def doesn't - we typically copy it. Copying it in this
case is too heavy.

Actually, cancelling a fiber manually or on timeout while it's reading
disk doesn't make much sense with PCIE attached flash drives. It used to
be reasonable with rotating disks, since a rotating disk controller
could retry reading a block indefinitely on read failure. It is still
relevant to Network Attached Storage. On the other hand, NAS has never
been tested, and what isn't tested, can and should be removed. For
complex SQL queries we'll be forced to rethink timeout handling anyway.

That being said, let's simply drop this functionality.