In the issue #3290 the important problem appeared - Tarantool can
not create completely internal collations with no ID, name,
owner. Just for internal usage.

Original struct coll can not be used for this since
* it has fields that are not needed in internals;
* collation name is public thing, and the collation cache uses
  it, so it would be necessary to forbid to a user usage of some
  system names;
* when multiple collations has the same comparator and only their
  names/owners/IDs are different, the separate UCollator objects
  are created, but it would be good to be able to reference a
  single one.

This patch renames coll to coll_id, coll_def to call_id_def and
introduces coll - pure collation object with no any user defined
things.

Needed for #3290.

A follow up for the patch for gh-3051

The format was lost when performing update operations as new tuple
was referenced to default runtime format.

Fixes #3051

Tuple has been checked instead of new_tuple returned as
box_tuple_update result.

No other changes.

No semantical changees.

No semantical changes.

Small refactoring: remove 'enum replica_state' since reuse a subset
from applier state machine 'enum replica_state' to check if we have
achieved replication quorum and hence can leave read-only mode.

Now netbox decodes iproto packet body twice: first, MessagePack -> Lua,
then Lua -> box.tuple.
Let's convert MessagePack to box.tuple objects without a round
trip to Lua. This should reduce the amount of garbage produced by
net.box.

Part of #3333

Improve the test by decreasing range_size so that it creates a lot of
ranges for test indexes, not just one. This helped find bugs causing
the crash described in #3393.

Follow-up #3393

Although the bug in vy_task_dump_complete() due to which a tuple could
be lost during dump was fixed, there still may be affected deployments
as the bug was persisted on disk. To avoid occasional crashes on such
deployments, let's make vinyl_iterator_secondary_next() skip tuples that
are present in a secondary index but missing in the primary.

Closes #3393

vy_task_dump_complete() creates a slice per each range overlapping with
the newly written run. It uses vy_range_tree_psearch(min_key) to find
the first overlapping range and nsearch(max_key) to find the range
immediately following the last overlapping range. This is incorrect as
nsearch rb tree method returns the element matching the search key if it
is present in the tree. That is, if the max key written to a run turns
out to be equal the beginning of a range, the slice won't be created for
it and it will be silently and persistently lost.

The issue manifests itself as crash in vinyl_iterator_secondary_next(),
when we fail to find the tuple in the primary index corresponding to a
statement found in a secondary index.

Part of #3393

vy_run_iterator_seek() is supposed to check that the resulting statement
matches the search key in case of ITER_EQ, but if the search key lies at
the beginning of the slice, it doesn't. As a result, vy_point_lookup()
may fail to find an existing tuple as demonstrated below.

Suppose we are looking for key {10} in the primary index which consists
of an empty mem and two runs:

    run 1: DELETE{15}
    run 2: INSERT{10}

vy_run_iterator_next() returns DELETE{15} for run 1 because of the
missing EQ check and vy_point_lookup() stops at run 1 (since the
terminal statement is found) and mistakenly returns NULL.

The issue manifests itself as crash in vinyl_iterator_secondary_next(),
when we fail to find the tuple in the primary index corresponding to a
statement found in a secondary index.

Part of #3393

Follows up #3396.

When a connection is closed, some of long-poll requests still may
by in TX thread with non-discarded input. If a connection is
closed, and then an input is discarded, then connection must not
try to read new data.

The bug was introduced here:
f4d66dae by me.

Closes #3400

In order to make CREATE/ALTER/DROP ACLs available to users other than
admin, make sure utility functions in schema.lua do not require
excess privileges to execute DDL.

Right now many of schema.lua functions work directly with system
spaces to construct a DDL statement or prepare a nice error message
for the user. Siwtch to virtual system views wherever possible.
System views for system spaces are available to role 'public'
and display the same data as system spaces, but filtered through
the user-specific access lense: i.e. they display only rows which
should be visible to the effective user.

Update test results, since now some error messages slightly differ.
Extend test coverage.

In scope of gh-3250 "Make grant/revoke not require accesses to universe"

It prevents rewriting result by an another thread after coio_call(), but
before lua_pushlstring(). Such case is possible because libeio uses
thread pool internally and static __thread storage can be reused before
lua_pushlstring() if many parallel digest.pbkdf2() calls are on the fly.

Fixes #3396.

Introduce _vsequence system space.

Prerequisite of #3250

Fix a regression introduced in the prevoius patch.
Dereference a deleted collation on commit, otherwise it leaks.

A collation is completely defined by a signature consisting of
collation properties: locale, ICU strength, normalization mode
etc.

When two collations have the same signature, they work identically.
Create a second collation cache, in which collations are stored
by signature instead of id. To reference the same collation in the
signature cache multiple times from id/name cache, introduce
collation reference counting.

Forbid update in _collation system space. It leads to an assertion
failure in collation cache, since it has no support for replace
operation.
Refactor collation deletion and insertion in alter.cc.

Besides, collation update potentially changes order of data
in the index and did not trigger rebuild of all indexes.

Lock the latch after allocating on_commit/on_rollback triggers
to avoid being stuck with a locked latch in case of OOM error.

Add a comment.

box/net.box.test creates a pair of long-poll requests, which are
not finalized then. If the next test appears to be
box/net_msg_max.test.lua, then it hangs since it expects, that
nobody occupies tx fiber pool.

When a netbox state machine is in 'fetch_schema' state, an async
request must not
* raise an error, because it is not 'error' state;
* wait for 'active' state, because async request must no wait
  anything.

Follow up #3107

Schema_version was used in netbox to update local box-like
schema. The box-like schema makes able to access spaces and
indexes via connection object.

It was updated each time, when a response from a server is
received with a schema version non-equal to the local value.

But there was no reason why a schema version is needed in a
request. It leads to ER_WRONG_SCHEMA_VERSION error sometimes,
but netbox on this error just resends the same request again. The
same behaviour can be reached with just no sending any schema
version to a server.

Remove schema_version from request, and just track schema version
changes in responses.

Part of #3351
Part of #3333
Follow up #3107

Now any netbox call blocks a caller-fiber until a result is read
from a socket, or time is out. To use it asynchronously it is
necessary to create a fiber per request. Sometimes it is
unwanted - for example if RPS is very high (for example, about
100k), and latency is about 1 second. Or when it is neccessary
to send multiple requests in parallel and then collect responses
(map-reduce).

The patch introduces a new option for all netbox requests:
is_async. With this option any called netbox method returns
immediately (but still yields for a moment) a 'future' object.

By a future object a user can check if the request is finalized,
get a result or error, wait for a timeout, discard a response.

Example of is_async usage:
future = conn:call(func, {params}, {..., is_async = true})
-- Do some work ...
if not future.is_ready() then
    result, err = future:wait_result(timeout)
end
-- Or:
result, error = future:result()

A future:result() and :wait_result() returns either an error or
a response in the same format, as the sync versions of the called
methods.

Part of #3107

Netbox has a table 'method_codec' that is used to encode a
request by a method name. But a response is decoded out of codec.
It leads to
1) decoding into Lua tables before decoding into tuples where
needed - it is double decoding and produces a lot of garbage;
2) each method contains hacks like one_tuple(), or single tuple
check.

These things can not be fixed with no real codec instead of
encoder only.

Also global table with decoders is needed for #3107, where
a request could be sent async with no fiber blocking. An async
response when received already does not have a call context - it
has only method name.

Needed for #3107

It is needed to return error via 'nil, error_object' notation,
and to store an error object to return it later.

Closes #3031

It did not work because raise is implemented as __index metatable
member, and error() is __call metatable member. The second one
takes additional implicit argument - self.

And it is not documented, so can be removed.

This commit allows userland to extend the space and index
metatables with their own functions or even metamethods. Reducing
barriers for this kind of experimentation is vital for user
contribution toward the improvement of Tarantool's API.

There are 4 metatables available for extending:
box.schema.space_mt - metatable of all spaces;
box.schema.index_mt - base metatable of all indexes - replicated
                      into the vinyl and memtx. See below how.
box.schema.vinyl_index_mt - metatable of all vinyl indexes;
box.schema.memtx_index_mt - metatable of all memtx indexes.

On the other hand local space/index metatables still can be
extended individually to save compatibility with existing
modules. Routinely space/index metatable is just a proxy for a
global mt. When a user attempts to extend a space or index
methods via local space/index metatable instead of from
box.schema mt, the local metatable is transformed. Its __index
metamethod starts looking up at first in self, and only then into
the global mt.

Closes #3204

Now space.bless() in Lua serves to choose a correct index
metatable that depends on a space engine. Vinyl index methods
must not use FFI since yield breaks it. Lets do not choose
correct methods one by one in space.bless, create them only
once on start, and then just do copy of needed table.

Function vy_apply_upsert_opts originaly appears in this commit:
5627e53b, where it is a
refactored version of a sophia upsertion. But when a vy_stmt was
introduced, vinyl_apply_upsert_ops works just like ordinary
tuple_upsert_execute. Remove this useless wrapper.

Space_mt and index_mt are created individually for each space
inside a giant space.bless function. It makes impossible to
implement #3204: exposing space and index metatables to a user,
because for this they must be global and shared between all
spaces and indexes.

Lets move their definition out of space.bless function, and do
their duplicate inside.

Needed #3204

Before the patch on any space alter Tarantool recreates its
indexes leaving old index objects be invalid. To protect a user
from errors about accessing outdated index object fields lets
update it in place instead of creating a new one.

Closes #3285