A txn_stmt keeps a reference to the space it modifies. Memtx uses this
space reference to revert the statement on error or voluntary rollback
so the space must stay valid throughout the whole transaction.

The problem is a DML statement may be followed by a DDL statement that
modifies the target space in the same transaction. If we try to roll
it back before running the rollback triggers installed by the DDL
statement, it will access an invalid space object (e.g. missing an
index), which will result in a crash.

To fix this problem, let's run triggers installed by a statement right
after rolling back the statement.

Closes #4368

We must un-account index.stat.txw.rows not only when a whole transaction
is rolled back, but also when we undo statements using a savepoint.

After 2.2.0-390-ga7c855e5b ("net.box: fetch '_vcollation' sysview into
the module") net.box fetches _vcollation view unconditionally, while the
view was added in 2.2.0-389-g3e3ef182f and, say, tarantool-1.10 and
tarantool-2.1 do not have it. This leads to a runtime error "Space '277'
does not exist" on a newer client that connects to an older server.

Now the view is fetched conditionally depending of a version of a
server: if it is above 2.2.1, then net.box will fetch it. Note: at the
time there are no release with a number above 2.2.1.

When _vcollation view is available, a collation in an index part will be
shown by its name (with 'collation' field), otherwise it will be shown
by its ID (in 'collation_id' field). For example:

Connect to tarantool 1.10:

 | tarantool> connection = require('net.box').connect('localhost:3301')
 | ---
 | ...
 |
 | tarantool> connection.space.s.index.sk.parts
 | ---
 | - - type: string
 |     is_nullable: false
 |     collation_id: 2
 |     fieldno: 2
 | ...

Connect to tarantool 2.2.1 (when it will be released):

 | tarantool> connection = require('net.box').connect('localhost:3301')
 | ---
 | ...
 |
 | tarantool> connection.space.s.index.sk.parts
 | ---
 | - - type: string
 |     is_nullable: false
 |     collation: unicode_ci
 |     fieldno: 2
 | ...

Fixes #4307.

Before this patch it was allowed to specify REAL/FLOAT/DOUBLE types
which matched with NUMBER NoSQL type in space format. However, NUMBER is
different from standard floating point types, since it is able to hold
integers in range [-2^63; 2^64-1] alongside with double precision
floating point values. Hence, to not confuse users it has been decided
to remove support of REAL/FLOAT/DOUBLE types from SQL grammar and use
instead original NUMBER type naming.

TEXT type is called "string" in the original Tarantool NoSQL, so it would
be rational to allow using the same type name in SQL.

When reverting to a savepoint inside a DDL transaction, apart from
undoing changes done by the DDL statements to the system spaces, we also
have to

 - Run rollback triggers installed after the savepoint was set, because
   otherwise changes done to the schema by DDL won't be undone.
 - Remove commit triggers installed after the savepoint, because they
   are not relevant anymore, apparently.

To achieve that let's append DDL triggers right to txn statements.
This allows us to easily discard commit triggers and run rollback
triggers when a statement is rolled back.

Note, txn commit/rollback triggers are not removed, because they are
still used by applier and Lua box.on_commit/on_rollback functions.

Closes #4364
Closes #4365

It's not used anywhere anymore.

txn_stmt_new is supposed to simply allocate an initialize a new txn_stmt
struct. Adding the new statement to the txn's statement list and setting
up a savepoint for rollback looks confusing. Move it to txn_begin_stmt.

Added the following checks:

* opts.headers is a table.
* opts.headers keys are strings.

Clarified an error message re Lua type of opts.headers values.

Found and fixed a memory leak that appears in http client when invalid
opts.headers is passed.

Closes #4281

Commit triggers must be run in the same order they are added, see commit
01343264 ("txn: fix execution order of commit triggers"). To achieve
that we added a new trigger method, trigger_add_tail(), which adds new
triggers to the trigger list tail rather than to the head, and now we
use this new method for adding commit triggers.

Come to think of it now, that solution was rather confusing. First,
commit triggers are still added to the head of the list from Lua.
Second, to revert triggers on rollback-to-savepoint it would be really
more convenient to have both commit and rollback trigger lists have the
same order.

So this patch reverts the above-mentioned commit and instead simply
uses a reverse iterator to run commit triggers.

To bring new rlist methods - rlist_foreach_entry_safe_reverse and
rlist_cut_before. Also, remove unit/rlist test as it is now a part
of the small suite.

Update test output.

The uninitialized variable found that caused the fail on build,
due to LTO builds with flag to check warnings as errors. Set the
variable to 0 initial value. The found issue was:

[100%] Built target xrow.test
src/box/sql/vdbe.c: In function ‘sqlVdbeExec’:
src/box/sql/vdbe.c:3691:11: error: ‘id’ may be used uninitialized
	in this function [-Werror=maybe-uninitialized]
  uint64_t id;
           ^
lto1: all warnings being treated as errors
lto-wrapper: fatal error: /usr/bin/c++ returned 1 exit status
compilation terminated.
/usr/bin/ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status

Closes #4378

Increased the maximum number of open file descriptors on macOS.

Travis-ci: needed the "ulimit -n <value>" call, because found
that the tests fail without it even with launchctl limit tool set.

Gitlab-ci: needed the "launchctl limit maxfiles <value>" call,
because under gitlib-ci it is needed the password to change the
limits and we call the sudo tool which separates the local set
of the environment.

Closes #4373

Closes #1260

@TarantoolBot document
Title: introduce func indexes in memtx
Now you can define a func_index using a registered persistent
function.

There are restrictions for function and key definition for
a functional index:
 - the referenced function must be persistent, deterministic
   and must return a scalar type or an array.
 - you must define key parts which describe the function return value
 - the function must return data which types match the
   defined key parts
 - the function may return multiple keys; this would be a multikey
   functional index; each key entry is indexed separately;
 - for multikey functional indexes, the key definition should
   start with part 1 and cover all returned key parts
 - key parts can't use JSON paths.
 - the function used for the functional index can not access tuple
   fields by name, only by index.

Functional index can't be primary.
It is not possible to change the used function after a functional
index is defined on it. The index must be dropped first.

Each key returned by functional index function (even when it is a
single scalar) must be returned as a table i.e. {1} and must
match the key definition.

To define a multikey functional index, create a function with
opts = {is_multikey = true} and return a table of keys.

Example:
s = box.schema.space.create('withdata')
s:format({{name = 'name', type = 'string'},
          {name = 'address', type = 'string'}})
pk = s:create_index('name', {parts = {1, 'string'}})
lua_code = [[function(tuple)
                local address = string.split(tuple[2])
                local ret = {}
                for _, v in pairs(address) do
			table.insert(ret, {utf8.upper(v)})
		end
                return ret
             end]]
box.schema.func.create('address', {body = lua_code,
                       is_deterministic = true, is_sandboxed = true,
                       opts = {is_multikey = true}})
idx = s:create_index('addr', {unique = false,
                     func = 'address',
                     parts = {{1, 'string', collation = 'unicode_ci'}}})
s:insert({"James", "SIS Building Lambeth London UK"})
s:insert({"Sherlock", "221B Baker St Marylebone London NW1 6XE UK"})
idx:select('Uk')
---
- - ['James', 'SIS Building Lambeth London UK']
  - ['Sherlock', '221B Baker St Marylebone London NW1 6XE UK']
...

Introduced a new object tuple_chunk: a memory allocation is
associated with given tuple. tuple_format's vtab is extended
with few new methods to manage tuple_chunks lifecycle.
Implemented corresponding methid for memtx engine: a memory
chunks are allocated with memtx's smalloc allocator.

Needed for #1260

Needed for #1260

@TarantoolBot document
Title: A new option is_multikey for function definition

A new option is_multikey allows to specify wether new function
returns multiple values packed in a table object. This is a
native way to define multikey func_index.

The problem was that a tagged revision is not deployed, so after a
release we did an empty commit to trigger deployment. Now it is worked
around by adding extra deployment rules that deploys tagged revisions.
The workaround was suggested by Hiro Asari in [1].

[1]: https://github.com/travis-ci/travis-ci/issues/7780#issuecomment-302389370

Fixes #3745.

Due to the new 7.65.3 curl version released on 2019-07-19,
removed temporary workaround that downgraded the curl to
7.65.0.

This reverts commit 2e880af0.

Follows up #4288

Update the signature of space_build_index wrapper instead.

If any sub-set of fields indexed by a unique index is indexed by another
unique index, we can skip the uniqueness check for it. We use this to
optimize out unnecessary uniqueness checks in vinyl, where they may be
pretty costly, especially if the bloom filter doesn't reflect them.

Currently, whether to check if an index is unique or not is determined
in vinyl space constructor, which sets vy_lsm::check_is_unique flag for
the space indexes. This looks ugly, because this means that no other
engine can make use of this optimization without duplicating the code
setting up the flags. True, no other engine needs it now, but still it
doesn't feel right. Besides, the check_is_unique flag isn't actually an
index property - it's rather a property of a space so storing it in
vy_lsm looks wrong. Because of that we have to update the flag when an
index is reassigned to another space by MoveIndex DDL operation, see
vinyl_space_swap_index().

So let's store the flags indicating whether a uniqueness check is
required for a particular index in a bitmap in the space struct and
set it up in the generic space constructor, space_create().

When an index definition is modified by DDL in such a way that doesn't
require index rebuild (e.g. a key part type is changed from unsigned to
integer), we move the index from the old space container to the new one,
see ModifyIndex. In case of Vinyl we also need to update key definitions
and options stored in vy_lsm. We do that in swap_index space method, but
this isn't entirely correct, as this method is also called when an index
is moved without any modifications, see MoveIndex. Let's do this from
update_def index method instead, which seems to be more suitable.

The only reason this code lives in swap_index space method now is that
we didn't have update_def index method when this functionality was
introduced in the first place.

We may write to txn->sub_stmt_begin[TXN_SUB_STMT_MAX] so the array size
must be TXN_SUB_STMT_MAX+1 (see txn_begin_stmt). This didn't lead to any
problems, because we would only overwrite txn::signature, which wouldn't
break anything. However, should we change the txn struct, we could get
an unexpected error or even a crash.

Function memtx_tree_index_build_array_deduplicate worked
incorrectly: when build_array is empty it changed it size to 1.

Follow up #1257
Needed for #1260

It is convential to prefix boolean predicates with _is_.

Renamed memtx_tree_delete_identical to memtx_tree_delete_value
because it is a more descriptive name. Changed its signature
to return a deleted item, because this item may require an own
destruction.
This refactoring is required in scope of functional indexes.

Needed for #1260

Updated multikey memtx_tree methods a bit to make possible to
reuse them in case of functional indexes. The BPS_TREE_IDENTICAL
is only used in bps_tree_delete_identical (when BPS_TREE_NO_DEBUG
macro is defined) therefore it may be reused to delete an
identical entry in case of functional index in further patches.

Needed for #1260

Build serpent using a symlink to not clog third party with build
artefacts.

Fix a typo in naming, use a bit more verbose names..

@TarantoolBot document
Title: > require('console').set_default_output("lua")

A user might need to use lua output as default serializer so instead
of requiring him setting up the output every new session that named
default output mode may be used instead with help of command

Also we provide

 > require('console').get_default_output()

to obtain current setting.

Part-of #3834

Without this option serpent tries to encode function body,
making output a plain mess. So just like for yaml output
where functions are represented as

 | "completion_handler: 'function: 0x4074c910'

for lua output we will have

 | completion_handler = function() --[[..skipped..]] end,

Part-of #3834

@TarantoolBot document
Title: document \set output lua

Historically we use YAML format to print results of operation to
a console. Moreover our test engine is aiming YAML as a primary format
to compare results of test runs. Still we need an ability to print
results in a different fasion, in particular one may need to use
the console in a REPL way so that the results would be copied and
pased back to further processing.

For this sake we introduce that named "output" command which allows
to specify which exactly output format to use. Currently only yaml
and lua formats are supported.

To specify lua output format type

 | tarantool> \set output lua

in the console. lua mode supports line oriented output (default) or
block mode.

For example

 | tarantool> a={1,2,3}
 | tarantool> a
 | ---
 | - - 1
 |   - 2
 |   - 3
 | ...
 | tarantool> \set output lua
 | true
 | tarantool> a
 | {1, 2, 3}
 | tarantool> \set output lua,block
 | true
 | tarantool> a
 | {
 |   1,
 |   2,
 |   3
 | }

By default YAML output format is kept for now, simply to not
break the test engine. The output is bound to a session, thus every
new session should setup own conversion if needed.

Since serializing lua data is not a trivial task we use "serpent"
third party module to convert data.

Part-of #3834

We will use it for Lua console output format serialization

Part-of #3834

In contrast to subsequent calls, the initial call to box.cfg didn't log
configuration changes to the default state. As a result, by looking at
a log file we coudn't tell which configuration was being used.

Closes #4236

If CHECK constraint is added using ALTER TABLE statement, row_count
should be incremented. Note that row_count does not increase if CHECK
constraint is created during the execution of CREATE TABLE statement.
For example:

box.execute('CREATE TABLE t1(id INTEGER PRIMARY KEY);')
box.execute('ALTER TABLE t1 ADD CONSTRAINT ck1 CHECK(id > 0);')

Should return:
- row_count: 1

However, prior to this patch it was always equal to zero. Let's fix
this passing appropriate flag to OP_SInsert opcode.

Closes #4363

After we fixed bundled libyaml to correctly print 4-byte Unicode
characters, it is no longer compatible with the upstream version, so
enable building with bundled libyaml for every platform.
This way the tests will pass.

Follow-up #4090