We're going to use S3 compatible storage for Deb and RPM repositories
instead of packagecloud.io service. The main reason is that
packagecloud.io provides a limited amount of storage, which is not
enough for keeping all packages (w/o regular pruning of old versions).

Note: At the moment packages are still pushed to packagecloud.io from
Travis-CI. Disabling this is out of scope of this patch.

This patch implements saving of packages on an S3 compatible storage and
regeneration of a repository metadata.

The layout is a bit different from one we have on packagecloud.io.

packagecloud.io:

 | - 1.10
 | - 2.1
 | - 2.2
 | - ...

S3 compatible storage:

 | - live
 |   - 1.10
 |   - 2.1
 |   - 2.2
 |   - ...
 | - release
 |   - 1.10
 |   - 2.1
 |   - 2.2
 |   - ...

Both 'live' and 'release' repositories track release branches (named as
<major>.<minor>) and master branch. The difference is that 'live' is
updated on every push, but 'release' is only for tagged versions
(<major>.<minor>.<patch>.0).

Packages are also built on '*-full-ci' branches, but only for testing
purposes: they don't pushed anywhere.

The core logic is in the tools/update_repo.sh script, which implements
the following flow:

- create metadata for new packages
- fetch relevant metadata from the S3 storage
- push new packages to the S3 storage
- merge and push the updated metadata to the S3 storage

The script uses 'createrepo' for RPM repositories and 'reprepro' for Deb
repositories.

Closes #3380

This patch makes the INSTEAD OF DELETE trigger work for every row
in VIEW. Prior to this patch, it worked only once for each group
of non-unique rows.

Also, this patch adds tests to check that the INSTEAD OF UPDATE
trigger work for every row in VIEW.

Closes #4740

Revert "Free all slabs on region reset" commit.

Closes #4736

Update decNumber library to silence the build warning produced on too
long integer constant.

Test multireturn in lua output mode and lack of a parameter in '\set
output <...>' command.

Co-developed-by: Alexander Turenko <alexander.turenko@tarantool.org>
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
Reviewed-by: Alexander Turenko <alexander.turenko@tarantool.org>

In case if output format is not specified we should exit
with more readable error message.

Fixes #4638

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
Reviewed-by: Alexander Turenko <alexander.turenko@tarantool.org>

Currently we handle only first member of multireturn statement.
Fix it processing each element separately.

n.b.: While at this file add vim settings.

 | tarantool> \set output lua
 | true;
 | tarantool> 1,2,3,4
 | 1, 2, 3, 4;

Fixes #4604

Reported-by: Alexander Turenko <alexander.turenko@tarantool.org>
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
Reviewed-by: Alexander Turenko <alexander.turenko@tarantool.org>

Transaction adds a redo log for each statement. The log is an xrow
header. Some requests don't have a header (local requests), some
do (from remote client, from replication).

When a request had a header, it was written as is to WAL. But
requests from remote client have an xrow header, however barely
filled. Most of its fields are default values, usually 0.
Including group id. Indeed, remote clients should not care about
setting such deep system fields.

That led to a problem when a space had group id local (!= 0), but
it was ignored because in a request header from a remote client
the group id was default (== 0). On the summary, it was possible
to force Tarantool to replicate a replica-local space.

Now group id setting is server-authoritative. Box always sets it
regardless of what is present in an xrow header received from a
client.

Thanks Kostja Osipov (@kostja) for the diagnostics and the
solution.

Closes #4729

Accidentally, assert in sql_type_result() checking argument types was
changed in 64745b10, so that now it may fail even on correct values.
Let's revert this change.

Closes #4728

'pragma collation_list' uses _collation space, although user
may have no access to it. Thus, we replace it with the
corresponding view.

Closes #4713

Usually functions return pair `nil, err` and expected that err is string.
Let's make the behaviour of error object closer to string
and define __concat metamethod.

The case of error "error_mt.__concat(): neither of args is an error"
is not covered by tests because of #4723

Closes #4489

SQL prepared statement cache is implemented as two data structures: hash
table <stmt_id : pointer-to-metadata> and GC queue. The latter is
required to avoid workload spikes on session's disconnect: instead of
cleaning up memory for all session-local prepared statements, prepared
statements to be deleted are moved to GC queue. When memory limit for PS
is reached, all elements from queue are removed at once. If statement
traps to the GC queue it is assumed to be already dead. Accidentally,
change of occupied by PS cache takes place only after GC queue clean-up,
so correct size of PS cache is displayed only after GC cycles. Let's fix
this and account PS cache size change right after entry removal (i.e. at
the moment PS gets into GC queue).

Struct of type tuple_format is being passed as an argument to
tuple_format_unref() where it might be freed. On such occasion any
further references to format fields should not take place.

Acked-by: Cyrill Gorcunov <gorcunov@gmail.com>

Closes #4658

- If an optional argument is provided for
  space_object:frommap() (which is {table = true|false}),
  type match for first arguments is omitted, which is
  incorrect. We should return the result only after making
  sure it is possible to build a tuple.

- If there is a type mismatch, however, frommap() does not
  return nil, err as it is mentioned in the description, so we
  change it to be this way.

Closes #4262

Despite the lack of documentation, fio.mktree() was designed to work
similar to mkdir -p: it creates the directory along with it's parents
and doesn't complain about existing ones.

But this function was subject to a race if two different processes were
trying to create the same directory at the same time. It was caused by
the fact that directory existence check and its creation aren't atomic.

This patch fixes the race by impoving error handling: it's not an error
if directory exists, even if it was created by someone else and mktree
failed.

Related to https://github.com/tarantool/doc/issues/1063
Closes #4660

It appears due to improper conflict resolution after pushing the
following commits in the reverse order:

* 2b9ef8d1 lua: don't modify pointer type in msgpack.decode*
* 84bcba52 lua: keeping the pointer type in msgpackffi.decode()

Originally 84bcba52 (which should land first) fixes the msgpackffi
module and introduces the test_decode_buffer() function locally for the
msgpackffi test. Then 2b9ef8d1 fixes the msgpack module in the same
way, expands and moves the test_decode_buffer() function to
serializer_test.lua (to use in msgpack and msgpackffi tests both).

After changes made to push the commits in the reverse order, those
commits doing something weird around tests. However the resulting state
is different from the right one just in the dead function in
msgpackffi.test.lua.

Follows up #3926.

Currently tuple_object:update() does not check the length
of operation string and just takes the first character
after decoding. This patch fixes this problem.

Follow-up #3884

Calling tuple_object:update() with invalid argument number
yields 'Unknown UPDATE operation' error. Instead, we replace this
error with explicit "wrong argument number", mentioning which operation
failed, or poiniting out at invalid operation code.

Fixes #3884

This patch corrects the result of typeof() for double values.
Previously, it gave the type "number" in the case of a
floating-point number. Now it gives "double".

Follow-up #3812

socket.test had a number of flaky problems:
- socket readiness expectation & read timeouts
- race conditions on socket shutdown in emulation test cases
- UDP datagrams losses on mac os
- excessive random port searches
Now they are solved. 127.0.0.1 is now used instead of 0.0.0.0 or
localhost to prevent wrong connections where appropriate. Socket test
is not fragile anymore.

Closes #4426
Closes #4451
Closes #4469

To track current memory occupied by prepared statements and number of
them, let's extend box.info submodule with .sql statistics: now it
contains current total size of prepared statements and their count.

@TarantoolBot document
Title: Prepared statements in SQL

Now it is possible to prepare (i.e. compile into byte-code and save to
the cache) statement and execute it several times. Mechanism is similar
to ones in other DBs. Prepared statement is identified by numeric
ID, which are returned alongside with prepared statement handle.
Note that they are not sequential and represent value of hash function
applied to the string containing original SQL request.
Prepared statement holder is shared among all sessions. However, session
has access only to statements which have been prepared in scope of it.
There's no eviction policy like in any cache; to remove statement from
holder explicit unprepare request is required. Alternatively, session's
disconnect also removes statements from holder.
Several sessions can share one prepared statement, which will be
destroyed when all related sessions are disconnected or send unprepare
request. Memory limit for prepared statements is adjusted by
box.cfg{sql_cache_size} handle (can be set dynamically;

Any DDL operation leads to expiration of all prepared statements: they
should be manually removed or re-prepared.
Prepared statements are available in local mode (i.e. via box.prepare()
function) and are supported in IProto protocol. In the latter case
next IProto keys are used to make up/receive requests/responses:
IPROTO_PREPARE - new IProto command; key is 0x13. It can be sent with
one of two mandatory keys: IPROTO_SQL_TEXT (0x40 and assumes string value)
or IPROTO_STMT_ID (0x43 and assumes integer value). Depending on body it
means to prepare or unprepare SQL statement: IPROTO_SQL_TEXT implies prepare
request, meanwhile IPROTO_STMT_ID - unprepare;
IPROTO_BIND_METADATA (0x33 and contains parameters metadata of type map)
and IPROTO_BIND_COUNT (0x34 and corresponds to the count of parameters to
be bound) are response keys. They are mandatory members of result of
IPROTO_PREPARE execution.

To track statistics of used memory and number of currently prepared
statements, box.info is extended with SQL statistics:

box.info:sql().cache.stmt_count - number of prepared statements;
box.info:sql().cache.size - size of occupied by prepared statements memory.

Typical workflow with prepared statements is following:

s = box.prepare("SELECT * FROM t WHERE id = ?;")
s:execute({1}) or box.execute(s.sql_str, {1})
s:execute({2}) or box.execute(s.sql_str, {2})
s:unprepare() or box.unprepare(s.query_id)

Structure of object is following (member : type):

- stmt_id: integer
  execute: function
  params: map [name : string, type : integer]
  unprepare: function
  metadata: map [name : string, type : integer]
  param_count: integer
...

In terms of remote connection:

cn = netbox:connect(addr)
s = cn:prepare("SELECT * FROM t WHERE id = ?;")
cn:execute(s.sql_str, {1})
cn:unprepare(s.query_id)

Closes #2592

This patch introduces support of prepared statements in IProto
protocol. To achieve this new IProto command is added - IPROTO_PREPARE
(key is 0x13). It is sent with one of two mandatory keys:
IPROTO_SQL_TEXT (0x40 and assumes string value) or IPROTO_STMT_ID (0x43
and assumes integer value). Depending on body it means to prepare or
unprepare SQL statement: IPROTO_SQL_TEXT implies prepare request,
meanwhile IPROTO_STMT_ID - unprepare.  Also to reply on PREPARE request a
few response keys are added: IPROTO_BIND_METADATA (0x33 and contains
parameters metadata of type map) and IPROTO_BIND_COUNT (0x34 and
corresponds to the count of parameters to be bound).

Part of #2592

This patch introduces local prepared statements. Support of prepared
statements in IProto protocol and netbox is added in the next patch.

Prepared statement is an opaque instance of SQL Virtual Machine. It can
be executed several times without necessity of query recompilation. To
achieve this one can use box.prepare(...) function. It takes string of
SQL query to be prepared; returns extended set of meta-information
including statement's ID, parameter's types and names, types and names
of columns of the resulting set, count of parameters to be bound.  Lua
object representing result of :prepare() invocation also features two
methods - :execute() and :unprepare(). They correspond to
box.execute(stmt.stmt_id) and box.unprepare(stmt.stmt_id), i.e.
automatically substitute string of prepared statement to be executed.
Statements are held in prepared statement cache - for details see
previous commit.  After schema changes all prepared statement located in
cache are considered to be expired - they must be re-prepared by
separate :prepare() call (or be invalidated with :unrepare()).

Two sessions can share one prepared statements. But in the current
implementation if statement is executed by one session, another is
not able to use it and will compile it from scratch and than execute.

SQL cache memory limit is regulated by box{sql_cache_size} which can be
set dynamically. However, it can be set to the value which is less than
the size of current free space in cache (since otherwise some statements
can disappear from cache).

Part of #2592

This patch introduces holder (as data structure) to handle prepared
statements and a set of interface functions (insert, delete, find) to
operate on it. Holder under the hood is implemented as a global hash
(keys are values of hash function applied to the original string containing
SQL query; values are pointer to wrappers around compiled VDBE objects) and
GC queue. Each entry in hash has reference counter. When it reaches 0
value, entry is moved to GC queue. In case prepared statements holder is
out of memory, it launches GC process: each entry in GC queue is deleted
and all resources are released. Such approach allows to avoid workload
spikes on session's disconnect (since on such event all statements must
be deallocated).
Each session is extended with local hash to map statement ids available
for it. That is, session is allowed to execute and deallocate only
statements which are previously prepared in scope of this session.
On the other hand, global hash makes it possible to share same prepared
statement object among different sessions.
Size of cache is regulated via box.cfg{sql_cache_size} parameter.

Part of #2592

join_vclock test is assumed to verify that changes are not being lost
on the replica. Due to this the test is changed to explicitly check
that all changes on master are applied on replica.
Previously this test was also indirectly verifying that changes are
being applied in the correct order. Now there is separate test for
this, called replica_apply_order.
As far as changed join_vclock test might fail due to #4669, we are now
creating cluster out of fresh instances instead of using default
instance. Considering mentioned fixes it is not fragile anymore.

Closes #4160

In case of high load vclock used to join replica could be in advance
comparing to an actual WAL. Therefore replica could have missed some
tuples from master. In order to fix this wal_sync is updated so that
now we can obtain up to date vclock on the flushed state using it.

Prerequisites #4160

This patch removes control pragmas. They are not needed now, after
the introduction of the _session_settings system space.

Closes #4511

Part of #4511

@TarantoolBot document
Title: _session_settings system space
The _session_settings system space used to view or change session
settings.

This space uses a new engine. This allows us to create tuples on
the fly when the get() or select() methods are called. This
engine does not support the insert(), replace(), and delete()
methods. The only way to change the setting value is update(),
which can only be used with the "=" operation.

Because space creates a tuple on the fly, it allows us to get a
tuple without saving it anywhere. But this means that every time
we get a tuple from this system space, it is a new tuple, even if
they look the same:

tarantool> s = box.space._session_settings
tarantool> name = 'sql_default_engine'
tarantool> s:get({name}) == s:get({name})
---
- false
...

Currently, this space contains only SQL settings, since the only
session settings are SQL settings.

List of currently available session settings:

sql_default_engine
sql_defer_foreign_keys
sql_full_column_names
sql_parser_debug
sql_recursive_triggers
sql_reverse_unordered_selects
sql_select_debug
sql_vdbe_debug

The default values of these settings cannot be changed by the
user.

Debug settings are disabled by default and can only be enabled in
the debug build.

Example of usage:
tarantool> s = box.space._session_settings
-- View session settings values.
tarantool> s:get({'sql_default_engine'})
---
- ['sql_default_engine', 'memtx']
...

tarantool> s:select()

s:select()
---
- - ['sql_default_engine', 'memtx']
  - ['sql_defer_foreign_keys', false]
  - ['sql_full_column_names', false]
  - ['sql_full_metadata', false]
  - ['sql_parser_debug', false]
  - ['sql_recursive_triggers', true]
  - ['sql_reverse_unordered_selects', false]
  - ['sql_select_debug', false]
  - ['sql_vdbe_debug', false]
...

tarantool> s:select('sql_g', {iterator='LE'})
---
- - ['sql_full_metadata', false]
  - ['sql_full_column_names', false]
  - ['sql_defer_foreign_keys', false]
  - ['sql_default_engine', 'memtx']
...

-- Change session setting value.
tarantool> s:update('sql_default_engine', {{'=', 'value', 'vinyl'}})
---
- ['sql_default_engine', 'vinyl']
...

This patch creates _session_settings system space. This space is
used to view and change session settings. This space is one of the
special spaces that have a "service" engine. The main idea of this
space is that it will not store tuples, but when you call the
get() or select() methods, it creates tuples on the fly. Because
of this, even if the same setting is asked, the returned tuples
will be different. In addition, this space allows you to change
the setting value using the update() method, in which case it
directly changes the setting value.

There are no settings at the moment, some will be added in the
next patch.

Part of #4511

This patch introduces a new engine called "service" that will be
used to create a new system space. The main idea of this engine is
that it will not have a predefined space_vtab. With this engine,
we can create unusual spaces with their own vtab and behavior.

Due to the nature of this engine, it can only be used to create
system spaces.

Part of #4511

The vdbe_addoptrace pragma provides a convenient way to track the
insertion of opcodes into VDBE during VDBE creation. This patch
makes it impossible to disable this feature in the debug build by
removing the pragma. So, now you do not need to enable pragma in
order to use this feature.

Part of #4511

Pragma sql_compound_select_limit was added in commit b2afe208
("sql: decrease SELECT_COMPOUND_LIMIT threshold"). However, there
is no need to make this parameter mutable. We also plan to rework
SELECT (#3700), so this limit will be removed in future.

Part of #4511

The pragmas "short_column_names" and "full_column_names" allow us
to use three ways to display the column name in metadata:
1) If both are turned off, then the column name was shown as it
was written by the user.
2) If "short_column_names" = OFF and "full_column_names" = ON,
then the column name is displayed as <table name>.<column name>.
3) If "short_column_names" = ON, then the column name is displayed
as <column name>. This is the default option.

But we need only two ways to show the column name:
1) Show the column name as <column name>. This should be the
default option.
2) Show the column name as <table name>.<column name>.

In this regard, we need only one of these pragmas.

Part of #4511

Pragma "count_changes" forces the INSERT, REPLACE, DELETE, and
UPDATE statements to return the number of changed rows as a
result set. This is not necessary, as these statements return the
number of changed rows as metadata.

Part of #4511

* Support to set default SQL engine using _session_settings space in
  addition to pragma sql_default_engine. This feature is only for
  *.test.sql tests. Needed for #4511.

* Use exact IPv4/IPv6 address in test_run:cmd() in order to avoid rare
  failures due to using wrong address (PR #197).

Put constraint names into the space's hash table and drop them on
insert/delete in corresponding system spaces (_index,
_fk_constraint, _ck_constraint).

Closes #3503

@TarantoolBot document
Title: Constraint names are unique in scope of table

SQL:
According to ANSI SQL, table constraint is one of the following
entities: PRIMARY KEY, UNIQUE, FOREIGN KEY, CHECK. (Also there
is NOT NULL, but we don't consider it a constraint.) Every
constraint has its own name passed by user or automatically
generated. And these names must be unique within one table/space.

For example:

    tarantool> box.execute([[CREATE TABLE test (
                                 a INTEGER PRIMARY KEY,
                                 b INTEGER,
                                 CONSTRAINT cnstr CHECK (a >= 0)
                             );]])
    ---
    - row_count: 1
    ...

    tarantool> box.execute('CREATE UNIQUE INDEX cnstr ON test(b);')
    ---
    - null
    - Constraint CHECK 'CNSTR' already exists in space 'TEST'
    ...

Unique index and CHECK are different constraint types, but they
share namespace, and can't have clashing names. The same for all
the other constraints.

Method decode_unchecked returns two values - the one that has
been decoded and a pointer to the new position within the buffer
given as a parameter. The type of returned pointer used to be
cdata<unsigned char *> and it was not possible to assign returned
value to buf.rpos due to the following error:

> cannot convert 'const unsigned char *' to 'char *'

The patch fixes this by making decode_unchecked method return either
cdata<char *> or cdata<const char *> depending on the given parameter.

Closes #3926