In Lua to get the only first value in a sequence it is
enough to place it into ().

Follow up #3323, #3322

Introducing a formal, trackable process for server enhancement.

Before working on a complex feature, please write an RFC document,
describing what and how you see changed, and get it approved.

All historical RFCs are kept in doc/rfc.

Currently, space_vtab::commit_alter is called before WAL write so we can
use it for preparing new indexes in vinyl. However, this is going to
change soon, because actually space_vtab::commit_alter should be called
after WAL write, like index_vtab::commit_drop or commit_create. Calling
it before WAL write may result in use-after-free in memtx (see #3289).
Besides, using this function for iterating over space indexes just feels
wrong, as we have index methods invoked by AlterSpaceOp descendants for
this.

So let's move all the operations performed by vinyl_space_commit_alter
somewhere else. Fortunately, it's easy to do without damaging code
readability or efficiency:

 - Update of vy_lsm::pk can be done by vinyl_space_swap_index and
   vinyl_build_secondary_key.

 - vy_lsm->check_is_unique can be computed by vinyl_engine_create_space
   and then set by vinyl_space_swap_index.

 - Call to key_def_update_optionality is implied by key_def_swap, which
   is already called by vinyl_space_swap_index, hence it can be removed.

Needed for #3289

Basically, index_begin_build() followed by index_end_build() is a no-op.
There's absolutely no point in calling it for primary indexes after
initial recovery has completed.

_request() is a wrapper for perform_request, that detects schema
changes, and waits until it is reloaded. Lets use _request()
instead of direct perform_request() for execute(). The reason why
the _request() was not used earlier was my attempt to avoid
multiple return values in _request(), that leads to minor fixes
in non-execute methods like index.select or eval/call_16, which
return the _request() directly.

But diplicating schema reloading logic for execute() is worse.

Closes #3323
Closes #3322

This API is deprecated in 1.7.4, so it must not be
supported in new methods like execute().

We read tuples from disk hence we should use disk_format, not
mem_format. Fix it. While we are at it, let's also update the
outdated comment to vy_run_rebuild_index.

The rtree is empty when this function is called (in fact, it is called
right after creating the index), there's no need to purge it.

To allow extending key definition for non-empty vinyl spaces, this patch
performs the following steps:

 - Revert commit c31dd19a ("vinyl: forbid vinyl index key definition
   alter") that forbade any key def alter. It isn't needed anymore.

 - Update key_def and cmp_def in vinyl_space_swap_index(). We simply
   swap the definitions between the old and new indexes in memory.
   Since all vinyl objects reference either vy_lsm::key_def or
   vy_lsm::cmp_def or both, and the change is compatible (does not
   change the order for existing tuples), this should work just fine.

 - Update key definition in vylog on ALTER. For this, we introduce a new
   vylog record type, VY_LOG_MODIFY_LSM, which updates key definition.
   To be able to replay it on recovery in case we failed to flush it
   before restart, we also store the LSN of the WAL record that
   triggered the ALTER.

It also adds the following test cases:

 - Modify key definition of primary and secondary indexes of a non-empty
   space (engine/ddl).

 - Modify key definition before snapshot and relay it to a newly joined
   replica (engine/replica_join).

 - Make sure key definition is updated in vylog on ALTER (vinyl/layout).

There's no point in writing this record to snapshot, because we can
store LSN of the last index dump right in VY_LOG_CREATE_LSM record.

So as to draw the line between LSN of index creation and LSN of last
index modification, which is introduced by later in the series.

As a part of SQL data dictionary integration, 'DROP INDEX' and
'DROP TABLE' statements proccessing has been refactored in order
to avoid using SQL specific internal structures.
However, triggers still aren't transfered to server, so to drop them
it is required to lookup SQL table and emit apporpriate opcodes.
Also, added comments and fixed codestyle for functions processing
'DROP' routine.

Part of #3222.

Some of legacy functions seem to be useless, since they serve as
wrappers around others; the rest rely on capabilities which are no
longer relevant. This patch provides slight refactoring of such
functions or removes them at all.

There are two functions in vy_stmt.c that blindly copy tuple field map,
vy_stmt_dup() and vy_stmt_replace_from_upsert(). Both these functions
take a tuple format to use for the new statement and require this format
to be the same as the source tuple format in terms of fields definition,
otherwise they'll just crash. The only reason why we did that is that
back when these functions were written we used a separate format for
UPSERT statements so we needed this extra argument for creating a
REPLACE from UPSERT. Now it's not needed, and we can use the source
tuple format instead. Moreover, passing the current tuple format to any
of those functions is even harmful, because tuple format can be extended
by ALTER, in which case these functions will crash if called on a
statement created before ALTER. That being said, let's drop the tuple
format argument.

space->format and cmp_def must be compatible, i.e. space->format has
is_nullable flag set for a field iff it is set for all key parts
indexing this field. Therefore there's no point to set is_nullable for
disk_format as it must have been initialized by tuple_format_create().
Remove the pointless loop.

Also, while we are at it, fix the minor memory leak - disk_format is
referenced twice for the primary key.

A piece of code left from the inglorious past, which doesn't even have
a forward declaration, let alone used anywhere. Remove it.

We allocate index formats in the only place, vy_lsm_new, so there's no
point in this debug-only check anymore.

We create new formats for all indexes of the new space in
vinyl_space_commit_alter() while we don't actually need to
do this, because the new formats have already been created
by vy_lsm_new() - all we need to do is reuse them somehow.

This patch does the trick: it implements the swap_index()
space virtual method for vinyl so that it swaps tuple formats
between the old and new spaces.

This function is called by MoveIndex and ModifyIndex ALTER operations,
i.e. when the index definition is not changed at all or is extended.
Making this method virtual will allow to avoid reallocation of vinyl
formats in vinyl_space_commit_alter().

If the new cmp_def of a secondary index is compatible with the old one
after the primary key parts have changed, we don't need to rebuild it,
we just need to update its definition.

Some pragmas turn out to be useless and can't be applied to Tarantool's SQL.
Moreover, couple of them lead to assertion faults.
List or removed pragmas:
- synchronous
- empty_result_callbacks
- WAL_autocheckpoint
- WAL_checkpoint
- secure_delete
- database_list
- data_version
- compile_options
- application_id
- user_version
- pragmas connected with HAS_CODEC macro
- pragmas connected with heap memory managment

If the primary key is modified, we schedule rebuild of all non-unique
(including nullable) secondary TREE indexes. This is valid for memtx,
but is not quite right for vinyl. For vinyl we have to rebuild all
secondary indexes, because they are all non-clustered (i.e. point to
tuples via primary key parts). This doesn't result in any bugs for now,
because rebuild of vinyl indexes is not supported, but hopefully this is
going to change soon. So let's introduce a new virtual index method,
index_vtab::depends_on_pk, which returns true iff the index needs to be
updated if the primary key changes, and define this new method for vinyl
and memtx TREE indexes.

This change removes REINDEX from keywords and also disables tests
which were using this syntax.

This patch affects only parser, REINDEX implementation still exists,
because it is planned to be fixed and enabled in the scope of #3195.

Note that REINDEX keyword is still in '%fallback' directive in
'parse.y'. Reserved keywords included there to avoid situation when
a keyword has no usages within 'parse.y' file.

Fixes #2174

The new method is called after successful update of index definition.
It is passed the signature of the WAL record that committed the
operation. It will be used by Vinyl to update key definition in vylog.

Fix non-working 'INDEXED BY' clause for SELECT statement,
which caused syntax-error during execution.

Closes #2996

Type of an operation for struct request can be passed directly
with no "proxying" by special codes.

1) Code is completely unreadable and complicated by strange
   optimizations;
2) It still supposes, that there can be multiple statements in a
   single string;
3) It pushes first letter of affinity together with column names
   - why? It is not used anywhere.

Lets remove prepared statements list, affinity first letters
pushing.

Remove unused error codes, use enum instead of
defines. Later the prefix SQLITE_ must be removed -
see #3315.

And rework sqlite3CompleteInsertion. Before the patch it
* accepts actually unused pTab;
* array of records to insert in different indexes, but inserts
  only one;
* has sqlite code style.

Code style fix is obvious. PTab is used only to check, that it is
not a view and to check, that a first index is primary:
1) caller functions already guarantee the pTab is not a view;
2) regardless of a first index nature: primary or not - it is not
   used here. It is useless check. With the same success we can
   check this in each function, that uses struct Table.

Array of records to insert makes no sense, since insertion is
being done in a primary index only. It is enough to pass a
register with a primary index tuple.

* Remove rewriting format of default logger in case of syslog option.
* Add facility option parsing and use parsed results in format message
  according to RFC3164. Possible values and default value of syslog
  facility are taken from nginx (https://nginx.ru/en/docs/syslog.html)
* Move initialization of logger type and format fucntion before
  initialization of descriptor in log_XXX_init, so that we can test
  format function of syslog logger.

Closes gh-3244.

The only difference between format of UPSERT statements and format of
other DML statements of the same index is that the former reserves one
byte for UPSERT counter, which is needed to schedule UPSERT squashing.
Since we store UPSERT counter on lsregion now, we don't need a special
format for UPSERTs anymore. Remove it.

Currently, we store upsert counter in tuple metadata (that's what
upsert_format is for), but since it's only relevant for tuples of
the memory level, we can store it on lsregion, right before tuple
data. Let's do it now so that we can get rid of upsert_format.

Filed gh-3311 to remove this export soon.

Fixes #3310.

If the size of a transaction is greater than the configured memory
limit (box.cfg.vinyl_memory), the transaction will hang on commit
for 60 seconds (box.cfg.vinyl_timeout) and then fail with the
following error message:

  Timed out waiting for Vinyl memory quota

This is confusing. Let's fail such transactions immediately with
OutOfMemory error.

Closes #3291

Extract expressions parsing into separate routine to
allow Tarantool's backend compile DEFAULT statements w/o
SQL machinery at all. So far, for DEFAULT values no context
is needed: only constant expressions and built-ins are allowed.
In future, table context will be added to allow use column
values for CHECK constraint expressions.

Move DEFAULT string value to space_def. Move compiled expresion
to field_def as well. Reason not to move compiled expression
to tuple_field as follows: we do not want to engage parser
during tuple validation. So, do it in alter.cc.

In order to allow expression duplication in alter.cc: expose
those routines from expr.c and make their names correspond to
coding style.

Since recovery is performed before SQL fronend initialization:
split it into two pieces: 1. create SQL handler, enable
all subsystems 2. Do recovery.  This will allow to run
parser during recovery, since it needs db handle so far.

Part of #3235