When it comes for huge queries, it may turn out to be useful to see
exact position of occurred error. Hence, let's now display line and
position within line near which syntax error takes place. Note that it
can be done only during parsing process (since AST can be analysed only
after its construction is completed), so most of semantic errors still
don't contain position. A few errors have been reworked to match new
formatting patterns.

First iteration of this patch is implemented by @romanhabibov

Closes #2611

This patch introduces type DOUBLE in SQL.

Closes #3812
Needed for #4233

@TarantoolBot document
Title: Tarantool DOUBLE field type and DOUBLE type in SQL
The DOUBLE field type was added to Tarantool mainly for adding the
DOUBLE type to SQL. Values of this type are stored as MP_DOUBLE in
msgpack. The size of the encoded value is always 9 bytes.

In Lua, only non-integer numbers and CDATA of type DOUBLE can be
inserted in this field. You cannot insert integers of type Lua
NUMBER or CDATA of type int64 or uint64 in this field. The same
rules apply to key in get(), select(), update() and upsert()
methods. It was done this way to avoid unwanted implicit casts
that could affect performance.

It is important to note that you can use the ffi.cast() function
to cast numbers to CDATA of type DOUBLE. An example of this can be
seen below.

Another very important point is that CDATA of type DOUBLE in lua
can be used in arithmetic, but arithmetic for them does not work
correctly. This comes from LuaJIT and most likely will not be
fixed.

Example of usage in Lua:
s = box.schema.space.create('s', {format = {{'d', 'double'}}})
_ = s:create_index('ii')
s:insert({1.1})
ffi = require('ffi')
s:insert({ffi.cast('double', 1)})
s:insert({ffi.cast('double', tonumber('123'))})
s:select(1.1)
s:select({ffi.cast('double', 1)})

In SQL, DOUBLE type behavior is different due to implicit casting.
In a column of type DOUBLE, the number of any supported type can
be inserted. However, it is possible that the number that will be
inserted will be different from that which is inserted due to the
rules for casting to DOUBLE. In addition, after this patch, all
floating point literals will be recognized as DOUBLE. Prior to
that, they were considered as NUMBER.

Example of usage in SQL:
box.execute('CREATE TABLE t (d DOUBLE PRIMARY KEY);')
box.execute('INSERT INTO t VALUES (10), (-2.0), (3.3);')
box.execute('SELECT * FROM t;')
box.execute('SELECT d / 100 FROM t;')
box.execute('SELECT * from t WHERE d < 15;')
box.execute('SELECT * from t WHERE d = 3.3;')

Closes #4244

@TarantoolBot document
Title: an ability to disable CK constraints

Now it is possible to disable and enable ck constraints.
All ck constraints are enabled by default when Tarantool is
configured. Ck constraints checks are not performed during
standard recovery, but performed during force_recovery -
all conflicting tuples are skipped in case of ck_constraint
conflict.

To change CK constraint "is_enabled" state, call
-- in LUA
ck_obj:enable(new_state in {true, false})
-- in SQL
ALTER TABLE {TABLE_NAME} {EN, DIS}ABLE CHECK CONSTRAINT {CK_NAME};

Example:
box.space.T6.ck_constraint.ck_unnamed_T6_1:enable(false)
box.space.T6.ck_constraint.ck_unnamed_T6_1
- space_id: 512
  is_enabled: false
  name: ck_unnamed_T6_1
  expr: a < 10
box.space.T6:insert({11})
-- passed
box.execute("ALTER TABLE t6 ENABLE CHECK CONSTRAINT \"ck_unnamed_T6_1\"")
box.space.T6:insert({12})
- error: 'Check constraint failed ''ck_unnamed_T6_1'': a < 10'

ENGINE became reserved keyword in 1013a744. There's no any actual
reason why ENGINE should be reserved keyword. What is more, we are going
to use this word as a name of some fields for tables forming
informational schema. Hence, until it is too late (it is not documented
yet), let's remove ENGINE from the list of reserved keywords and allow
identifiers be that word.

`tarantoolctl start` patches box.cfg two times:
1) before the init script to set default values and enforce some others,
2) after the init script to prevent changing a pid_file in runtime.

The second patching fails if an init file does not call
box.cfg{} before it's finished. This can take a place in apps with
managed instances which receive configuration from external server.

This patch moves the second patching into the box.cfg
wrapper created during the first patching. So the second patching
is performed only after box.cfg{} was invoked, so it does not fail anymore.

However there is relatively minor flaw for applications that
invoke box.cfg{} after init script is finished:
`tarantoolctl start` goes to background only when box.cfg{} is called.
Though this is not the case for daemon management systems like systemd,
as they handle backgrounding on their side

Fixes #4435

@TarantoolBot document
Title: tarantoolctl allows to start instances without a box.cfg{}

tarantoolctl now works for instances without box.cfg{} or
with delayed box.cfg{} call. This can be managed instances which receive
configuration from external server.

For such instances `tarantoolctl start` goes to background when
box.cfg{} is called, so it will wait until options for box.cfg are received.
However this is not the case for daemon management systems like systemd,
as they handle backgrounding on their side.

Closes #4422

@TarantoolBot document
Title: Introduce <WITH ENGINE> clause for CREATE TABLE statement

To allow user to specify engine as per table option, CREATE TABLE
statement has been extended with optional <WITH ENGINE = engine_name>
clause. This clause comes at the end of CREATE TABLE statement.
For instance:

CREATE TABLE t_vinyl (id INT PRIMARY KEY) WITH ENGINE = 'vinyl';

Name of engine is considered to be string literal ergo should be
enclosed in single quotation marks and be lower-cased. Note that engine
specified in WITH ENGINE clause overwrites default engine, which is set
via 'pragma sql_default_engine'.

This patch adds the methods necessary to encode and decode decimals to
MsgPack. MsgPack EXT type (MP_EXT) together with a new extension type
MP_DECIMAL is used as a record header.

The decimal MsgPack representation looks like this:
+--------+-------------------+------------+===============+
| MP_EXT | length (optional) | MP_DECIMAL | PackedDecimal |
+--------+-------------------+------------+===============+
The whole record may be encoded and decoded with
mp_encode_decimal() and mp_decode_decimal(). This is equivalent to
performing mp_encode_extl()/mp_decode_extl() on the first 3 fields and
decimal_pack/unpack() on the PackedDecimal field.

It is also possible to decode and encode decimals to msgpack from lua,
which means you can insert decimals into spaces, but only into unindexed
fields for now.

Follow up #692
Part of #4333

Originally, mask in struct Keyword served to reduce set of reserved
keywords for build-dependent features. For instance, it was allowed to
disable triggers as a compilation option, and in this case TRIGGER
wouldn't be reserved word. Nowadays, our build always comprises all
features, so there's no need in this option anymore. Hence, we can
remove mask alongside with related to it macros.

Closes #4155

Current patch introduces new type available in SQL:
 - VARBINARY now is reserved keyword;
 - Allow to specify VARBINARY column and CAST type;
 - All literals which start from 'x' are assumed to be of this type;
 - There's no available implicit or explicit conversions between
   VARBINARY and other types;
 - Under the hood all values of VARBINARY type are stored as MP_BIN
   msgpack format type.

Closes #4206

Before this patch it was allowed to specify REAL/FLOAT/DOUBLE types
which matched with NUMBER NoSQL type in space format. However, NUMBER is
different from standard floating point types, since it is able to hold
integers in range [-2^63; 2^64-1] alongside with double precision
floating point values. Hence, to not confuse users it has been decided
to remove support of REAL/FLOAT/DOUBLE types from SQL grammar and use
instead original NUMBER type naming.

TEXT type is called "string" in the original Tarantool NoSQL, so it would
be rational to allow using the same type name in SQL.

Since all preparations concerning internal handling of unsigned values
have been done, now nothing prevents us from using UNSIGNED type in SQL.
This patch allows to specify UNSIGNED as a column type and adds CAST
rules, which are the same as for casual INTEGER, but with additional
check - result must be positive. Otherwise, error is raised.

Closes #4015

Built-in modules are bundled into tarantool in the following way. A lua
file from src/lua or src/box/lua is stored as a string literal in a C
file, then built and linked into tarantool. During startup tarantool
calls luaL_loadbuffer() on this string.

When a Lua source is converted to a C literal, proper escaping is
performed. However there is one case, which was not covered: trigraphs.
The patch adds escaping of question mark symbols to avoid matching ??X
sequences as trigraphs by C preprocessor.

The most simple way to check that it works is to apply the following
patch:

 | diff --git a/src/lua/string.lua b/src/lua/string.lua
 | index 6e12c59ae..2da2dbf4d 100644
 | --- a/src/lua/string.lua
 | +++ b/src/lua/string.lua
 | @@ -425,3 +425,6 @@ string.fromhex    = string_fromhex
 |  string.strip      = string_strip
 |  string.lstrip      = string_lstrip
 |  string.rstrip      = string_rstrip
 | +string.foo = function()
 | +    return '??('
 | +end

And call the function like so:

 | ./src/tarantool -e 'print(string.foo()) os.exit()'

If it printfs `??(`, then everything is okay. If it prints `[`, then
`??(` was preprocessed as the trigraph.

We hit this problem when tried to bundle luarocks-3: it contains
"^(.-)(%??)$" regexp, where `??)` was interpreted as `]`. Debug build or
a build with -DENABLE_WERROR reports an error in the case, but usual
RelWithDebInfo build passes (with -Wtrigraphs warnings) and can show
this unexpected behaviour.

Fixes #4291.

* Added a new command `test` that is new feature of luarocks 3.x.
* Added a new command `make_manifest` that is command from
  luarocks-admin.
* Added a new command `help` for getting help of internal luarocks
  commands.

Also refined arguments descriptions of existing tarantoolctl rocks
commands.

Made tarantoolctl compatible with luarocks-3.x.

Fixes #4052.

Removes the following unused macros:
SQL_OMIT_AUTOINCREMENT
SQL_OMIT_CONFLICT_CLAUSE
SQL_OMIT_PRAGMA
SQL_PRINTF_PRECISION_LIMIT
SQL_OMIT_COMPOUND_SELECT
SQL_POWERSAFE_OVERWRITE
SQL_OMIT_PROGRESS_CALLBACK
SQL_OMIT_AUTORESET
SQL_OMIT_DECLTYPE
SQL_ENABLE_COLUMN_METADATA
SQL_TRACE_SIZE_LIMIT
SQL_OMIT_LIKE_OPTIMIZATION
SQL_OMIT_OR_OPTIMIZATION
SQL_OMIT_BETWEEN_OPTIMIZATION
SQL_EXPLAIN_ESTIMATED_ROWS
SQL_ENABLE_COLUMN_USED_MASK
SQL_DISABLE_DIRSYNC
SQL_DEBUG_SORTER_THREADS
SQL_DEFAULT_WORKER_THREADS
SQL_LIMIT_WORKER_THREADS
SQL_MAX_WORKER_THREADS
SQL_ENABLE_MEMORY_MANAGEMENT
SQL_ENABLE_UNKNOWN_SQL_FUNCTION
SQL_SUBSTR_COMPATIBILITY
SQL_ENABLE_STMT_SCANSTATUS
Removed the remains of multithreading in the VDBE
sorting tools.
Updates tests.

Closes #3978

These function are going to yield in scope of #4250, because
swim:new() will start a fiber, while swim:delete() cancels and
gives it a control.

Needed for #4250

Encryption with an arbitrary algorithm and any mode with a
configurable private key.

Closes #3234

Users of Lua SWIM module likely will use Lua objects as a
payload. Lua objects are serialized into MessagePack
automatically, and deserialized back on other instances. But
deserialization of 1.2Kb payload on each member:payload()
invocation is quite heavy operation. This commit caches decoded
payloads to return them again until change.

A microbenchmark showed, that cached payload is returned ~100
times faster, than it is decoded each time. Even though a tested
payload was quite small and simple:

    s:set_payload({a = 100, b = 200})

Even this payload is returned 100 times faster, and does not
affect GC.

Part of #3234

SWIM as a library can be useful not only for server internals,
but for users as well. This commit exposes Lua bindings to SWIM
C API. Here only basic bindings are introduced to create, delete,
quit, check a SWIM instance. With sanity tests.

Part of #3234

Static allocator gives memory blocks from cyclic BSS memory
block of 3 pages 4096 bytes each. It is much faster than
malloc, when a temporary buffer is needed. Moreover, it does not
complicate GC job. Despite being faster than malloc, it is still
slower, than ffi.new() of size <= 128 known in advance (according
to microbenchmarks).

ffi.new(size<=128) works either much faster or with the same
speed as static_alloc, because internally FFI allocator library
caches small blocks and can return them without malloc().

A simple micro benchmark showed, that ffi.new() vs
buffer.static_alloc() is ~100 times slower on allocations of
> 128 size,  on <= 128 when size is not inlined.

To better understand what is meant as 'inlined': this

    ffi.new('char[?]', < <=128 >)

works ~100 times faster than this:

    local size = <= 128
    ffi.new('char[?]', size)

ffi.new() with inlined size <= 128 works faster than light, and
even static allocator can't beat it.

This patch stops the parser if any error occurs. Prior to this
patch, it was possible to replace the error with another one,
since the parser may continue to work, even if an error occurred.

For example:
box.execute("insert into not_exist values(1) a")

The first error is "Space 'NOT_EXIST' does not exist", but "Syntax
error near 'a'" is displayed.

After this patch, the first error will be displayed.

Closes #3964
Closes #4195

crypto.lua is a public module using OpenSSL directly. But now
lib/crypto encapsulates OpenSSL with additional checks and
similar but more conforming API. It allows to replace OpenSSL
cipher in crypto.lua with lib/crypto methods.

Tarantool has a strict rule for naming methods of libraries - use
the library name as a prefix. For crypto lib methods it should be
'crypto_', not 'tnt_'.

Needed for #3276.

This patch introduces basic facilities to operate on boolean type:
boolean literals "true" and "false" where true > false; alias to null -
unknown; column type "BOOLEAN" and shortcut "BOOL"; opportunity to
insert and select boolean values from table; OR and AND predicates
accept boolean arguments; CAST operation involving boolean type;
comparison between boolean values (including VDBE sorter routines).

Part of #3648

According to the ANSI standard, ltrim, rtrim and trim should
be merged into one unified TRIM() function. The specialization of
trimming (left, right or both and trimming characters) determined
in arguments of this function.

Closes #3879

@TarantoolBot document
Title: TRIM() function

Modify signature of SQL function TRIM(). This function removes
characters included in <trim character> (binary) string from
<trim source> (binary) string until encounter a character that doesn't
belong to <trim character>. Removal occurs on the side, specified by
<trim specification>. Now, syntax is following:
TRIM([ [ <trim specification> ] [ <trim character> ] FROM ] <trim source>).

<trim specification> can be one of the following keywords: LEADING,
TRAILING and BOTH.
<trim character> is the set of trimming characters.
<trim source> is the string, that will be trimmed.
If FROM is specified, then:
1) Either <trim specification> or <trim character> or both shall be
specified.
2) If <trim specification> is not specified, then BOTH is implicit.
3) If <trim character> is not specified, then ' ' is implicit.

There can be a lot of small files with vinyl.

Added a check whether box.cfg() is called within an instance
file. If box.cfg() is missed, point a user the reason of a
fail explicitly.

Before this commit the error was look so:

/usr/bin/tarantoolctl:541: attempt to index a nil value

Closes #3953

Stop a file processing loop only when it is guaranteed that we will not
find a record that match user-provided filters later in this file. If
--replica R is provided one time and we're meet a record from R with a
LSN equal or above of a --to value, we'll stop the loop. Otherwise (no
--replica, several --replica arguments) a file will be read until an end
even if --to is provided.

Fixes #3827.

Exposed this unified code (filter_xlog() function) and wrote a unit
test.

Allow to run app-tap/tarantoolctl.test.lua w/o test-run.

Needed for #3827.

It's OK to use json format with the boot logger. As for syslog, let's
add a check to Lua's log_format() so that it fails gracefully rather
than raising an assertion.

Closes #3946

At this point, an ANALYZE statement can lead to many problems. It
was decided to temporarily ban this statement.

Closes #4069

BLOB column type is represented by SCALAR field type in terms of NoSQL.
We attempted at emulating BLOB behaviour, but such efforts turn out to
be not decent enough. For this reason, we've decided to abandon these
attempts and fairly replace it with SCALAR column type.  SCALAR column
type acts in the same way as it does in NoSQL: it is aggregator-type for
INTEGER, NUMBER and STRING types. So, column declared with this type can
contain values of these three (available in SQL) types. It is worth
mentioning that CAST operator in this case does nothing.

Still, we consider BLOB values as entries encoded in msgpack with MP_BIN
format. To make this happen, values to be operated should be represented
in BLOB form x'...' (e.g. x'000000'). What is more, there are two
built-in functions returning BLOBs: randomblob() and zeroblob().  On the
other hand, columns with STRING NoSQL type don't accept BLOB values.

Closes #4019
Closes #4023

@TarantoolBot document
Title: SQL types changes
There are couple of recently introduced changes connected with SQL
types.

Firstly, we've removed support of DATE/TIME types from parser due to
confusing behaviour of these types: they were mapped to NUMBER NoSQL
type and have nothing in common with generally accepted DATE/TIME types
(like in other DBs). In addition, all built-in functions related to
these types (julianday(), date(), time(), datetime(), current_time(),
current_date() etc) are disabled until we reimplement TIME-like types as
a native NoSQL ones (see #3694 issue).

Secondly, we've removed CHAR type (i.e. alias to VARCHAR and TEXT). The
reason is that according to ANSI SQL CHAR(len) must accept only strings
featuring length exactly equal to given in type definition. Obviously,
now we don't provide such checks. Types VARCHAR and TEXT are still
legal.

For the same reason, we've removed NUMERIC and DECIMAL types, which were
aliases to NUMBER NoSQL type. REAL, FLOAT and DOUBLE are still exist as
aliases.

Finally, we've renamed BLOB column type to SCALAR. We've decided that
all our attempts to emulate BLOB behaviour using SCALAR NoSQL type don't
seem decent enough, i.e. without native NoSQL type BLOB there always
will be inconsistency, especially taking into account possible NoSQL-SQL
interactions. In SQL SCALAR type works exactly in the same way as in
NoSQL: it can store values of INTEGER, FLOAT and TEXT SQL types at the
same time. Also, with this change behaviour of CAST operator has been
slightly corrected: now cast to SCALAR doesn't affect type of value at
all. Couple of examples:

CREATE TABLE t1 (a SCALAR PRIMARY KEY);
INSERT INTO t1 VALUES ('1');
SELECT * FROM t1 WHERE a = 1;
-- []
Result is empty set since column "a" contains string literal value '1',
not integer value 1.

CAST(123 AS SCALAR); -- Returns 123 (integer)
CAST('abc' AS SCALAR); -- Returns 'abc' (string)

Note that in NoSQL values of BLOB type defined as ones decoded in
msgpack with MP_BIN format. In SQL there are still a few ways to force
this format: declaring literal in "BLOB" format (x'...') or using one of
two built-in functions (randomblob() and zeroblob()). TEXT and VARCHAR
SQL types don't accept BLOB values:

CREATE TABLE t (a TEXT PRIMARAY KEY);
INSERT INTO t VALUES (randomblob(5));
---
- error: 'Tuple field 1 type does not match one required: expected string'
...

BLOB itself is going to be reimplemented in scope of #3650.

NMERIC and DECIMAL were allowed to be specified as column types. But in
fact, they were just synonyms for FLOAT type and mapped to NUMERIC
Tarantool NoSQL type. So, we've decided to remove this type from parser
and return back when NUMERIC will be implemented as a native type.

Part of #4019

Currently, there is no native (in Tarantool terms) types to represent
time-like types. So, until we add implementation of those types, it
makes no sense to allow to specify those types in table definition.
Note that previously they were mapped to NUMBER type. For the same
reason all built-in functions connected with DATE/TIME are disabled as
well.

Part of #4019

Add optional 'chars' parameter to string.strip, string.lstrip
and string.rstrip for specifying the unwanted characters.
Behavior modeled after the equivalent Python built-ins.

Closes: #2977

@TarantoolBot document
Title: string.strip(inp, chars)
Update the documentation for string.strip,
string.lstrip and string.rstrip to reflect
the addition of the optional param.

Replace all usage of sqlite3_, sqlite, SQLite prefixes with simple sql_
All other occurrences of SQLite are substituted with SQL word.
SQL test suit is purified as well.