Originally, mask in struct Keyword served to reduce set of reserved
keywords for build-dependent features. For instance, it was allowed to
disable triggers as a compilation option, and in this case TRIGGER
wouldn't be reserved word. Nowadays, our build always comprises all
features, so there's no need in this option anymore. Hence, we can
remove mask alongside with related to it macros.

Closes #4155

Current patch introduces new type available in SQL:
 - VARBINARY now is reserved keyword;
 - Allow to specify VARBINARY column and CAST type;
 - All literals which start from 'x' are assumed to be of this type;
 - There's no available implicit or explicit conversions between
   VARBINARY and other types;
 - Under the hood all values of VARBINARY type are stored as MP_BIN
   msgpack format type.

Closes #4206

Before this patch it was allowed to specify REAL/FLOAT/DOUBLE types
which matched with NUMBER NoSQL type in space format. However, NUMBER is
different from standard floating point types, since it is able to hold
integers in range [-2^63; 2^64-1] alongside with double precision
floating point values. Hence, to not confuse users it has been decided
to remove support of REAL/FLOAT/DOUBLE types from SQL grammar and use
instead original NUMBER type naming.

TEXT type is called "string" in the original Tarantool NoSQL, so it would
be rational to allow using the same type name in SQL.

Since all preparations concerning internal handling of unsigned values
have been done, now nothing prevents us from using UNSIGNED type in SQL.
This patch allows to specify UNSIGNED as a column type and adds CAST
rules, which are the same as for casual INTEGER, but with additional
check - result must be positive. Otherwise, error is raised.

Closes #4015

Built-in modules are bundled into tarantool in the following way. A lua
file from src/lua or src/box/lua is stored as a string literal in a C
file, then built and linked into tarantool. During startup tarantool
calls luaL_loadbuffer() on this string.

When a Lua source is converted to a C literal, proper escaping is
performed. However there is one case, which was not covered: trigraphs.
The patch adds escaping of question mark symbols to avoid matching ??X
sequences as trigraphs by C preprocessor.

The most simple way to check that it works is to apply the following
patch:

 | diff --git a/src/lua/string.lua b/src/lua/string.lua
 | index 6e12c59ae..2da2dbf4d 100644
 | --- a/src/lua/string.lua
 | +++ b/src/lua/string.lua
 | @@ -425,3 +425,6 @@ string.fromhex    = string_fromhex
 |  string.strip      = string_strip
 |  string.lstrip      = string_lstrip
 |  string.rstrip      = string_rstrip
 | +string.foo = function()
 | +    return '??('
 | +end

And call the function like so:

 | ./src/tarantool -e 'print(string.foo()) os.exit()'

If it printfs `??(`, then everything is okay. If it prints `[`, then
`??(` was preprocessed as the trigraph.

We hit this problem when tried to bundle luarocks-3: it contains
"^(.-)(%??)$" regexp, where `??)` was interpreted as `]`. Debug build or
a build with -DENABLE_WERROR reports an error in the case, but usual
RelWithDebInfo build passes (with -Wtrigraphs warnings) and can show
this unexpected behaviour.

Fixes #4291.

* Added a new command `test` that is new feature of luarocks 3.x.
* Added a new command `make_manifest` that is command from
  luarocks-admin.
* Added a new command `help` for getting help of internal luarocks
  commands.

Also refined arguments descriptions of existing tarantoolctl rocks
commands.

Made tarantoolctl compatible with luarocks-3.x.

Fixes #4052.

Removes the following unused macros:
SQL_OMIT_AUTOINCREMENT
SQL_OMIT_CONFLICT_CLAUSE
SQL_OMIT_PRAGMA
SQL_PRINTF_PRECISION_LIMIT
SQL_OMIT_COMPOUND_SELECT
SQL_POWERSAFE_OVERWRITE
SQL_OMIT_PROGRESS_CALLBACK
SQL_OMIT_AUTORESET
SQL_OMIT_DECLTYPE
SQL_ENABLE_COLUMN_METADATA
SQL_TRACE_SIZE_LIMIT
SQL_OMIT_LIKE_OPTIMIZATION
SQL_OMIT_OR_OPTIMIZATION
SQL_OMIT_BETWEEN_OPTIMIZATION
SQL_EXPLAIN_ESTIMATED_ROWS
SQL_ENABLE_COLUMN_USED_MASK
SQL_DISABLE_DIRSYNC
SQL_DEBUG_SORTER_THREADS
SQL_DEFAULT_WORKER_THREADS
SQL_LIMIT_WORKER_THREADS
SQL_MAX_WORKER_THREADS
SQL_ENABLE_MEMORY_MANAGEMENT
SQL_ENABLE_UNKNOWN_SQL_FUNCTION
SQL_SUBSTR_COMPATIBILITY
SQL_ENABLE_STMT_SCANSTATUS
Removed the remains of multithreading in the VDBE
sorting tools.
Updates tests.

Closes #3978

These function are going to yield in scope of #4250, because
swim:new() will start a fiber, while swim:delete() cancels and
gives it a control.

Needed for #4250

Encryption with an arbitrary algorithm and any mode with a
configurable private key.

Closes #3234

Users of Lua SWIM module likely will use Lua objects as a
payload. Lua objects are serialized into MessagePack
automatically, and deserialized back on other instances. But
deserialization of 1.2Kb payload on each member:payload()
invocation is quite heavy operation. This commit caches decoded
payloads to return them again until change.

A microbenchmark showed, that cached payload is returned ~100
times faster, than it is decoded each time. Even though a tested
payload was quite small and simple:

    s:set_payload({a = 100, b = 200})

Even this payload is returned 100 times faster, and does not
affect GC.

Part of #3234

SWIM as a library can be useful not only for server internals,
but for users as well. This commit exposes Lua bindings to SWIM
C API. Here only basic bindings are introduced to create, delete,
quit, check a SWIM instance. With sanity tests.

Part of #3234

Static allocator gives memory blocks from cyclic BSS memory
block of 3 pages 4096 bytes each. It is much faster than
malloc, when a temporary buffer is needed. Moreover, it does not
complicate GC job. Despite being faster than malloc, it is still
slower, than ffi.new() of size <= 128 known in advance (according
to microbenchmarks).

ffi.new(size<=128) works either much faster or with the same
speed as static_alloc, because internally FFI allocator library
caches small blocks and can return them without malloc().

A simple micro benchmark showed, that ffi.new() vs
buffer.static_alloc() is ~100 times slower on allocations of
> 128 size,  on <= 128 when size is not inlined.

To better understand what is meant as 'inlined': this

    ffi.new('char[?]', < <=128 >)

works ~100 times faster than this:

    local size = <= 128
    ffi.new('char[?]', size)

ffi.new() with inlined size <= 128 works faster than light, and
even static allocator can't beat it.

This patch stops the parser if any error occurs. Prior to this
patch, it was possible to replace the error with another one,
since the parser may continue to work, even if an error occurred.

For example:
box.execute("insert into not_exist values(1) a")

The first error is "Space 'NOT_EXIST' does not exist", but "Syntax
error near 'a'" is displayed.

After this patch, the first error will be displayed.

Closes #3964
Closes #4195

crypto.lua is a public module using OpenSSL directly. But now
lib/crypto encapsulates OpenSSL with additional checks and
similar but more conforming API. It allows to replace OpenSSL
cipher in crypto.lua with lib/crypto methods.

Tarantool has a strict rule for naming methods of libraries - use
the library name as a prefix. For crypto lib methods it should be
'crypto_', not 'tnt_'.

Needed for #3276.

This patch introduces basic facilities to operate on boolean type:
boolean literals "true" and "false" where true > false; alias to null -
unknown; column type "BOOLEAN" and shortcut "BOOL"; opportunity to
insert and select boolean values from table; OR and AND predicates
accept boolean arguments; CAST operation involving boolean type;
comparison between boolean values (including VDBE sorter routines).

Part of #3648

According to the ANSI standard, ltrim, rtrim and trim should
be merged into one unified TRIM() function. The specialization of
trimming (left, right or both and trimming characters) determined
in arguments of this function.

Closes #3879

@TarantoolBot document
Title: TRIM() function

Modify signature of SQL function TRIM(). This function removes
characters included in <trim character> (binary) string from
<trim source> (binary) string until encounter a character that doesn't
belong to <trim character>. Removal occurs on the side, specified by
<trim specification>. Now, syntax is following:
TRIM([ [ <trim specification> ] [ <trim character> ] FROM ] <trim source>).

<trim specification> can be one of the following keywords: LEADING,
TRAILING and BOTH.
<trim character> is the set of trimming characters.
<trim source> is the string, that will be trimmed.
If FROM is specified, then:
1) Either <trim specification> or <trim character> or both shall be
specified.
2) If <trim specification> is not specified, then BOTH is implicit.
3) If <trim character> is not specified, then ' ' is implicit.

There can be a lot of small files with vinyl.

Added a check whether box.cfg() is called within an instance
file. If box.cfg() is missed, point a user the reason of a
fail explicitly.

Before this commit the error was look so:

/usr/bin/tarantoolctl:541: attempt to index a nil value

Closes #3953

Stop a file processing loop only when it is guaranteed that we will not
find a record that match user-provided filters later in this file. If
--replica R is provided one time and we're meet a record from R with a
LSN equal or above of a --to value, we'll stop the loop. Otherwise (no
--replica, several --replica arguments) a file will be read until an end
even if --to is provided.

Fixes #3827.

Exposed this unified code (filter_xlog() function) and wrote a unit
test.

Allow to run app-tap/tarantoolctl.test.lua w/o test-run.

Needed for #3827.

It's OK to use json format with the boot logger. As for syslog, let's
add a check to Lua's log_format() so that it fails gracefully rather
than raising an assertion.

Closes #3946

At this point, an ANALYZE statement can lead to many problems. It
was decided to temporarily ban this statement.

Closes #4069

BLOB column type is represented by SCALAR field type in terms of NoSQL.
We attempted at emulating BLOB behaviour, but such efforts turn out to
be not decent enough. For this reason, we've decided to abandon these
attempts and fairly replace it with SCALAR column type.  SCALAR column
type acts in the same way as it does in NoSQL: it is aggregator-type for
INTEGER, NUMBER and STRING types. So, column declared with this type can
contain values of these three (available in SQL) types. It is worth
mentioning that CAST operator in this case does nothing.

Still, we consider BLOB values as entries encoded in msgpack with MP_BIN
format. To make this happen, values to be operated should be represented
in BLOB form x'...' (e.g. x'000000'). What is more, there are two
built-in functions returning BLOBs: randomblob() and zeroblob().  On the
other hand, columns with STRING NoSQL type don't accept BLOB values.

Closes #4019
Closes #4023

@TarantoolBot document
Title: SQL types changes
There are couple of recently introduced changes connected with SQL
types.

Firstly, we've removed support of DATE/TIME types from parser due to
confusing behaviour of these types: they were mapped to NUMBER NoSQL
type and have nothing in common with generally accepted DATE/TIME types
(like in other DBs). In addition, all built-in functions related to
these types (julianday(), date(), time(), datetime(), current_time(),
current_date() etc) are disabled until we reimplement TIME-like types as
a native NoSQL ones (see #3694 issue).

Secondly, we've removed CHAR type (i.e. alias to VARCHAR and TEXT). The
reason is that according to ANSI SQL CHAR(len) must accept only strings
featuring length exactly equal to given in type definition. Obviously,
now we don't provide such checks. Types VARCHAR and TEXT are still
legal.

For the same reason, we've removed NUMERIC and DECIMAL types, which were
aliases to NUMBER NoSQL type. REAL, FLOAT and DOUBLE are still exist as
aliases.

Finally, we've renamed BLOB column type to SCALAR. We've decided that
all our attempts to emulate BLOB behaviour using SCALAR NoSQL type don't
seem decent enough, i.e. without native NoSQL type BLOB there always
will be inconsistency, especially taking into account possible NoSQL-SQL
interactions. In SQL SCALAR type works exactly in the same way as in
NoSQL: it can store values of INTEGER, FLOAT and TEXT SQL types at the
same time. Also, with this change behaviour of CAST operator has been
slightly corrected: now cast to SCALAR doesn't affect type of value at
all. Couple of examples:

CREATE TABLE t1 (a SCALAR PRIMARY KEY);
INSERT INTO t1 VALUES ('1');
SELECT * FROM t1 WHERE a = 1;
-- []
Result is empty set since column "a" contains string literal value '1',
not integer value 1.

CAST(123 AS SCALAR); -- Returns 123 (integer)
CAST('abc' AS SCALAR); -- Returns 'abc' (string)

Note that in NoSQL values of BLOB type defined as ones decoded in
msgpack with MP_BIN format. In SQL there are still a few ways to force
this format: declaring literal in "BLOB" format (x'...') or using one of
two built-in functions (randomblob() and zeroblob()). TEXT and VARCHAR
SQL types don't accept BLOB values:

CREATE TABLE t (a TEXT PRIMARAY KEY);
INSERT INTO t VALUES (randomblob(5));
---
- error: 'Tuple field 1 type does not match one required: expected string'
...

BLOB itself is going to be reimplemented in scope of #3650.

NMERIC and DECIMAL were allowed to be specified as column types. But in
fact, they were just synonyms for FLOAT type and mapped to NUMERIC
Tarantool NoSQL type. So, we've decided to remove this type from parser
and return back when NUMERIC will be implemented as a native type.

Part of #4019

Currently, there is no native (in Tarantool terms) types to represent
time-like types. So, until we add implementation of those types, it
makes no sense to allow to specify those types in table definition.
Note that previously they were mapped to NUMBER type. For the same
reason all built-in functions connected with DATE/TIME are disabled as
well.

Part of #4019

Add optional 'chars' parameter to string.strip, string.lstrip
and string.rstrip for specifying the unwanted characters.
Behavior modeled after the equivalent Python built-ins.

Closes: #2977

@TarantoolBot document
Title: string.strip(inp, chars)
Update the documentation for string.strip,
string.lstrip and string.rstrip to reflect
the addition of the optional param.

Replace all usage of sqlite3_, sqlite, SQLite prefixes with simple sql_
All other occurrences of SQLite are substituted with SQL word.
SQL test suit is purified as well.

Nikita Pettik suggests me that free(NULL) is no-op according to POSIX.

This is follow up of 9dbcaa3a.

Make os.exit() call tarantool_exit(), just like the signal handler does.
Now on_shutdown triggers are not run only when a fatal signal is
received.

Closes #1607

@TarantoolBot document
Title: Document box.ctl.on_shutdown triggers
on_shutdown triggers may be set similar to space:on_replace triggers:
```
box.ctl.on_shutdown(new_trigger, old_trigger)
```
The triggers will be run when tarantool exits due to receiving one of
the signals: `SIGTERM`, `SIGINT`, `SIGHUP` or when user executes
`os.exit()`.

Note that the triggers will not be run if tarantool receives a fatal
signal: `SIGSEGV`, `SIGABORT` or any signal causing immediate program
termination.

The "box.sql.execute('values(blob)')" causes an accert in the
expression processing, because the parser doesn't distinguish the
keyword "BLOB" from the binary value (in the form X'hex').

This fix adds an additional checks in the SQL grammar.
Thus the expressions such as "VALUES(BLOB)", "SELECT FLOAT"
and so on are treated as a syntax errors.

Closes #3888

It catched by ASAN at build time (lemon is executed to generate
parse.[ch]), so tarantool couldn't be built with -DENABLE_ASAN=ON.

Allow exploring rocks documentation locally with
`tarantoolctl rocks doc ...`

In context of #3753

GLOB is a legacy extension for LIKE from SQLite. As we want our SQL to
be close to ANSI SQL & LIKE to depend on collations, we do not want to
support it. This patch totally removes it from Tarantool along with any
mentions of it.

Part of #3589
Part of #3572

As a main part of introducing strict typing in SQL it is required to
prohibit typeless columns in parser's grammar. Originally, SQLite simply
assigns typeless columns to BLOB affinity. Moreover, due to historical
reasons, all columns were stored with <SCALAR> type in Tarantool core
(except for <INTEGER> when it comes to primary key).  Column type should
be defined on table creation. Allowed data types are: <TEXT>, <VARCHAR>,
<CHAR>, <BLOB>, <INT[EGER]>, <REAL>, <FLOAT>, <NUMERIC>, <DECIMAL>,
<DOUBLE> <DATE> and <DATETIME>. However, still any declared data type is
converted to one of <BLOB>, <TEXT>, <REAL> or <INTEGER> affinities.
While affinity reaches space format, it is (again) converted to
Tarantool's field type. To be more precise, table of conversions:

+----------+----------+------------+
| SQL TYPE | AFFINITY | FIELD TYPE |
+----------+----------+------------+
| FLOAT    | REAL     | NUMBER     |
| REAL     | REAL     | NUMBER     |
| DOUBLE   | REAL     | NUMBER     |
| NUMERIC  | REAL     | NUMBER     |
| DECIMAL  | REAL     | NUMBER     |
| INTEGER  | INTEGER  | INTEGER    |
| TEXT     | TEXT     | STRING     |
| VARCHAR  | TEXT     | STRING     |
| CHAR     | TEXT     | STRING     |
| BLOB     | BLOB     | SCALAR     |
| DATETIME | REAL     | NUMBER     |
| DATE     | REAL     | NUMBER     |
| TIME     | REAL     | NUMBER     |
+----------+----------+------------+

<VARCHAR> and <CHAR> types should be specified with length
(e.g. name VARCHAR(10)), but this length currently is not used when
types are processed. Only purpose is to support ANSI syntax.
The same for <NUMERIC> and <DECIMAL> - it is allowed to specify scale
and precision, but they don't affect the way they are stored in memory.

Note that patch is not self-sufficient: a lot of tests still fail due to
wrong types conversions. Fix for that comes as next two patches.

Closes #3018
Closes #3104
Closes #2494
Closes #3459