app-tap.tarantoolctl.test.lua fails after
17df9edf ('tarantoolctl: allow to start
instances with delayed box.cfg{}').

The commit fixes the test case that did check that an error is reported
if box.cfg() was not called in an instance script.

Follows up #4435.
Fixes #4448.

To compare floating point values and integers in SQL functions
compare_uint_float() and compare_int_float() are used. Unfortunately,
they contain bug connected with checking border case: that's not correct
to cast UINT64_MAX (2^64 - 1) to double. Proper way is to use exp2(2^64)
or predefined floating point constant. To not bother fixing function
which in turn may contain other tricky places, let's use instead already
verified double_compare_uint64(). So that we have unified way of
integer<->float comparison.

Closes #4422

@TarantoolBot document
Title: Introduce <WITH ENGINE> clause for CREATE TABLE statement

To allow user to specify engine as per table option, CREATE TABLE
statement has been extended with optional <WITH ENGINE = engine_name>
clause. This clause comes at the end of CREATE TABLE statement.
For instance:

CREATE TABLE t_vinyl (id INT PRIMARY KEY) WITH ENGINE = 'vinyl';

Name of engine is considered to be string literal ergo should be
enclosed in single quotation marks and be lower-cased. Note that engine
specified in WITH ENGINE clause overwrites default engine, which is set
via 'pragma sql_default_engine'.

Name duplicates are allowed for savepoints (both in our SQL
implementation and in ANSI specification). ANSI SQL states that previous
savepoint should be deleted. What is more, our doc confirms this fact
and says that "...it is released before the new savepoint is set."
Unfortunately, it's not true - currently old savepoint remains in the
list. For instance:

SAVEPOINT t;
SAVEPOINT t;
RELEASE SAVEPOINT t;
RELEASE SAVEPOINT t; -- no error is raised

Let's fix this and remove old savepoint from the list.

This procedure is processed in several steps. Firstly, we add name
to struct txn_savepoint since we should be capable of operating on named
savepoints (which in turn is SQL feature). Still, anonymous (in the sense
of name absence) savepoints are also valid. Then, we add list (as
implementation of stailq) of savepoints to struct txn: it allows us to
find savepoint by its name. Finally, we patch rollback to/release
savepoint routines: for rollback tail of the list containing savepoints
is cut (but subject of rollback routine remains in the list); for
release routine we cut tail including node being released.

Closes #4413

@TarantoolBot document
Title: update operations on decimal fields.

tuple:update and space:update now support deicmal operands for
arithmetic operations ('+' and '-'). The syntax is as usual:
```
d = box.tuple.new(decimal.new('1'))
---
...
d:update{{'+', 1, decimal.new('0.5')}}
---
- [1.5]
...
```

Insertion ('!') and assignment ('=') are also supported:
```
a = decimal.new('1')
---
...
b = decimal.new('1e10')
---
...
c = decimal.new('1e-10')
---
...
d = box.tuple.new{5, a, 6, b, 7, c, "string"}
---
...
d
---
- [5, 1, 6, 10000000000, 7, 0.0000000001, 'string']
...

d:update{{'!', 3, dec.new('1234.5678')}}
---
- [5, 1, 1234.5678, 6, 10000000000, 7, 0.0000000001, 'string']
...
d:update{{'=', -1, dec.new('0.12345678910111213')}}
---
- [5, 1, 6, 10000000000, 7, 0.0000000001, 0.12345678910111213]

When performing an arithmetic operation ('+', '-'), where either the
updated field or the operand is decimal, the result will be decimal.

When both the updated field and the operand are decimal, the result
will, of course, be decimal.

...
```

Closes #4333

@TarantoolBot document
Title: Document decimal field type.

Decimals may now be stored in spaces. A corresponding field type is
introduced: 'decimal'. Decimal values are also allowed in 'scalar',
'any' and 'number' fields.

'decimal' field type is appropriate for both memtx HASH and TREE
indices, as well as for vinyl TREE index.

```
To create an index 'pk' over a decimal field, say
```
tarantool> box.space.test:create_index('pk', {parts={1, 'decimal'}})
---
- unique: true
  parts:
  - type: decimal
    is_nullable: false
    fieldno: 1
  id: 0
  space_id: 512
  type: TREE
  name: pk
...
```
Now you can insert some decimal values:
```
tarantool> for i = 1,10 do
         > box.space.test:insert{decimal.new((i-5)/10)}
         > end
---
...
```
tarantool> box.space.test:select{}
---
- - [-0.4]
  - [-0.3]
  - [-0.2]
  - [-0.1]
  - [0]
  - [0.1]
  - [0.2]
  - [0.3]
  - [0.4]
  - [0.5]
...
```
Decimals may alse be inserted into `scalar` and `number` fields. In this
case all the number values are sorted correctly:
```
tarantool> box.schema.space.create('test')
tarantool> box.space.test:create_index('pk', {parts={1, 'number'}})
tarantool> box.space.test:insert{-1.0001, 'number'}
---
- [-1.0001, 'number']
...

tarantool> box.space.test:insert{decimal.new(-1.00001), 'decimal'}
---
- [-1.00001, 'decimal']
...

tarantool> box.space.test:insert{-1, 'number'}
---
- [-1, 'number']
...

tarantool> box.space.test:insert{decimal.new(-0.999), 'decimal'}
---
- [-0.999, 'decimal']
...

tarantool> box.space.test:insert{-0.998, 'number'}
---
- [-0.998, 'number']
...

tarantool> box.space.test:insert{-0.9, 'number'}
---
- [-0.9, 'number']
...

tarantool> box.space.test:insert{-0.95, 'number'}
---
- [-0.95, 'number']
...

tarantool> box.space.test:insert{decimal.new(-0.92), 'decimal'}
---
- [-0.92, 'decimal']
...

tarantool> box.space.test:insert{decimal.new(-0.971), 'decimal'}
---
- [-0.971, 'decimal']
...

tarantool> box.space.test:select{}
---
- - [-1.0001, 'number']
  - [-1.00001, 'decimal']
  - [-1, 'number']
  - [-0.999, 'decimal']
  - [-0.998, 'number']
  - [-0.971, 'decimal']
  - [-0.95, 'number']
  - [-0.92, 'decimal']
  - [-0.9, 'number']
...

```
Uniqueness is also preserved between decimals and other number types:
```
tarantool> box.space.test:insert{-0.92}
---
- error: Duplicate key exists in unique index 'pk' in space 'test'
...

tarantool> box.space.test:insert{decimal.new(-0.9)}
---
- error: Duplicate key exists in unique index 'pk' in space 'test'
...

```

You can also set decimal fields in space format:
```
tarantool> _ = box.schema.space.create('test')
---
...

tarantool> _ = box.space.test:create_index('pk')
---
...

tarantool> box.space.test:format{{name='id', type='unsigned'}, {name='balance', type='decimal'}}
---
...

tarantool> box.space.test:insert{1}
---
- error: Tuple field 2 required by space format is missing
...

tarantool> box.space.test:insert{1, 'string'}
---
- error: 'Tuple field 2 type does not match one required by operation: expected decimal'
...

tarantool> box.space.test:insert{1, 1.2345}
---
- error: 'Tuple field 2 type does not match one required by operation: expected decimal'
...

tarantool> box.space.test:insert{1, decimal.new('1337.420')}
---
- [1, 1337.420]
...

```

Update decNumber library, add methods to convert decimals to uint64_t
and int64_t, add unit tests.
Also replace decimal_round() function with decimal_round_with_mode() to
allow setting rounding mode.
We need to round with mode DEC_ROUND_DOWN in to_int64 conversions in
order to be consistent with double to int conversions.
It will be needed to compute hints for decimal fields.

Prerequisite #4333

This patch adds the methods necessary to encode and decode decimals to
MsgPack. MsgPack EXT type (MP_EXT) together with a new extension type
MP_DECIMAL is used as a record header.

The decimal MsgPack representation looks like this:
+--------+-------------------+------------+===============+
| MP_EXT | length (optional) | MP_DECIMAL | PackedDecimal |
+--------+-------------------+------------+===============+
The whole record may be encoded and decoded with
mp_encode_decimal() and mp_decode_decimal(). This is equivalent to
performing mp_encode_extl()/mp_decode_extl() on the first 3 fields and
decimal_pack/unpack() on the PackedDecimal field.

It is also possible to decode and encode decimals to msgpack from lua,
which means you can insert decimals into spaces, but only into unindexed
fields for now.

Follow up #692
Part of #4333

Prior to this patch format checking was broken for 'i' (integer) and 'N'
(big-endian integer). pickle.pack() rejected negative integers with
these formats. Fix this

When a number having a positive exponent is encoded, the internal
decPackedFromNumber function returns a negative scale, which differs
from the scale, returned by decimal_scale(). This leads to errors in
decoding. Account for negative scale in decimal_pack() and
decimal_unpack().

Follow-up #692

Previously decimal comparison with nil failed with following error:
`expected decimal, number or string as 2 argument`.
Fix this. Throw a more verbose error in case of '>', '<', '>=', '<='
and fix equality check.

Follow-up #692

Hold libcurl-7.65.3. This version is not affected by the following
issues:

* #4180 ('httpc: redirects are broken with libcurl-7.30 and older');
* #4389 ('libcurl memory leak');
* #4397 ('HTTPS seem to be unstable').

After this patch libcurl will be statically linked when
ENABLE_BUNDLED_LIBCURL option is set. This option is set by default.

Closes #4318

@TarantoolBot document
Title: Tarantool dependency list was changed

* Added build dependencies: autoconf, automake, libtool, zlib-devel
  (zlib1g-dev on Debian).
* Added runtime dependencies: zlib (zlib1g on Debian).
* Removed build dependencies: libcurl-devel (libcurl4-openssl-dev on
  Debian).
* Removed runtime dependencies: curl.

The reason is that now we use compiled-in libcurl: so we don't depend on
a system libcurl, but inherit its dependencies.

There's no reason to use a special method instead of the generic
space_execute_dml for applying rows received from a master during the
initial join stage. Moreover, using the special method results in not
running space.before_replace trigger, which makes it impossible to, for
example, update space engine on a replica, see the on_schema_init test
of the replication test suite.

So this patch removes the special method altogether and makes the code
that used it switch to space_execute_dml.

Closes #4417

This patch does two things: renames existing scalar min/max
functions and reserves names for them in NoSQL cache.

Moreover it is an important step to get rid of function's name
overloading required for replace FuncDef cache with Tarantool's
function cache.

Closes #4405
Needed for #2200, #4113, #2233

@TarantoolBot document
Title: Scalar functions MIN/MAX are renamed to LEAST/GREATEST

The MIN/MAX functions are typically used only as aggregate
functions in other RDBMS(MSSQL, Postgress, MySQL, Oracle) while
Tarantool's SQLite legacy code use them also in meaning
GREATEST/LEAST scalar function. Now it fixed.

Vdbe field ref is a dynamic index over tuple fields storing
offsets to each field and filling the offset array on demand.
It is highly used in SQL, because it strongly relies on fast and
repetitive access to any field, not only indexed.

There is an optimisation for the case when a requested field
fieldno is indexed, and the tuple itself stores offset to the
field in its own small field map, used by indexes. vdbe_field_ref
then uses that map to retrieve offset value without decoding
anything. But when SQL requests any field > fieldno, then
vdbe_field_ref decodes the tuple from the beginning in the worst
case. Even having previously accessed fieldno. But it could
start decoding from the latter.

An updated vdbe_field_ref fetcher class uses a bitmask of
initialized slots to use pre-calculated offsets when possible.
This speed-ups SQL in some corner case and doesn't damage
performance in general scenarios.

Closes #4267

It was forgotten to update result file of sql/bind.test.lua
in previous patch. Let's fix that and refresh sql/bind.result with
up-to-date results.

It was decided that for all integer literals we would return "INTEGER"
type, not "UNSIGNED". Accidentally, after substitution of unsigned
binding value type was set to "UNSIGNED". Let's fix that and set
"INTEGER" type.

https://github.com/LuaJIT/LuaJIT/issues/505

wal_sync() simply flushes the tx<->wal request queue, it doesn't
guarantee that all pending writes are successfully committed to disk.
This works for now, but in order to implement replica join off the
current read view, we need to make sure that all pending writes have
been persisted and won't be rolled back before we can use memtx
snapshot iterators. So this patch adds a return code to wal_sync():
since now on it returns -1 if rollback is in progress and hence
some in-memory changes are going to be rolled back. We will use
this method after opening memtx snapshot iterators used for feeding
a consistent read view a newly joined replica so as to ensure that
changes frozen by the iterators have made it to the disk.

Found that on high loaded hosts the test flaky fails at:

[004] --- app/socket.result	Mon Jul 15 07:18:57 2019
[004] +++ app/socket.reject	Tue Jul 16 16:37:35 2019
[004] @@ -1118,7 +1118,7 @@
[004]  ...
[004]  ch:get(1)
[004]  ---
[004] -- true
[004] +- null
[004]  ...
[004]  s:error()
[004]  ---

Found that the test in previous was used for testing the
the channel get() function timeout and the error occurred
on it, but later the checking error changed to:
"builtin/socket.lua: attempt to use closed socket" and the
test became not correct. Because for now it passes when the
socket read function runs before the socket closing, but in
this way read call doesn't wait. In the other way on high
loaded hosts the close call may occure before read call and
in this way read call halts and socket get call returns
'null'. As seen both ways are not correct to check the error.
Decided to remove this subtest.

Check commit ba7a4fee ("Add tests for socket:close closes #360")

Fixes #4354

Now, as vy_lsm and index are basically the same object, we can implement
reference counting right in struct index. This will allow us to prevent
an index from destruction when a space object it belongs to is freed
anywhere in the code, not just in vinyl.

Part of #4403

@TarantoolBol document
Title: Document decimal modulo operator

There is now a modulo operator for decimal numbers:
```
a = decimal.new(172.51)
a % 1
---
- '0.51'
...
a % 0.3
---
- '0.01'
...
a % 13.27
---
- '0.00'
...
a % 173
---
- '172.51'
...
a % 72
---
- '28.51'
...
720 % a
---
- '29.96'
...
```

JSON paths has no a strict standard, but definitely there is no
an implementation, allowing to omit '.' after [], if a next token
is a key. For example:

    [1]key

is invalid. It should be written like that:

    [1].key

Strangely, but we even had tests on the invalid case.

Closes #4419

In one place that test sends a packet and expects that it has
arrived two lines below. Under high load it may take more time.
The patch makes the test explicitly wait for the packet arrival.

Closes #4392

There is compile time option PACKAGE in cmake to define
current build distribution info. By default it's
"Tarantool" for the community version and "Tarantool Enterprise"
for the enterprise version.

It's displayed in console greeting and in `box.info().package`,
but, unfortunately, it can't be accessed from Lua before `box.cfg`.

This patch exposes `require('tarantool').package`.

Close #4408

@TarantoolBot document
Title: Extend module "tarantool" with the field "package"

Beside from build info and version, module "tarantool" now provides
"package" field. By default it equals string "Tarantool", but
can differ for other distributions like "Tarantool Enterprise".

Example:

```console
tarantool> require('tarantool')
---
- version: 2.3.0-3-g302bb3241
  build:
    target: Linux-x86_64-RelWithDebInfo
    options: cmake . -DCMAKE_INSTALL_PREFIX=/opt/tarantool-install
-DENABLE_BACKTRACE=ON
    mod_format: so
    flags: ' -fexceptions -funwind-tables -fno-omit-frame-pointer
-fno-stack-protector
      -fno-common -fopenmp -msse2 -std=c11 -Wall -Wextra
-Wno-strict-aliasing -Wno-char-subscripts
      -Wno-format-truncation -fno-gnu89-inline -Wno-cast-function-type'
    compiler: /usr/bin/cc /usr/bin/c++
  pid: 'function: 0x40016cd0'
  package: Tarantool
  uptime: 'function: 0x40016cb0'
...

```

The _func_index space trigger used to reject an insertion of a
tuple that defines an invalid functional index.
As insertion in _index space had been completed before, a garbage
is kept in _index space in such case.

We need to do something with the yelding _func_index trigger(that
rebuilds an index) to wrap all index creation operation in DDL
transaction in further patches (because only the first DDL
operation may yeld now).

This problem could be trivially solved with preparatory
initialization of index_def function ponter: the memtx_tree
index construction would perform all required job in such case.
Therefore the following index rebuild in _func_index trigger
becomes redundant and should be omitted.

In other words, a trivial prefetch operation makes possible
a transactional index creation (with space:create_index operation).

As for index construction during recovery (a lack of function
cache during recovery was the main motivation to introduce
_func_index space), it's workflow is kept unchanged.

Follow up #1250
Needed for #4348
Closes #4401

In case an error occurs between relay_start() and cord_costart() in
relay_subscribe(), the relay status won't be reset to STOPPED. As a
result, any further attempt to re-subscribe will fail with ER_CFG:
duplicate connection with the same replica UUID. This may happen, for
example, if the WAL directory happens to be temporarily inaccessible on
the master.

Closes #4399

It was decided that null value in SQL by default should be of type
boolean. Justification of such change is that according to ANSI boolean
type in fact has three different values: true, false and unknown. The
latter is basically an alias to null value.

If during creation of constraint its name is not specified, then it is
generated automatically. Occasionally, default names for each type of
constraint turn out to be different. This patch makes them follow the
same pattern: "shortcut"_unnamed_"table_name"_"ordinal_numb". For
instance: fk_unnamed_T1_1 or ck_unnamed_T1_3 etc

If UNIQUE constraint is specified in CREATE TABLE statement and it has
given name, one is mangled with pattern "unique_%s_%d", where %s is
original name of constraint and %d - current iid. For instance:

CREATE TABLE t (id INT PRIMARY KEY, a INT CONSTRAINT i1 UNIQUE);

This statement results in secondary index creation with name
"unique_I1_1". Justification for mangling is that constraint namespace
should be independent from index namespace. However, ALTER TABLE ADD
CONSTRAINT UNIQUE which is alias to CREATE INDEX doesn't mangle name.
What is more, users may wonder why name of index is different from
name of created constraint. Hence, it has been decided to remove this
mangling and create index exactly with specified constraint's name.

Its purpose is to copy integer value from one memory cell to another.
In other words, it is particular case of OP_SCopy instruction. Since it
is used only during creation of AUTOINCREMENT property, it seems to be
reasonable to replace it with a bit general OP_SCopy and erase
OP_IntCopy at all reducing size of SQL codebase.

In accordance with the Lua coding style in Tarantool, all errors
returned in Lua should be returned using 'return nil, error'.
However, box.execute() throws an exception in case of an error.
This patch causes box.execute() to return an error, as described
in the coding style.

Closes #4390

According to ANSI, LIKE should match characters taking into
account passed collation.

ISO/IEC JTC 1/SC 32 2011, Part 2: Foundation, 8.5

Closes #3589

@TarantoolBot document
Title: LIKE depends on collations

Now <LIKE> operator depends on arguments' collations. Collation to be
used is determined by common rules: if explicit collation or collations
(and in case they are compatible) is specified, then it is used to
process pattern matching; otherwise implicit collation of arguments are
checked to be compatible and (in case they are) resulting collation is
used. Moreover, it means that PRAGMA "case_sensitive_like" has been
removed.

According to ANSI, LIKE should match characters taking into account
collations of arguments, and this is done in the next patch. In turn,
this patch makes LIKE be always case sensitive and erases
case_sensitive_like pragma. Alongside with it, code related to no-case
LIKE optimization is removed as well.

Part of #3589

Current patch introduces new type available in SQL:
 - VARBINARY now is reserved keyword;
 - Allow to specify VARBINARY column and CAST type;
 - All literals which start from 'x' are assumed to be of this type;
 - There's no available implicit or explicit conversions between
   VARBINARY and other types;
 - Under the hood all values of VARBINARY type are stored as MP_BIN
   msgpack format type.

Closes #4206

According to ANSI specification, if TRIM function accepts binary string
and trim octet is not specified, then it is implicitly set to X'00'.
Before this patch trim octet was set to ' ' both for string and binary
string arguments. In turn, ' ' is equal to X'20' in hex representation.
Hence, TRIM function cut wrong characters:

TRIM(X'004420') -> X‘0044'

This patch sets default trim octet to X'00' for binary string arguments.

Part of #4206

We are going to introduce new column type 'VARBINARY', which allows to
store values with MP_BIN msgpack format. On the other hand, now it is also
possible to meet this type: all literals in form of x'...' are
supposed to be inserted into SCALAR column type exactly with MP_BIN
encoding. Prior to this moment type of such values (encoded as MP_BIN)
was called 'blob'. Thus, let's fix all visible to user messages using
'varbinary' name of type instead of 'blob'.

Before this patch, resulting type for CASE-WHEN statement was assumed to
be the same as type of argument of first THEN clause. Obviously, it is
wrong and could lead to sad consequence (e.g. creating ephemeral table
with inconsistent format). To deal with this, we check all THEN
arguments: if all of them have the same type, then such type will be
resulting of the whole statement; if at least two types are different,
we can't determine actual resulting type during compilation stage and
assign SCALAR as a most general type in SQL now.

Need for #4206

Box recently added support of transactional DDL allowing to do
any number of non-yielding DDL operations atomically. This is
really a big relief of one of the biggest pains of SQL. Before
this patch each multirow SQL DDL statement needed to prepare its
own rollback procedure for a case if something would go wrong.

Now with box support SQL wraps each DDL statement into a
transaction, and doesn't need own escape-routes in a form of
'struct save_record' and others.

Closes #4086

@TarantoolBot document
Title: SQL DDL is transactional

SQL DDL operations are atomic now. For example, if a CREATE TABLE
request fails somewhere in the middle, it won't leave any
garbage. Like a space without indexes, or unused sequences. Even
if the instance is powered off during the request.

Also, SQL DDL can be manually included into transactions, with
certain limitations - such a transaction can't yield.

For example, this is legal:

    START TRANSACTION;
    CREATE TABLE test(a INTEGER PRIMARY KEY, b INTEGER);
    CREATE INDEX test_a ON test(a);
    COMMIT;

If you want to test it in the console, then wrap it into a
function to do not get a rollback by yield, because the console
yields after each command:

    function create()
        box.execute('START TRANSACTION;')
        box.execute('CREATE TABLE test(a INTEGER PRIMARY KEY, b INTEGER);')
        box.execute('CREATE INDEX test_a ON test(a);')
        box.execute('COMMIT;')
    end

    create()

But the following example is illegal and you will get an error:

    box.execute('CREATE TABLE test(a INTEGER PRIMARY KEY, b INTEGER, c INTEGER);')
    box.execute('INSERT INTO test VALUES (1, 1, 1), (2, 2, 2), (3, 3, 3);')

    function index()
        box.execute('START TRANSACTION;')
        box.execute('CREATE INDEX test_b ON test(b);')
        box.execute('CREATE INDEX test_c ON test(c);')
        box.execute('COMMIT;')
    end

    tarantool> index()
    ---
    - error: Can not perform index build in a multi-statement transaction
    ...

The error is because an attempt to build an index on a non-empty
space leads to immediate yield.