The _vfunc system space is the sysview for the _func system space.
However, the _vfunc format is different from the _func format. This
patch makes the _vfunc format the same as the _func format.

Closes #7822

NO_DOC=bugfix

(cherry picked from commit 707da125)

Since the function is actually an eval, by default there should
be no execute access right in public role.

Closes tarantool/security#14

NO_DOC=bugfix

(cherry picked from commit 815788c8)

The _session_settings space is a virtual space which only contains
information about the current session, hence, it should be accessible
by everyone without granting any additional privileges.

New users are granted public role by default: grant read,write access
on _session_settings space to public role.

Closes #6310

@TarantoolBot document
Title: public role rw access on _session_settings space

Public role (which is granted to new users by default) now has read,
write access on _session_settings_space.

This patch removes SQL built-in functions from _func. These functions
could be called directly from Lua, however all they did was returned an
error. After this patch, no SQL built-in functions can be called
directly from LUA.

Part of #6106

This patch introduces a new SQL built-in function UUID().

Closes #5886

@TarantoolBot document
Title: SQL built-in function UUID()

SQL built-in function UUID() takes zero or one argument. If no argument
is specified, a UUID v4 is generated. If the version of the UUID to
generate is specified as an argument, the function returns the new UUID
of the given version. Currently only version 4 of UUID is supported.

After this patch, the persistent functions "box.schema.user.info" and
"LUA" will have the same rights as the user who executed them.

The problem was that setuid was unnecessarily set. Because of this,
these functions had the same rights as the user who created them.
However, they must have the same rights as the user who used them.

Fixes tarantool/security#1

This patch creates _session_settings system space. This space is
used to view and change session settings. This space is one of the
special spaces that have a "service" engine. The main idea of this
space is that it will not store tuples, but when you call the
get() or select() methods, it creates tuples on the fly. Because
of this, even if the same setting is asked, the returned tuples
will be different. In addition, this space allows you to change
the setting value using the update() method, in which case it
directly changes the setting value.

There are no settings at the moment, some will be added in the
next patch.

Part of #4511

Unicode_ci collation breaks the general
rule for objects naming, so we remove it
in version 2.3.1

Closes #4561

Now it is possible to disable and enable ck constraints in LUA.
This option is persistent. All ck constraints are constructed
in enabled state when Tarantool is configured. This ability
may be usefulwhen processed data is verified and constraints
validation is not required. For instance, during casual recovery
process there's no need to provide any checks since data is
assumed to be consistent.

Persisting is_enabled flag is an important feature.
If the option is not stored the following scenario is
possible:
- the option is turned off
- data is changed so that a constraint is violated
- the system is restarted while the option state is lost
- there is no way (even in theory) to discover it and find that
  data is incorrect.

Part of #4244

This patch does two things: renames existing scalar min/max
functions and reserves names for them in NoSQL cache.

Moreover it is an important step to get rid of function's name
overloading required for replace FuncDef cache with Tarantool's
function cache.

Closes #4405
Needed for #2200, #4113, #2233

@TarantoolBot document
Title: Scalar functions MIN/MAX are renamed to LEAST/GREATEST

The MIN/MAX functions are typically used only as aggregate
functions in other RDBMS(MSSQL, Postgress, MySQL, Oracle) while
Tarantool's SQLite legacy code use them also in meaning
GREATEST/LEAST scalar function. Now it fixed.

Closes #1260

@TarantoolBot document
Title: introduce func indexes in memtx
Now you can define a func_index using a registered persistent
function.

There are restrictions for function and key definition for
a functional index:
 - the referenced function must be persistent, deterministic
   and must return a scalar type or an array.
 - you must define key parts which describe the function return value
 - the function must return data which types match the
   defined key parts
 - the function may return multiple keys; this would be a multikey
   functional index; each key entry is indexed separately;
 - for multikey functional indexes, the key definition should
   start with part 1 and cover all returned key parts
 - key parts can't use JSON paths.
 - the function used for the functional index can not access tuple
   fields by name, only by index.

Functional index can't be primary.
It is not possible to change the used function after a functional
index is defined on it. The index must be dropped first.

Each key returned by functional index function (even when it is a
single scalar) must be returned as a table i.e. {1} and must
match the key definition.

To define a multikey functional index, create a function with
opts = {is_multikey = true} and return a table of keys.

Example:
s = box.schema.space.create('withdata')
s:format({{name = 'name', type = 'string'},
          {name = 'address', type = 'string'}})
pk = s:create_index('name', {parts = {1, 'string'}})
lua_code = [[function(tuple)
                local address = string.split(tuple[2])
                local ret = {}
                for _, v in pairs(address) do
			table.insert(ret, {utf8.upper(v)})
		end
                return ret
             end]]
box.schema.func.create('address', {body = lua_code,
                       is_deterministic = true, is_sandboxed = true,
                       opts = {is_multikey = true}})
idx = s:create_index('addr', {unique = false,
                     func = 'address',
                     parts = {{1, 'string', collation = 'unicode_ci'}}})
s:insert({"James", "SIS Building Lambeth London UK"})
s:insert({"Sherlock", "221B Baker St Marylebone London NW1 6XE UK"})
idx:select('Uk')
---
- - ['James', 'SIS Building Lambeth London UK']
  - ['Sherlock', '221B Baker St Marylebone London NW1 6XE UK']
...

Closes #4182
Closes #4219
Needed for #1260

@TarantoolBot document
Title: Persistent Lua functions

Now Tarantool supports 'persistent' Lua functions.
Such functions are stored in snapshot and are available after
restart.
To create a persistent Lua function, specify a function body
in box.schema.func.create call:
e.g. body = "function(a, b) return a + b end"

A Lua persistent function may be 'sandboxed'. The 'sandboxed'
function is executed in isolated environment:
  a. only limited set of Lua functions and modules are available:
    -assert -error -pairs -ipairs -next -pcall -xpcall -type
    -print -select -string -tonumber -tostring -unpack -math -utf8;
  b. global variables are forbidden

Finally, the new 'is_deterministic' flag allows to mark a
registered function as deterministic, i.e. the function that
can produce only one result for a given list of parameters.

The new box.schema.func.create interface is:
box.schema.func.create('funcname', <setuid = true|FALSE>,
	<if_not_exists = true|FALSE>, <language = LUA|c>,
	<body = string ('')>, <is_deterministic = true|FALSE>,
	<is_sandboxed = true|FALSE>, <comment = string ('')>)

This schema change is also reserves names for sql builtin
functions:
    TRIM, TYPEOF, PRINTF, UNICODE, CHAR, HEX, VERSION,
    QUOTE, REPLACE, SUBSTR, GROUP_CONCAT, JULIANDAY, DATE,
    TIME, DATETIME, STRFTIME, CURRENT_TIME, CURRENT_TIMESTAMP,
    CURRENT_DATE, LENGTH, POSITION, ROUND, UPPER, LOWER,
    IFNULL, RANDOM, CEIL, CEILING, CHARACTER_LENGTH,
    CHAR_LENGTH, FLOOR, MOD, OCTET_LENGTH, ROW_COUNT, COUNT,
    LIKE, ABS, EXP, LN, POWER, SQRT, SUM, TOTAL, AVG,
    RANDOMBLOB, NULLIF, ZEROBLOB, MIN, MAX, COALESCE, EVERY,
    EXISTS, EXTRACT, SOME, GREATER, LESSER, SOUNDEX,
    LIKELIHOOD, LIKELY, UNLIKELY,
    _sql_stat_get, _sql_stat_push, _sql_stat_init, LUA

A new Lua persistent function LUA is introduced to evaluate
LUA strings from SQL in future.

This names could not be used for user-defined functions.

Example:
lua_code = [[function(a, b) return a + b end]]
box.schema.func.create('summarize', {body = lua_code,
		is_deterministic = true, is_sandboxed = true})
box.func.summarize
---
- aggregate: none
  returns: any
  exports:
    lua: true
    sql: false
  id: 60
  is_sandboxed: true
  setuid: false
  is_deterministic: true
  body: function(a, b) return a + b end
  name: summarize
  language: LUA
...
box.func.summarize:call({1, 3})
---
- 4
...

@kostja: fix style, remove unnecessary module dependencies,
add comments

This reverts commit 4e3470ce.

The RFC did not pass review yet.

Closes #4182
Needed for #1260

@TarantoolBot document
Title: Persistent Lua functions

Now Tarantool supports 'persistent' Lua functions.
Such functions are stored in snapshot and are available after
restart.
To create a persistent Lua function, specify a function body
in box.schema.func.create call:
e.g. body = "function(a, b) return a + b end"

A Lua persistent function may be 'sandboxed'. The 'sandboxed'
function is executed in isolated environment:
  a. only limited set of Lua functions and modules are available:
    -assert -error -pairs -ipairs -next -pcall -xpcall -type
    -print -select -string -tonumber -tostring -unpack -math -utf8;
  b. global variables are forbidden

Finally, the new 'is_deterministic' flag allows to mark a
registered function as deterministic, i.e. the function that
can produce only one result for a given list of parameters.

The new box.schema.func.create interface is:
box.schema.func.create('funcname', <setuid = true|FALSE>,
	<if_not_exists = true|FALSE>, <language = LUA|c>,
	<body = string ('')>, <is_deterministic = true|FALSE>,
	<is_sandboxed = true|FALSE>, <comment = string ('')>)

Example:
lua_code = [[function(a, b) return a + b end]]
box.schema.func.create('sum', {body = lua_code,
		is_deterministic = true, is_sandboxed = true})
box.func.sum
---
- is_sandboxed: true
  is_deterministic: true
  id: 2
  setuid: false
  body: function(a, b) return a + b end
  name: sum
  language: LUA
...
box.func.sum:call({1, 3})
---
- 4
...

Add "_vcollation" sysview to read it from net.box. This
sysview is always readable, except when a user doesn't have
"public" role.

Needed for #3941

@TarantoolBot document
Title: box.space._vcollation

_vcollation is a system space that represents a virtual view.
The structure of its tuples is identical to that of _collation.
Tuples of this sysview is always readable, except when the user
doesn't have "public" role.

This patch introduces a new system space to persist check
constraints. The format of the new system space is

_ck_constraint (space id = 364)
[<space id> UINT, <constraint name> STR,
 <is_deferred>BOOL, <language>STR, <code>STR]

A CK constraint is local for a space, so every pair
<space id, CK name> is unique
(it is also the PK in the _ck_constraint space).

After insertion into this space, a new instance describing check
constraint is created. Check constraint holds an exspression AST.
While space features some check constraints, it isn't allowed to
be dropped. The :drop() space method firstly deletes all check
constraints and then removes an entry from the _space.

Because the space alter, the index alter and the space truncate
operations cause space recreation process, a new
RebuildCkConstrains object is introduced. This alter object
compiles a new ck constraint object, replaces and removes
an existent instances atomically (but if the assembly of some
ck constraint object fails, nothing is changed).
In fact, in scope of this patch we don't really need to recreate
a ck_constraint object in such situations (it is enough to patch
space_def pointer in AST tree like we did it before, but we are
going to recompile a VDBE that represents ck constraint in
further patches, and that operation is not safe).

The main motivation for these changes is an ability to support
ADD CHECK CONSTRAINT operation in the future. CK constraints are
easier to manage as self-sustained objects: such change is
managed with atomic insertion(unlike the current architecture).

Finally, the xfer optimization is disabled now if some space have
ck constraints. In following patches this xfer optimisation
becomes impossible, so there is no reason to rewrite this code
now.

Needed for #3691

Rather than passing 'sequence_part' along with 'sequence' on index
create/alter, pass a table with the following fields:

 - id: sequence id or name
 - field: auto increment field id or name or path in case of json index

If id is omitted, the sequence will be auto-generated (equivalent to
'sequence = true'). If field is omitted, the first indexed field is
used. Old format, i.e. passing false/true or sequence name/id instead of
a table is still supported.

Follow-up #4009

Closes #4009

@TarantoolBot document
Title: Sequence can now be set for an index part other than the first

Initially one could attach a sequence (aka autoincrement) only to the
first index part. Now it's possible to attach a sequence to any primary
index part. The part still must be integer though.

Syntax:

```
box.schema.space.create('test')
box.space.test:create_index('primary', {
    parts = {{1, 'string'}, {2, 'unsigned'}, {3, 'unsigned'}},
    sequence = true, sequence_part = 2
})
box.space.test:insert{'a', box.null, 1} -- inserts {'a', 1, 1}
```

Note, `sequence_part` option is 1-base.

If `sequence_part` is omitted, 1 is used, which assures backward
compatibility with the original behavior.

One can also attach a sequence to another index part using
`index.alter` (the code below continues the example above):

```
box.space.test.index.primary:alter{sequence_part = 3}
box.space.test:insert{'a', 1, box.null, 'x'} -- inserts {'a', 1, 2, 'x'}
```

These tables won't be used anymore and should be deleted.

Note, this patch breaks backward compatibility between 2.1.1 and
2.1.2, but that's okay as 2.1.1 was beta and we didn't recommend
anyone to use it.

Part of #2843
Follow up #4069

Adds a new set of default collations.
The collation 'unicode_ky_s1' supports the difference
between Cyrillic letters 'Е' and 'Ё'. The standard case insensitive
collation ('unicode_ci') doesn't distinguish these letters.
Adds tests for a new collations.

Closes #4007

Before the patch, collations with no strength set used
tertiary strength. But it was not easy to understand it,
because box.space._collation:select{} would return
... [1, 'unicode', 1, 'ICU', '', {}] ... for such collations.
After the patch default value is set explicitly, so
user would observe : ... [1, 'unicode', 1, 'ICU', '',
{strength='tertiary'}] ...

Closes #3573

@TarantoolBot document
Title: default collation strength is explicit tertiary now
Before the patch we already have tertiary strength is default
strength for collations, but it was implicit:
[1, 'unicode', 1, 'ICU', '', {}]
After the patch it's just become explicit:
1, 'unicode', 1, 'ICU', '', {'strength' = 'tertiary'}]

Also please fix this https://tarantool.io/en/doc/2.1/book/box/data_model/#collations
There is line saying: "unicode collation observes all weights,
from L1 to Ln (identical)" It was not true and now this fact
would just become obvious.

Tarantool could not start from the snapshot created by version
2.1.0 because the new version 2.1.1 does not support the
index.opts.sql index opt and stops the execution.
Introduced a special state OPT_DEF_LEGACY macro to ignore legacy
options and introduced migration code in upgrade.lua.

They aren't needed there as we reset them anyway once the snapshot is
replayed on initial bootstrap. OTOH having them results in calling
replica_{set,clear}_id twice on initial bootstrap, which will look weird
when I patch them to log the ids. So let's remove them from the initial
snapshot. This makes the initial bootstrap impossible to recover from as
it is, but that shouldn't be an issue since one can always bootstrap a
new instance in a normal way. This also allows us to make cluster uuid
truly immutable (currently, one can update it with REPLACE).

To fix the warning on bootstrap:

  xlog.c:1920 E> can't open tx: bootstrap: has some data after eof marker at 1906

After patch that introduced "none" collation (a953051e),
box.internal.bootstrap() started to fail due to inability to drop
mentioned collation. Lets turn off system triggers for _collation space
in order to process its complete purification during bootstrap.

This patch introduces two new collation sequences: "none" and "binary".
Despite the fact that they use the same comparing algorithm (simple
byte-by-byte comparison), they don't mean the same. "binary" collation
get to the format if user explicitly points it: either specifies this
collation in space format manually or adds <COLLATE BINARY> clause to
column definition within CREATE TABLE statement. "none" collation is
used when user doesn't specify any collation at all. "none" collation always
comes with id == 0 and it can't be changed (since its id vastly used
under the hood as an indicator of absence of collation).
Difference between these collations is vital for ANSI SQL: mixing
"binary" with other collations is prohibited, meanwhile "none" collation
can be used alongside with others.
In this respect current patch extends list of available collations: now
not only ICU collations are allowed, but also BINARY.

Note, that in SQL some queries have changed their query plan. That
occurred due to the fact that our parser allows using <COLLATE> clause
with numeric fields:

CREATE TABLE (id INT PRIMARY KEY);
SELECT id COLLATE "binary" ...

In the example collation of LHS (id column) is NULL, but collation of
RHS is "binary". Before this patch both collations were NULL. Hence,
usage of certain indexes may not be allowed by query planner.
On the other hand, this feature is obviously broken, so that doesn't
seem to be big deal.

Needed for #3185

The space is a blackhole. It will be used for writing deferred DELETE
statements generated by vinyl compaction tasks to WAL so that we can
recover deferred DELETEs that hadn't been dumped to disk before the
server was restarted.

Since the new space doesn't depend on other system spaces, let's assign
the minimal possible id to it, i.e. 257.

Needed for #2129

When granting or revoking a privilege on an entire entity, id 0 was used
to indicate the fact that we don't grant a privilege on a single object,
but on a whole entity. This caused confusion, because for entity USER,
for example, id 0 is a valid object id (user 'guest' uses it).
Any non-zero id dedicated to this cause obviously may be confused as well.
Fix this by creating separate schema_object_types for entities:
SC_ENTITY_SPACE, SC_ENTITY_SEQUENCE, etc.

Closes #3574
Needed for #3524

This patch introduces new system space to persist foreign keys
constraints. Format of the space:

_fk_constraint (space id = 358)

[<constraint name> STR, <parent id> UINT, <child id> UINT,
 <is deferred> BOOL, <match> STR, <on delete action> STR,
 <on update action> STR, <child cols> ARRAY<UINT>,
 <parent cols> ARRAY<UINT>]

FK constraint is local to space, so every pair <FK name, child id>
is unique (and it is PK in _fk_constraint space).

After insertion into this space, new instance describing FK constraint
is created. FK constraints are held in data-dictionary as two lists
(for child and parent constraints) in struct space.

There is a list of FK restrictions:
 - At the time of FK creation parent and child spaces must exist;
 - VIEWs can't be involved into FK processing;
 - Child space must be empty;
 - Types of referencing and referenced fields must be comparable;
 - Collations of referencing and referenced fields must match;
 - Referenced fields must compose unique index;
 - Referenced fields can not contain duplicates.

Until space (child) features FK constraints it isn't allowed to be
dropped. Implicitly referenced index also can't be dropped
(and that is why parent space can't be dropped). But :drop() method
of child space firstly deletes all FK constraint (the same as SQL
triggers, indexes etc) and then removes entry from _space.

Part of #3271

Review fixes

As we would like to lookup triggers by space_id in future
on space deletion to delete associated _trigger tuples we
need to introduce new field space_id as secondary key.

Part of #3273.

Merge upgrade to 1.8.2/.4 into 2.1.0. Versions 1.8.2/.4 are
broken because of invalid _trigger creation and deprecated
versioning policy.

Vinyl/Xlog upgrade tests from 1.7.7 are broken too since they
actually contains 1.8 snapshots, so remove them. Add the test on
upgrade from 1.10 to ensure the new upgrade_to_2_1_0 works.

Introduce _vsequence system space.

Prerequisite of #3250

_schema represented as key-value storage for various
values common for Tarantool, like next id for space creation.
SQL requires format to be fully specified for columns being
access. Unfortunatelly, _schema is inserted into _space
before _space's format is set and since DD triggers are
disabled during upgrade, format for _schema tuple in _space
stays the same. So, set value nullable field in upgrade,
regenerate initial snap, update tests.
Also, as far as _schema's tuple in _space is not updated:
relax fieldno check in sql.c

If Tarantool is being starteed using data from older
version, where stat tables were not system, it will definitely
crash, since SQL statistics analyzer will try to access non-
existing statistics tables (which will be added during upgrade).

To avoid such crash: create dummy statistics spaces during
schema init.

Also, fix types for statistics space format: prohibit null values
for statistic values and make sample format stronger.

  Making `stat[14]` a system spaces enables us to allow users to create
spaces which starts with `_`, because it cannot affect internal state
anymore.
  Since now the only constraints on table names is:
    - consists of printable symbols only
    - length < 65k

Closes #2126

Add a role which contains all ACLs.

Fixes gh-3022.

Useful for any quick start:

box.schema.user.grant('guest', 'super')

Add system privileges 'session' and 'usage'

* 'session' privilege lets user connect to database server
* 'usage' privilege lets user use his/her rights on database objects
* Both privileges are assigned to all users by default.

Implementation details:
* system privileges are special grant rights to 'universe'.
Therefore, they can be granted only by admin. Because of this fact,
during creation or deletion of user, we have to switch to 'admin' to
grant or revoke these rights.

Important changes:
* changed bootstrap.snap due to need to start admin with new privileges
* added auto upgrade script for 1.7.7

Fixes gh-2898.

With contributions by @kostja.

+ Don't use id=0 for collations

Follow up #2649

Add a new space that contains collation definitions for future
index collation and more.

Add a collation cache for fast retrieving a collation object by
its name.

Needed for #2649

The value returned by space.auto_increment() is not guaranteed to be
monotonically growing, because this method uses max+1 for the next
generated value, which can be less than a value generated before in case
the max key is deleted. Besides, calling max() may be expensive, e.g. it
requires a disk access in case of Vinyl. So this patch implements auto
increment based on persistent sequences.

To be used for auto increment, a sequence needs to be attached to a
space. To attach an existing sequence to a space, pass 'sequence=ID' in
index options when creating or modifying the primary key. Here ID is the
name or the numeric identifier of the sequence to attach. Note,
sequences can be attached only to spaces with integer primary key.
After a sequence is attached, it will be used for auto increment when
the user passes 'nil' for the primary key value. If the user passes a
value different from 'nil', the value will be used to update the
attached sequence.

To detach a sequence from the space it was attached to, either drop the
primary key of the space or alter the primary key with 'sequence=false'.

It is not necessary to pre-create a sequence to use it for auto
increment - a sequence can be automatically generated. To generate a
sequence automatically, pass 'sequence=true' in index options.
Automatically generated sequences are named '_auto_ID', where ID is the
space id, and removed when the space is dropped or the sequence is
detached.

A sequence cannot be dropped as long as it is attached to any spaces.

Closes #389