The _vfunc system space is the sysview for the _func system space.
However, the _vfunc format is different from the _func format. This
patch makes the _vfunc format the same as the _func format.

Closes #7822

NO_DOC=bugfix

Currently, in case of recovery from an old snapshot, Tarantool allows to
perform DDL operations on an instance with non-upgraded schema.
It leads to various unpredictable errors (because the DDL code assumes
that the schema is already upgraded). This patch forbids the following
operations unless the user has the most recent schema version:
- box.schema.space.create
- box.schema.space.drop
- box.schema.space.alter
- box.schema.index.create
- box.schema.index.drop
- box.schema.index.alter
- box.schema.sequence.create
- box.schema.sequence.drop
- box.schema.sequence.alter
- box.schema.func.create
- box.schema.func.drop

Closes #7149

NO_DOC=bugfix

By default a user might not have privileges to access the _schema space,
that will cause an error during schema_needs_upgrade(), which calls
get_version(). Fix this by using C variable dd_version_id, which is
updated in the _schema.version replace trigger.

There's a special case for upgrade() during bootstrap() - triggers are
disabled during bootstrap, that's why dd_version_id is not being updated.
Handle this by passing _initial_version=1.7.5 to the upgrade function.

Part of #7149

NO_DOC=internal
NO_CHANGELOG=internal

Since the function is actually an eval, by default there should
be no execute access right in public role.

Closes tarantool/security#14

NO_DOC=bugfix

'public' role and thus guest and any newly created user has 'write'
privilege to _collation space. It is not enough to modify _collation as
one also needs 'create' privilege to add a collation and 'drop privilege
to delete one. So it is safe. Yet looks like we don't have any purpuse
of exposing _collation now.

NO_DOC=bugfix

The _session_settings space is a virtual space which only contains
information about the current session, hence, it should be accessible
by everyone without granting any additional privileges.

New users are granted public role by default: grant read,write access
on _session_settings space to public role.

Closes #6310

@TarantoolBot document
Title: public role rw access on _session_settings space

Public role (which is granted to new users by default) now has read,
write access on _session_settings_space.

Direct upgrade support from pre-1.7.5 versions was removed in commit
7d3b80e7
(Forbid upgrade from Tarantool < 1.7.5 and refactor upgrade.lua)
The reason for that was the mandatory space format checks introduced
back then. With these space format checks, old schema couldn't be
recovered on new Tarantool versions, because newer versions had
different system space formats. So old schema couldn't be upgraded
because it couldn't even be recovered.

Actually this was rather inconvenient. One had to perform an extra
upgrade step when upgrading from, say, 1.6 to 2.x: instead of
performing a direct upgrade one had to do 1.6 -> 1.10 -> 2.x upgrade
which takes twice the time.

Make it possible to boot from snapshots coming from Tarantool version
1.6.8 and above.

In order to do so, introduce before_replace triggers on system spaces,
which work during snapshot/xlog recovery. The triggers will set tuple
formats to the ones supported by current Tarantool (2.x). This way the
recovered data will have the correct format for a usual schema upgrade.

Also add upgrade_to_1_7_5() handler, which finishes transformation of
old schema to 1.7.5. The handler is fired together with other
box.schema.upgrade() handlers, so there's no user-visible behaviour
change.

Side note: it would be great to use the same technique to allow booting
from pre-1.6.8 snapshots. Unfortunately, this is not possible.

Current triggers don't break the order of schema upgrades, so 1.7.1
upgrades come before 1.7.2 and 1.7.5. This is because all the upgrades
in these versions are replacing existing tuples and not inserting new
ones, so the upgrades may be handled by the before_replace triggers.

Upgrade to 1.6.8 requires inserting new tuples: creating sysviews, like
_vspace, _vuser and so on. This can't be done from the before_replace
triggers, so we would have to run triggers for 1.7.x first which would
allow Tarantool to recover the snapshot, and then run an upgrade handler for
1.6.8. This looks really messy.

Closes #5894

This patch removes SQL built-in functions from _func. These functions
could be called directly from Lua, however all they did was returned an
error. After this patch, no SQL built-in functions can be called
directly from LUA.

Part of #6106

This patch introduces a new SQL built-in function UUID().

Closes #5886

@TarantoolBot document
Title: SQL built-in function UUID()

SQL built-in function UUID() takes zero or one argument. If no argument
is specified, a UUID v4 is generated. If the version of the UUID to
generate is specified as an argument, the function returns the new UUID
of the given version. Currently only version 4 of UUID is supported.

After this patch, the persistent functions "box.schema.user.info" and
"LUA" will have the same rights as the user who executed them.

The problem was that setuid was unnecessarily set. Because of this,
these functions had the same rights as the user who created them.
However, they must have the same rights as the user who used them.

Fixes tarantool/security#1

Check schema version (stored in box.space._schema) on start and
print a warning if it doesn't match last available schema version.
It is needed because some users forget to call
box.schema.upgrade() after Tarantool update and get stuck with an
old schema version until they encounter some hard to debug
problems.

Closes #4574

Co-developed-by: Roman Khabibov <roman.habibov@tarantool.org>

Part of #4681

Reviewed-by: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Co-authored-by: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>

The bug was in an attempt to update a record in _space_sequence
in-place, to add field path and number. This was not properly
supported by the system space's trigger, and was banned in the
previous patch of this series.

But delete + tuple update + insert work fine. The patch uses them.

To test it the old disabled and heavily outdated
xlog/upgrade.test.lua was replaced with a smaller analogue, which
is supposed to be created separately for each upgrade bug.
According to the new policy of creating test files.

The patch tries to make it easy to add new upgrade tests and
snapshots. A new test should consist of fill.lua script to
populate spaces, snapshot, needed xlogs, and a .test.lua file.
Fill script and binaries should be in the same folder as test file
name, which is located in version folder. Like this:

 xlog/
 |
 + <test_name>.test.lua
 |
 +- upgrade/
    |
    +- <version>/
    |   |
    |   +-<test_name>/
    |     |
    |     +- fill.lua
    |     +- *.snap
    |     +- *.xlog

Version is supposed to say explicitly what a version files in
there have.

Closes #4771

box.internal.bootstrap() before doing anything turns off system
space triggers, because it is likely to do some hard changes
violating existing rules. And eliminates data from all system
spaces to fill it from the scratch.

Each time when a new space is added, its erasure and turning off
its triggers should have been called explicitly here. As a result
it was not done sometimes, by accident. For example, triggers
were not turned off for _sequence_data, _sequence,
_space_sequence.

Content removal wasn't done for _space_sequence.

The patch makes a generic solution which does not require manual
patching of trigger manipulation and truncation anymore.

The bug was discovered while working on #4771, although it is not
related.

This patch creates _session_settings system space. This space is
used to view and change session settings. This space is one of the
special spaces that have a "service" engine. The main idea of this
space is that it will not store tuples, but when you call the
get() or select() methods, it creates tuples on the fly. Because
of this, even if the same setting is asked, the returned tuples
will be different. In addition, this space allows you to change
the setting value using the update() method, in which case it
directly changes the setting value.

There are no settings at the moment, some will be added in the
next patch.

Part of #4511

Unicode_ci collation breaks the general
rule for objects naming, so we remove it
in version 2.3.1

Closes #4561

Now it is possible to disable and enable ck constraints in LUA.
This option is persistent. All ck constraints are constructed
in enabled state when Tarantool is configured. This ability
may be usefulwhen processed data is verified and constraints
validation is not required. For instance, during casual recovery
process there's no need to provide any checks since data is
assumed to be consistent.

Persisting is_enabled flag is an important feature.
If the option is not stored the following scenario is
possible:
- the option is turned off
- data is changed so that a constraint is violated
- the system is restarted while the option state is lost
- there is no way (even in theory) to discover it and find that
  data is incorrect.

Part of #4244

This patch does two things: renames existing scalar min/max
functions and reserves names for them in NoSQL cache.

Moreover it is an important step to get rid of function's name
overloading required for replace FuncDef cache with Tarantool's
function cache.

Closes #4405
Needed for #2200, #4113, #2233

@TarantoolBot document
Title: Scalar functions MIN/MAX are renamed to LEAST/GREATEST

The MIN/MAX functions are typically used only as aggregate
functions in other RDBMS(MSSQL, Postgress, MySQL, Oracle) while
Tarantool's SQLite legacy code use them also in meaning
GREATEST/LEAST scalar function. Now it fixed.

Closes #1260

@TarantoolBot document
Title: introduce func indexes in memtx
Now you can define a func_index using a registered persistent
function.

There are restrictions for function and key definition for
a functional index:
 - the referenced function must be persistent, deterministic
   and must return a scalar type or an array.
 - you must define key parts which describe the function return value
 - the function must return data which types match the
   defined key parts
 - the function may return multiple keys; this would be a multikey
   functional index; each key entry is indexed separately;
 - for multikey functional indexes, the key definition should
   start with part 1 and cover all returned key parts
 - key parts can't use JSON paths.
 - the function used for the functional index can not access tuple
   fields by name, only by index.

Functional index can't be primary.
It is not possible to change the used function after a functional
index is defined on it. The index must be dropped first.

Each key returned by functional index function (even when it is a
single scalar) must be returned as a table i.e. {1} and must
match the key definition.

To define a multikey functional index, create a function with
opts = {is_multikey = true} and return a table of keys.

Example:
s = box.schema.space.create('withdata')
s:format({{name = 'name', type = 'string'},
          {name = 'address', type = 'string'}})
pk = s:create_index('name', {parts = {1, 'string'}})
lua_code = [[function(tuple)
                local address = string.split(tuple[2])
                local ret = {}
                for _, v in pairs(address) do
			table.insert(ret, {utf8.upper(v)})
		end
                return ret
             end]]
box.schema.func.create('address', {body = lua_code,
                       is_deterministic = true, is_sandboxed = true,
                       opts = {is_multikey = true}})
idx = s:create_index('addr', {unique = false,
                     func = 'address',
                     parts = {{1, 'string', collation = 'unicode_ci'}}})
s:insert({"James", "SIS Building Lambeth London UK"})
s:insert({"Sherlock", "221B Baker St Marylebone London NW1 6XE UK"})
idx:select('Uk')
---
- - ['James', 'SIS Building Lambeth London UK']
  - ['Sherlock', '221B Baker St Marylebone London NW1 6XE UK']
...

Closes #4182
Closes #4219
Needed for #1260

@TarantoolBot document
Title: Persistent Lua functions

Now Tarantool supports 'persistent' Lua functions.
Such functions are stored in snapshot and are available after
restart.
To create a persistent Lua function, specify a function body
in box.schema.func.create call:
e.g. body = "function(a, b) return a + b end"

A Lua persistent function may be 'sandboxed'. The 'sandboxed'
function is executed in isolated environment:
  a. only limited set of Lua functions and modules are available:
    -assert -error -pairs -ipairs -next -pcall -xpcall -type
    -print -select -string -tonumber -tostring -unpack -math -utf8;
  b. global variables are forbidden

Finally, the new 'is_deterministic' flag allows to mark a
registered function as deterministic, i.e. the function that
can produce only one result for a given list of parameters.

The new box.schema.func.create interface is:
box.schema.func.create('funcname', <setuid = true|FALSE>,
	<if_not_exists = true|FALSE>, <language = LUA|c>,
	<body = string ('')>, <is_deterministic = true|FALSE>,
	<is_sandboxed = true|FALSE>, <comment = string ('')>)

This schema change is also reserves names for sql builtin
functions:
    TRIM, TYPEOF, PRINTF, UNICODE, CHAR, HEX, VERSION,
    QUOTE, REPLACE, SUBSTR, GROUP_CONCAT, JULIANDAY, DATE,
    TIME, DATETIME, STRFTIME, CURRENT_TIME, CURRENT_TIMESTAMP,
    CURRENT_DATE, LENGTH, POSITION, ROUND, UPPER, LOWER,
    IFNULL, RANDOM, CEIL, CEILING, CHARACTER_LENGTH,
    CHAR_LENGTH, FLOOR, MOD, OCTET_LENGTH, ROW_COUNT, COUNT,
    LIKE, ABS, EXP, LN, POWER, SQRT, SUM, TOTAL, AVG,
    RANDOMBLOB, NULLIF, ZEROBLOB, MIN, MAX, COALESCE, EVERY,
    EXISTS, EXTRACT, SOME, GREATER, LESSER, SOUNDEX,
    LIKELIHOOD, LIKELY, UNLIKELY,
    _sql_stat_get, _sql_stat_push, _sql_stat_init, LUA

A new Lua persistent function LUA is introduced to evaluate
LUA strings from SQL in future.

This names could not be used for user-defined functions.

Example:
lua_code = [[function(a, b) return a + b end]]
box.schema.func.create('summarize', {body = lua_code,
		is_deterministic = true, is_sandboxed = true})
box.func.summarize
---
- aggregate: none
  returns: any
  exports:
    lua: true
    sql: false
  id: 60
  is_sandboxed: true
  setuid: false
  is_deterministic: true
  body: function(a, b) return a + b end
  name: summarize
  language: LUA
...
box.func.summarize:call({1, 3})
---
- 4
...

@kostja: fix style, remove unnecessary module dependencies,
add comments

This reverts commit 4e3470ce.

The RFC did not pass review yet.

Closes #4182
Needed for #1260

@TarantoolBot document
Title: Persistent Lua functions

Now Tarantool supports 'persistent' Lua functions.
Such functions are stored in snapshot and are available after
restart.
To create a persistent Lua function, specify a function body
in box.schema.func.create call:
e.g. body = "function(a, b) return a + b end"

A Lua persistent function may be 'sandboxed'. The 'sandboxed'
function is executed in isolated environment:
  a. only limited set of Lua functions and modules are available:
    -assert -error -pairs -ipairs -next -pcall -xpcall -type
    -print -select -string -tonumber -tostring -unpack -math -utf8;
  b. global variables are forbidden

Finally, the new 'is_deterministic' flag allows to mark a
registered function as deterministic, i.e. the function that
can produce only one result for a given list of parameters.

The new box.schema.func.create interface is:
box.schema.func.create('funcname', <setuid = true|FALSE>,
	<if_not_exists = true|FALSE>, <language = LUA|c>,
	<body = string ('')>, <is_deterministic = true|FALSE>,
	<is_sandboxed = true|FALSE>, <comment = string ('')>)

Example:
lua_code = [[function(a, b) return a + b end]]
box.schema.func.create('sum', {body = lua_code,
		is_deterministic = true, is_sandboxed = true})
box.func.sum
---
- is_sandboxed: true
  is_deterministic: true
  id: 2
  setuid: false
  body: function(a, b) return a + b end
  name: sum
  language: LUA
...
box.func.sum:call({1, 3})
---
- 4
...

Add "_vcollation" sysview to read it from net.box. This
sysview is always readable, except when a user doesn't have
"public" role.

Needed for #3941

@TarantoolBot document
Title: box.space._vcollation

_vcollation is a system space that represents a virtual view.
The structure of its tuples is identical to that of _collation.
Tuples of this sysview is always readable, except when the user
doesn't have "public" role.

This patch introduces a new system space to persist check
constraints. The format of the new system space is

_ck_constraint (space id = 364)
[<space id> UINT, <constraint name> STR,
 <is_deferred>BOOL, <language>STR, <code>STR]

A CK constraint is local for a space, so every pair
<space id, CK name> is unique
(it is also the PK in the _ck_constraint space).

After insertion into this space, a new instance describing check
constraint is created. Check constraint holds an exspression AST.
While space features some check constraints, it isn't allowed to
be dropped. The :drop() space method firstly deletes all check
constraints and then removes an entry from the _space.

Because the space alter, the index alter and the space truncate
operations cause space recreation process, a new
RebuildCkConstrains object is introduced. This alter object
compiles a new ck constraint object, replaces and removes
an existent instances atomically (but if the assembly of some
ck constraint object fails, nothing is changed).
In fact, in scope of this patch we don't really need to recreate
a ck_constraint object in such situations (it is enough to patch
space_def pointer in AST tree like we did it before, but we are
going to recompile a VDBE that represents ck constraint in
further patches, and that operation is not safe).

The main motivation for these changes is an ability to support
ADD CHECK CONSTRAINT operation in the future. CK constraints are
easier to manage as self-sustained objects: such change is
managed with atomic insertion(unlike the current architecture).

Finally, the xfer optimization is disabled now if some space have
ck constraints. In following patches this xfer optimisation
becomes impossible, so there is no reason to rewrite this code
now.

Needed for #3691

Rather than passing 'sequence_part' along with 'sequence' on index
create/alter, pass a table with the following fields:

 - id: sequence id or name
 - field: auto increment field id or name or path in case of json index

If id is omitted, the sequence will be auto-generated (equivalent to
'sequence = true'). If field is omitted, the first indexed field is
used. Old format, i.e. passing false/true or sequence name/id instead of
a table is still supported.

Follow-up #4009

Closes #4009

@TarantoolBot document
Title: Sequence can now be set for an index part other than the first

Initially one could attach a sequence (aka autoincrement) only to the
first index part. Now it's possible to attach a sequence to any primary
index part. The part still must be integer though.

Syntax:

```
box.schema.space.create('test')
box.space.test:create_index('primary', {
    parts = {{1, 'string'}, {2, 'unsigned'}, {3, 'unsigned'}},
    sequence = true, sequence_part = 2
})
box.space.test:insert{'a', box.null, 1} -- inserts {'a', 1, 1}
```

Note, `sequence_part` option is 1-base.

If `sequence_part` is omitted, 1 is used, which assures backward
compatibility with the original behavior.

One can also attach a sequence to another index part using
`index.alter` (the code below continues the example above):

```
box.space.test.index.primary:alter{sequence_part = 3}
box.space.test:insert{'a', 1, box.null, 'x'} -- inserts {'a', 1, 2, 'x'}
```

These tables won't be used anymore and should be deleted.

Note, this patch breaks backward compatibility between 2.1.1 and
2.1.2, but that's okay as 2.1.1 was beta and we didn't recommend
anyone to use it.

Part of #2843
Follow up #4069

Adds a new set of default collations.
The collation 'unicode_ky_s1' supports the difference
between Cyrillic letters 'Е' and 'Ё'. The standard case insensitive
collation ('unicode_ci') doesn't distinguish these letters.
Adds tests for a new collations.

Closes #4007

Before the patch, collations with no strength set used
tertiary strength. But it was not easy to understand it,
because box.space._collation:select{} would return
... [1, 'unicode', 1, 'ICU', '', {}] ... for such collations.
After the patch default value is set explicitly, so
user would observe : ... [1, 'unicode', 1, 'ICU', '',
{strength='tertiary'}] ...

Closes #3573

@TarantoolBot document
Title: default collation strength is explicit tertiary now
Before the patch we already have tertiary strength is default
strength for collations, but it was implicit:
[1, 'unicode', 1, 'ICU', '', {}]
After the patch it's just become explicit:
1, 'unicode', 1, 'ICU', '', {'strength' = 'tertiary'}]

Also please fix this https://tarantool.io/en/doc/2.1/book/box/data_model/#collations
There is line saying: "unicode collation observes all weights,
from L1 to Ln (identical)" It was not true and now this fact
would just become obvious.

Tarantool could not start from the snapshot created by version
2.1.0 because the new version 2.1.1 does not support the
index.opts.sql index opt and stops the execution.
Introduced a special state OPT_DEF_LEGACY macro to ignore legacy
options and introduced migration code in upgrade.lua.

The set_system_triggers and erase routines in upgrade.lua did not
proceed actions for _fk_constraint space.

They aren't needed there as we reset them anyway once the snapshot is
replayed on initial bootstrap. OTOH having them results in calling
replica_{set,clear}_id twice on initial bootstrap, which will look weird
when I patch them to log the ids. So let's remove them from the initial
snapshot. This makes the initial bootstrap impossible to recover from as
it is, but that shouldn't be an issue since one can always bootstrap a
new instance in a normal way. This also allows us to make cluster uuid
truly immutable (currently, one can update it with REPLACE).

After patch that introduced "none" collation (a953051e),
box.internal.bootstrap() started to fail due to inability to drop
mentioned collation. Lets turn off system triggers for _collation space
in order to process its complete purification during bootstrap.

This patch introduces two new collation sequences: "none" and "binary".
Despite the fact that they use the same comparing algorithm (simple
byte-by-byte comparison), they don't mean the same. "binary" collation
get to the format if user explicitly points it: either specifies this
collation in space format manually or adds <COLLATE BINARY> clause to
column definition within CREATE TABLE statement. "none" collation is
used when user doesn't specify any collation at all. "none" collation always
comes with id == 0 and it can't be changed (since its id vastly used
under the hood as an indicator of absence of collation).
Difference between these collations is vital for ANSI SQL: mixing
"binary" with other collations is prohibited, meanwhile "none" collation
can be used alongside with others.
In this respect current patch extends list of available collations: now
not only ICU collations are allowed, but also BINARY.

Note, that in SQL some queries have changed their query plan. That
occurred due to the fact that our parser allows using <COLLATE> clause
with numeric fields:

CREATE TABLE (id INT PRIMARY KEY);
SELECT id COLLATE "binary" ...

In the example collation of LHS (id column) is NULL, but collation of
RHS is "binary". Before this patch both collations were NULL. Hence,
usage of certain indexes may not be allowed by query planner.
On the other hand, this feature is obviously broken, so that doesn't
seem to be big deal.

Needed for #3185

This patch adds an upgrade script to grant CREATE, ALTER, DROP
privileges to users which have READ+WRITE on respective objects.
This is needed after removing 1.7 compatibility mode for privileges.

Closes #3539

The space is a blackhole. It will be used for writing deferred DELETE
statements generated by vinyl compaction tasks to WAL so that we can
recover deferred DELETEs that hadn't been dumped to disk before the
server was restarted.

Since the new space doesn't depend on other system spaces, let's assign
the minimal possible id to it, i.e. 257.

Needed for #2129

When granting or revoking a privilege on an entire entity, id 0 was used
to indicate the fact that we don't grant a privilege on a single object,
but on a whole entity. This caused confusion, because for entity USER,
for example, id 0 is a valid object id (user 'guest' uses it).
Any non-zero id dedicated to this cause obviously may be confused as well.
Fix this by creating separate schema_object_types for entities:
SC_ENTITY_SPACE, SC_ENTITY_SEQUENCE, etc.

Closes #3574
Needed for #3524

This patch introduces new system space to persist foreign keys
constraints. Format of the space:

_fk_constraint (space id = 358)

[<constraint name> STR, <parent id> UINT, <child id> UINT,
 <is deferred> BOOL, <match> STR, <on delete action> STR,
 <on update action> STR, <child cols> ARRAY<UINT>,
 <parent cols> ARRAY<UINT>]

FK constraint is local to space, so every pair <FK name, child id>
is unique (and it is PK in _fk_constraint space).

After insertion into this space, new instance describing FK constraint
is created. FK constraints are held in data-dictionary as two lists
(for child and parent constraints) in struct space.

There is a list of FK restrictions:
 - At the time of FK creation parent and child spaces must exist;
 - VIEWs can't be involved into FK processing;
 - Child space must be empty;
 - Types of referencing and referenced fields must be comparable;
 - Collations of referencing and referenced fields must match;
 - Referenced fields must compose unique index;
 - Referenced fields can not contain duplicates.

Until space (child) features FK constraints it isn't allowed to be
dropped. Implicitly referenced index also can't be dropped
(and that is why parent space can't be dropped). But :drop() method
of child space firstly deletes all FK constraint (the same as SQL
triggers, indexes etc) and then removes entry from _space.

Part of #3271

Review fixes

As we would like to lookup triggers by space_id in future
on space deletion to delete associated _trigger tuples we
need to introduce new field space_id as secondary key.

Part of #3273.

Merge upgrade to 1.8.2/.4 into 2.1.0. Versions 1.8.2/.4 are
broken because of invalid _trigger creation and deprecated
versioning policy.

Vinyl/Xlog upgrade tests from 1.7.7 are broken too since they
actually contains 1.8 snapshots, so remove them. Add the test on
upgrade from 1.10 to ensure the new upgrade_to_2_1_0 works.