This patch makes schema_object_type to be exported.

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Rename the members as well. Keep uint16_t in box_check_acess_space
as it is for other exported functions

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

1. Current commit introduces 'box_auth_data_prepare()' to prepare
   a data string for any supported authentication methods.
2. The user name argument is refactored in the auth methods: the
   null-terminated string is replaced with an address range approach.
   Now Rust users don't need to re-allocate username with CString.
3. Password length type was set to uint32_t (previously it was size_t,
   int, uint32_t for different functions). Tarantool uses murmur3a,
   so all the hashed strings should be up to 32 bit long.

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Introduce fully temporary spaces: same as data-temporary space but with
temporary metadata. Basically temporary spaces now do not exist on
restart and do not exist on replicas. They can also be created, altered
and deleted when box.cfg.read_only = true.

To avoid conflicts with spaces created on replicas, the temporary
space ids by default start in a special range starting at
BOX_SPACE_ID_TEMPORARY_MIN.

Temporary spaces currently do not support several features e.g.
foreign key references (to and from), functional indexes, sql sequences,
sql triggers, etc. This may change in the future.

Implementing temporary spaces requires temporary tuples to be
inserted into system spaces: tuples which are neither replicated or
persisted. This mostly done in on_replace_dd_* triggers by dropping the
txn->stmt->row.

Closes #8323

@TarantoolBot document
Title: Introduce fully temporary spaces with temporary metadata

Temporary spaces are now data-temporary spaces with temporary metadata.
Created by specifying { type = "temporary" } in the options.
Temporary spaces will not exist upon server restart and will not
exist on replicas. They can also be created in read-only mode.

A tiny preparatory commit for meta-temporary spaces

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Everywhere where we refer to temporary spaces we now say data-temporary.
This is because temporary spaces were never truly temporary because
their definitions would still be persisted and replicated and they
couldn't be created on read-only replicas. In a following commit we will
introduce a new fully temporary type of spaces, which will be just
called 'temporary', so this commit signifies this terminology change.

NO_DOC=renaming
NO_CHANGELOG=renaming
NO_TEST=renaming

Introduces a new field `type` to the space definition. Currently it can
only be "normal" or "data-temporary". It is backwards compatible with
temporary=true.

@TarantoolBot document
Title: Introduce space field type

A new space definition field "type" can now be used to specify the type
of the space. Usage: box.schema.create_space("s", { type = "normal" }).
Currently only 2 types are supported: "normal" & "data-temporary", which
is equivalent to { temporary = true }. Old-style { temporary = true } is
still supported, but only one option either 'temporary' or 'type' may be
specified at the same time.

Space type "temporary" will be introduced in a later commit.
In the future options "local", "synchronous", etc. may also be
supported.

NO_TEST=will be tested in the following commit

Aleksandr Lyapunov authored 1 year ago and Georgy Moshkin committed 1 year ago

The function update_view_references is called when an SQL view
is created or dropped. The goal of this function is to modify
(increment or decrement) view_ref_count member of spaces that
the view references.

There were a several issues that deserves to be refactored:
* By design in case of error it left the job partially done, so
  some space references were modified while some other - not.
  Although there was no bug since special steps were made in case
  of error, this pattern is inconvenient and should be avoided.
* In case of error the failing space name was returned via special
  argument which is not flexible and even requires allocation.
* Another argument - suppress_error - has actually never
  suppressed any error because the only case when an error could
  occur is creation of a view, which used suppress_error = false.
* Fail of that function was not actually covered with tests.

So this commit:
* Makes the function to do all or nothing.
* Forces the function to set diag by itself in case of error.
* Removes suppress_error argument while adding several asserts.\
* Adds a small test that fulfills coverage.

NO_DOC=refactoring
NO_CHANGELOG=reafactoring

Aleksandr Lyapunov authored 1 year ago and Georgy Moshkin committed 1 year ago

By design a newly created SrcList object contains one element
with NULL name. That was confusing and led to strange NULL checks
in a list that could not contain NULL names.

Fix it by clearing the list before usage.

NO_DOC=refactoring
NO_CHANGELOG=reafactoring
NO_TEST=refactoring

Aleksandr Lyapunov authored 1 year ago and Georgy Moshkin committed 1 year ago

Since we panic on OOM now, no OOM error handling is needed now.
Fix both internals of the function and how it is used in alter.

NO_DOC=refactoring
NO_CHANGELOG=reafactoring
NO_TEST=refactoring

Aleksandr Lyapunov authored 1 year ago and Georgy Moshkin committed 1 year ago

There must be a couple of rules:
* foreign key from non-temporary space to temporary space must be
  forbidden since after restart all existing links will be broken.
* foreign key from non-local space to local space must be forbidden
  on any replica all existing can be broken.

This patch implements the rules.

Closes #8936

NO_DOC=bugfix

Vladimir Davydov authored 1 year ago and Georgy Moshkin committed 1 year ago

To achieve that, we bypass the read-only check for the _truncate system
space in box_process1() and perform it in the on_replace system trigger
instead, when we know which space is truncated.

Note, we have to move the check for insertion of a new record into the
_truncate system space before the read-only check in the on_replace
trigger callback; this is needed for initial recovery with a non-empty
_truncate space to work. While we are at it, let's use recovery_state to
make the check explicit.

Closes #5616

@TarantoolBot document
Title: Mention that temp and local spaces can be truncated in ro mode

DML operations on temporary and local spaces can be performed even if
the instance is in the read-only mode, but DDL operations (such as
`alter`) are forbidden in this case[^1]. Technically, `truncate` is
a DDL operation so initially it was forbidden as well. However, it
should be safe to perform this operation on a temporary or local space
because logically it only modifies the data stored in the space (like
DML) and it isn't replicated (see tarantool/tarantool#4263). So starting
from Tarantool 2.11.1 we allow users to truncate temporary spaces in the
read-only mode.

[^1]: https://www.tarantool.io/en/doc/latest/concepts/replication/repl_architecture/#replication-local

User name is usually used as a salt for user password in order to
avoid password repeating.
For instance, postgres md5 authentication stores passwords as
md5("password", "user"), so that the same passwords are represented by
different hashes.

part of picodata/picodata/sbroad!377

@TarantoolBot document
Title: Document updated `box.schema.user.password` declaration.

Since auth methods can use user name for hashing, user name is
added to argument list of `box.schema.user.password`.

NO_TEST=there are no methods that use user name

During creation of a function definition, an error condition for spaces is
diagnosed for wrong options: change it to its own error code.

While we are here, cover missing function definition error conditions.

Closes #7972

NO_DOC=bugfix

Currently, core constraints are dropped on commit. That is why
it is impossible to drop constraint and drop objects it references to
at the same transaction. Let's drop constraints in two steps - detach
them when DDL occurs, then reattach on rollback or delete on commit.

Closes #7339

NO_DOC=bugfix

This patch removes code that was used to implement the SQL check
constraint as they are now replaced by BOX constraint. Also, the syntax
for enabling/disabling check constraints has been removed as BOX
constraints do not support this feature.

Follow-up #6986

NO_DOC=Already introduced.
NO_CHANGELOG=Already introduced.

This patch removes code that was used to implement the SQL foreign key
as they are now replaced by BOX foreign keys.

Follow-up #6986

NO_DOC=Refactoring.
NO_TEST=Refactoring
NO_CHANGELOG=Refactoring.

See the doc bot request for the description of the new fields.

Note that we only store the value of the 'last_modified' field
in struct user_def, because 'auth_history' will be used only in
Lua code.

Needed for https://github.com/tarantool/tarantool-ee/issues/298
Needed for https://github.com/tarantool/tarantool-ee/issues/299

NO_CHANGELOG=no user-visible effects in CE; will be added to EE

@TarantoolBot document
Title: Document auth_history and last_modified _user space fields

Field name: auth_history. Field no: 6. Type: array.
Description: The field stores an array of previous authentication data:
when a user password is changed, the last value of the 'auth' field is
appended to 'auth_history'.  The length of the history is configured by
the `box.cfg.password_history_length` option, which is available only
in Tarantool EE, where it's used to prevent users from reusing old
passwords. In Tarantool CE, the array is always empty.

Field name: last_modified. Field no: 7. Type: unsigned.
Description: The field stores the timestamp (seconds since Unix epoch)
of the last user password update. It's never used in Tarantool CE.
In Tarantool EE, it's used to disable users that haven't changed the
password for more than `box.cfg.password_lifetime_days`.

`box.schema.upgrade()` sets the new field values to an empty array
and 0 for users that haven't updated them yet.

This patch refactors the SQL memory allocation system. There are three
main changes:
1) now, when allocating memory, no additional 8 bytes are allocated to
remember the size of the allocated memory, so instead of
sql_malloc()/sqlRealloc()/sql_free(), the malloc()/realloc()/free()
functions are used;
2) the malloc()/realloc() functions were used through the
xmalloc()/xrealloc() macros, so checks for memory allocation errors were
removed;
3) there is no need for an explicit "sql *db" argument for most of the
functions, so it has been omitted.

Part of #1544

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

This commit introduces an abstraction for the authentication code so
that one can easily add new methods. To add a new method, one just needs
to define a set of authentication callbacks in a struct auth_method and
register it with auth_method_register.

The IPROTO_AUTH and _user.auth formats were initially designed with
extensibility in mind: both take the authentication method name
(currently, only 'chap-sha1' is supported) so no changes to the schema
are required.

Note that although 'chap-sha1' is now implemented in its own file
src/box/auth_chap_sha1.c, we don't merge src/scramble.c into it.
This will be done later, in the scope of #7987.

Since we call authentication plug-ins "methods" (not "mechanisms"),
let's rename BOX_USER_FIELD_AUTH_MECH_LIST to BOX_USER_FIELD_AUTH while
we are at it. Anyway, the corresponding field of the _user system space
is called 'auth' (not 'auth_mech_list').

Closes #7986

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

We will need to store some extra data in the user_def struct to support
different authentication mechanisms. Let's introduce convenient helpers
for allocating and freeing this struct so that we don't have to patch
all the places in the code where it's allocated or freed when we extend
the struct.

While we are at it, switch to grp_alloc, shorted the license text, and
replace include guards with pragma.

Needed for #7986

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

This patch removes three rules in the parser.

NO_CHANGELOG=will be added later

@TarantoolBot document
Title: Changes in parsing rules.

Three rules related to foreign keys have been removed.

The first is the "reference trigger action" rule, which determines the
behavior when a referenced tuple is deleted. This rule has been dropped
because the new foreign keys only support RESTRICT.

The second is "constraint check time" rule, which determines when a
foreign key constraint should be validated. This rule has been dropped
because the new foreign keys only support INITIALLY IMMEDIATELY.

The third is "match type" rule, which determines how inserted values are
validated. This rule has been dropped because the new foreign keys only
support FULL.

Passing the error code to opts_decode complicates adding the space field
or index part number to the error message. Let's use the IllegalParams
error in opts_decode instead and set the proper ClientError at each call
site.

Needed for #7933

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

As it breaks sane usage of region as a data stack:

	size_t region_svp = region_used(&fiber()->gc);
	/* some allocation on fiber gc and usage of allocated memory. */
	region_truncate(&fiber()->gc, region_svp);

If in the above snippet one calls a function that in turn calls
`fiber_gc` then the snippet code may have use-after-free and later UB
on truncation.

For this reason let's get read of fiber_gc. However we need to make sure
we won't introduce leaks this way. So before actually removing
fiber_gc we make it perform leak check instead and only after fixing
all the leaks the fiber_gc was removed.

In order to find the leak easily the backtrace of the first fiber gc
allocation that is not truncated is saved and then reported.

In order to catch leaks that are not triggered by the current test suit
and to prevent introducing leaks in future patches the leak check is
added on fiber exit/recycle and for long living system fibers on every loop
iteration.

Leak check in release build is on but without leak backtrace info by
default for performance reasons. Backtrace can be provided by using
`fiber.leak_backtrace_enable()` knob before starting leaking fiber.

Regularly leaks are only reported in log but it will not help to
catch errors when running test suits so build option ABORT_ON_LEAK
is added. When it is on we abort on leak. This option is turned off
for all builds that used in CI.

Closes #5665

NO_CHANGELOG=internal
NO_DOC=internal

This patch renames index_def_is_valid() to index_def_check() and makes
it return 0 or -1 because that function sets a diag.

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

If a complex (tuple) foreign key is added along with a new field name
that participates in that foreign key, the foreign key does not work
correctly. This happens because the new name of a local field is added
to new_space->def->dict only in ModifySpace::alter, while before that,
new_space->def->dict points to the old dictionary with old names (see
ModifySpace::alter_def). So when local_field_no is initialized earlier
in alter_space_do -> space_create -> tuple_constraint_fkey_init, it's
unable to find the new field number in the old dictionary.

Fix this by moving `new_def->dict = alter->old_space->def->dict;` from
ModifySpace::alter_def() to ModifySpace::alter(). Note that, as before,
we refer to the old dictionary from the new space (just put new names
into it), because that dict is referenced by existing tuple formats.

Closes #7652

NO_DOC=bugfix

To match space_is_temporary.

In-scope-of #6707

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

We need to duplicate a function for handling space upgrade in read view.
We can't just use func_new(func->def) to do this, because func_new sets
the given func_def to func->def, without copying. Usually, foo_new
duplicates the provided foo_def, e.g. see space_new. Let's make func_new
do the same.

Needed for https://github.com/tarantool/tarantool-ee/issues/163

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

func_def_new takes function id, name, body, comment, and owner id and
allocates a new func_def struct, setting the rest of the members to
their default values. We need this function to create a new func_def
object for handling space upgrade in read view.

Note, this isn't a pure refactoring - before this patch, we used
FUNC_LANGUAGE_LUA for SQL builtin functions, which were deprecated in
2.9. This worked fine, because we never actually called them - it was
needed solely for upgrade from older versions. In this commit, we create
an SQL builtin function just like any other function, but set its vtab
to a dummy, which raises an error on an attempt to call it. This should
make the code clearer.

Needed for https://github.com/tarantool/tarantool-ee/issues/163

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Non privileged user (thru public role) has write access to _truncate
table in order to be able to perform truncates on it's tables. Normally
it should be able to modify records only for the tables he has write
access. Yet now due to bootstrap check it is not so.

Closes tarantool/security#5

NO_DOC=bugfix

struct trigger is about to get a new field, and it's mandatory that this
field is specified in all initializers. Let's introduce a macro to avoid
adding every new field to all the initializers and at the same time keep
the benefits of static initialization.

Also while we're at it fix `lbox_trigger_reset` setting all trigger
fileds manually.

Part-of #4264

NO_DOC=refactoring
NO_CHANGELOG=refactoring
NO_TEST=refactoring

Add a missed check to on_replace_dd_truncate, similar to
on_replace_dd_space and on_replace_dd_index.

Closes #7309

NO_DOC=bugfix

We don't rebuild the tuples stored in the space when a space format is
updated, just check that they are compatible with the new format. For
the old tuple fields to be accessible by the new format field names, we
point the old format dictionary to the new format dictionary (a field
dictionary is a reference counted object so the same dictionary can be
used by multiple formats).

This isn't necessary for online space upgrade, because in this case
we upgrade each tuple stored in the space to the new format before
returning it to the user. Moreover, using the new dictionary for the old
tuples would even be harmful in this case, because the old tuples may be
incompatible with the new format, while the space upgrae function may
use the old field names.

Needed for https://github.com/tarantool/tarantool-ee/issues/112

NO_DOC=ee
NO_TEST=ee
NO_CHANGELOG=ee

space_upgrade_check_alter() must be called after the new space
definition is updated.

Follow-up commit 38b25832 ("box: add space upgrade stubs").

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

So that the caller can figure out the end of fields array without
calling mp_next(). Needed for decoding space upgrade format.

While we are at it, let's also
 - Replace MP_ARRAY assertion with diag_set. This makes the function
   easier to use when the MsgPack format isn't guaranteed to be
   MP_ARRAY.
 - Rearrange arguments so that [out] arguments are grouped together.

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

To implement online space upgrade, we need to add stub calls to the
following code paths and data structures:

 - struct space: add a pointer to space upgrade state.
 - struct space_opts: add a pointer to space upgrade definition.
 - CheckSpaceFormat::prepare: skip space check if the format is changed
   in the scope of space upgrade.
 - alter_space_do: check space upgrade state and fail alter if upgrade
   is in progress.
 - alter_space_commit: run background worker for space upgrade.
 - space_on_final_recovery_complete: restart upgrade after recovery.
 - result_processor: apply space upgrade transformations to tuples
   returned to the user by box functions.

We also need to:
 - Add a new error code ER_WRONG_SPACE_UPGRADE_OPTIONS, which we will
   use on error decoding upgrade options, stored in space options.
 - Load space upgrade Lua modules. The modules are supposed to define
   box.internal.space.upgrade method, which if available is used by
   box.schema.space.upgrade.
 - Add check_param, check_param_table and normalize_format helpers to
   box.internal, because we will use them from space.upgrade Lua code.

Note, the space upgrade state will be reference counted, because
background space upgrade may complete while some fiber is reading
from the upgraded space (there may be yields in Vinyl). For this fiber
to process the result correctly, it has to increment the reference
counter of the space upgrade state before reading from the space.

NO_DOC=ee
NO_TEST=ee
NO_CHANGELOG=ee

Half of allocation functions in space_def_dup and space_def_new already
use xmalloc. Use xmalloc for the rest of allocations.

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

So that we can reuse it for online space upgrade.

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring