Introduce API for opening a read view over the specified spaces and
indexes and creating iterators over the index read views.

NO_DOC=picodata patch

box_auth_data_prepare() method declared to return a tuple while in
reality it returned a region allocated message pack string. Fixed.

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

test-run.py sets OMP_NUM_THREADS which causes this test to misbehave:
it doesn't expect to see a warning regarding the use of this env var
as the first log line.

Alternatively, we could update test-run's submodule, but in practice
this breaks other tests e.g. replication-luatest/linearizable_test.lua.

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

Ilya Verbin authored 1 year ago and Dmitry Ivanov committed 1 year ago

OpenMP is no longer used since commit 4f617b70 ("box: introduce
memtx_sort_threads config parameter"). All dependencies on libgomp
should be removed.

Follow-up #7689

NO_DOC=build
NO_TEST=build

Nikolay Shirokovskiy authored 1 year ago and Dmitry Ivanov committed 1 year ago

Closes #3389
Closes #7689
Closes #4646

@TarantoolBot document
Title: new box.cfg parameter memtx_sort_threads

The parameter sets the number of threads used to sort keys of secondary
indexes on loading memtx database. The parameter cannot be changed
dynamically (as it does not make sense).

Maximum value is 256, minimum is 1. Default is to use all available cores.

Usage example:
```
box.cfg{memtx_sort_threads=4}
```

Nikolay Shirokovskiy authored 1 year ago and Dmitry Ivanov committed 1 year ago

The algorithm runs sort in multiple threads and does not use OpenMP. It
has better threads utilization right from the beginning but probably
a worse constant than parallel qsort. See details in code comments.

Besides sort is not performed in calling thread but instead in spawned
worker threads. Calling thread yields waiting for worker threads to
finish. Exception is small data size, in this case sorting is executed
in calling thread saving time on spawning a thread. This should speed up
test execution. This is existing behaviour of qsort_arg but data size
threshold is reduced from 128000 to 1024.

Part of #3389

NO_CHANGELOG=internal
NO_DOC=internal

1. Current commit introduces 'box_auth_data_prepare()' to prepare
   a data string for any supported authentication methods.
2. The user name argument is refactored in the auth methods: the
   null-terminated string is replaced with an address range approach.
   Now Rust users don't need to re-allocate username with CString.
3. Password length type was set to uint32_t (previously it was size_t,
   int, uint32_t for different functions). Tarantool uses murmur3a,
   so all the hashed strings should be up to 32 bit long.

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Дмитрий Кибирев authored 1 year ago and Yaroslav Dynnikov committed 1 year ago

Add the temporary path for macos CI.

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Introduce new methods:

1. box_user_id_by_name - get the user identifier by name;
2. box_effective_user_id - get current effective user
   identifier;
3. box_session_user_id - get current session user identifier;
4. box_session_su - change current session user;

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Introduce fully temporary spaces: same as data-temporary space but with
temporary metadata. Basically temporary spaces now do not exist on
restart and do not exist on replicas. They can also be created, altered
and deleted when box.cfg.read_only = true.

To avoid conflicts with spaces created on replicas, the temporary
space ids by default start in a special range starting at
BOX_SPACE_ID_TEMPORARY_MIN.

Temporary spaces currently do not support several features e.g.
foreign key references (to and from), functional indexes, sql sequences,
sql triggers, etc. This may change in the future.

Implementing temporary spaces requires temporary tuples to be
inserted into system spaces: tuples which are neither replicated or
persisted. This mostly done in on_replace_dd_* triggers by dropping the
txn->stmt->row.

Closes #8323

@TarantoolBot document
Title: Introduce fully temporary spaces with temporary metadata

Temporary spaces are now data-temporary spaces with temporary metadata.
Created by specifying { type = "temporary" } in the options.
Temporary spaces will not exist upon server restart and will not
exist on replicas. They can also be created in read-only mode.

A tiny preparatory commit for meta-temporary spaces

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Move code that handles txn row counters into a separate function in
preparation of meta-temporary spaces introduction.

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Everywhere where we refer to temporary spaces we now say data-temporary.
This is because temporary spaces were never truly temporary because
their definitions would still be persisted and replicated and they
couldn't be created on read-only replicas. In a following commit we will
introduce a new fully temporary type of spaces, which will be just
called 'temporary', so this commit signifies this terminology change.

NO_DOC=renaming
NO_CHANGELOG=renaming
NO_TEST=renaming

Introduces a new field `type` to the space definition. Currently it can
only be "normal" or "data-temporary". It is backwards compatible with
temporary=true.

@TarantoolBot document
Title: Introduce space field type

A new space definition field "type" can now be used to specify the type
of the space. Usage: box.schema.create_space("s", { type = "normal" }).
Currently only 2 types are supported: "normal" & "data-temporary", which
is equivalent to { temporary = true }. Old-style { temporary = true } is
still supported, but only one option either 'temporary' or 'type' may be
specified at the same time.

Space type "temporary" will be introduced in a later commit.
In the future options "local", "synchronous", etc. may also be
supported.

NO_TEST=will be tested in the following commit

Aleksandr Lyapunov authored 1 year ago and Georgy Moshkin committed 1 year ago

The function update_view_references is called when an SQL view
is created or dropped. The goal of this function is to modify
(increment or decrement) view_ref_count member of spaces that
the view references.

There were a several issues that deserves to be refactored:
* By design in case of error it left the job partially done, so
  some space references were modified while some other - not.
  Although there was no bug since special steps were made in case
  of error, this pattern is inconvenient and should be avoided.
* In case of error the failing space name was returned via special
  argument which is not flexible and even requires allocation.
* Another argument - suppress_error - has actually never
  suppressed any error because the only case when an error could
  occur is creation of a view, which used suppress_error = false.
* Fail of that function was not actually covered with tests.

So this commit:
* Makes the function to do all or nothing.
* Forces the function to set diag by itself in case of error.
* Removes suppress_error argument while adding several asserts.\
* Adds a small test that fulfills coverage.

NO_DOC=refactoring
NO_CHANGELOG=reafactoring

Aleksandr Lyapunov authored 1 year ago and Georgy Moshkin committed 1 year ago

By design a newly created SrcList object contains one element
with NULL name. That was confusing and led to strange NULL checks
in a list that could not contain NULL names.

Fix it by clearing the list before usage.

NO_DOC=refactoring
NO_CHANGELOG=reafactoring
NO_TEST=refactoring

Aleksandr Lyapunov authored 1 year ago and Georgy Moshkin committed 1 year ago

Since we panic on OOM now, no OOM error handling is needed now.
Fix both internals of the function and how it is used in alter.

NO_DOC=refactoring
NO_CHANGELOG=reafactoring
NO_TEST=refactoring

Aleksandr Lyapunov authored 1 year ago and Georgy Moshkin committed 1 year ago

There must be a couple of rules:
* foreign key from non-temporary space to temporary space must be
  forbidden since after restart all existing links will be broken.
* foreign key from non-local space to local space must be forbidden
  on any replica all existing can be broken.

This patch implements the rules.

Closes #8936

NO_DOC=bugfix

Vladimir Davydov authored 1 year ago and Georgy Moshkin committed 1 year ago

To achieve that, we bypass the read-only check for the _truncate system
space in box_process1() and perform it in the on_replace system trigger
instead, when we know which space is truncated.

Note, we have to move the check for insertion of a new record into the
_truncate system space before the read-only check in the on_replace
trigger callback; this is needed for initial recovery with a non-empty
_truncate space to work. While we are at it, let's use recovery_state to
make the check explicit.

Closes #5616

@TarantoolBot document
Title: Mention that temp and local spaces can be truncated in ro mode

DML operations on temporary and local spaces can be performed even if
the instance is in the read-only mode, but DDL operations (such as
`alter`) are forbidden in this case[^1]. Technically, `truncate` is
a DDL operation so initially it was forbidden as well. However, it
should be safe to perform this operation on a temporary or local space
because logically it only modifies the data stored in the space (like
DML) and it isn't replicated (see tarantool/tarantool#4263). So starting
from Tarantool 2.11.1 we allow users to truncate temporary spaces in the
read-only mode.

[^1]: https://www.tarantool.io/en/doc/latest/concepts/replication/repl_architecture/#replication-local

Andrey Saranchin authored 2 years ago and Georgy Moshkin committed 1 year ago

Currently, _schema.max_id is used to generate sequentially growing
space ids. The main drawback of this approach is that generated space id
can be not unique if one mixes implicit and explicit space ids. Let's use
actual maximal space id to generate a new one, and scan for free id if
overflow happened.

Closes #8036

NO_DOC=bugfix

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

Ilya Verbin authored 2 years ago and Дмитрий Кольцов committed 1 year ago

Now a field can be assigned a default value in the space format. When a new
tuple is inserted into a space, and some of the fields contain null values,
those fields will be filled with their respective default values.

Closes #8157

@TarantoolBot document
Title: Document default field values
Product: Tarantool
Since: 3.0
Root document: https://www.tarantool.io/en/doc/latest/reference/reference_lua/box_space/format/

The format clause contains, for each field, a definition within braces:
`{name='...',type='...'[,is_nullable=...][,default=...]}`, where:

* the optional `default` value contains a default value for the field.
  Its type must be compatible with the field type. If default value is set,
  it is applied regardless of whether `is_nullable` is true or false.

Example:

```lua
tarantool> box.space.tester:format{
         > {name = 'id', type = 'unsigned'},
         > {name = 'name', type = 'string', default = 'Noname'},
         > {name = 'pass', type = 'string'},
         > {name = 'shell', type = 'string', default = '/bin/sh'}}
---
...

tarantool> box.space.tester:insert{1000, nil, 'qwerty'}
---
- [1000, 'Noname', 'qwerty', '/bin/sh']
...
```

Ilya Verbin authored 1 year ago and Дмитрий Кольцов committed 1 year ago

It encapsulates the logic that helps to build a new MsgPack array by
concatenating tuple fields from various locations. The idea is to
postpone memory allocation and copying until the finalization.

Needed for #8157

NO_DOC=internal
NO_CHANGELOG=internal

Ilya Verbin authored 2 years ago and Дмитрий Кольцов committed 1 year ago

In order to avoid collision with the upcoming core default_value.

Also rename tuple_field::default_value_expr to
tuple_field::sql_default_value_expr.

Part of #8157

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Mergen Imeev authored 1 year ago and Дмитрий Кольцов committed 1 year ago

This patch removes the 'size' argument from macros, as it was only used
to set an error on failure, which is not possible for x* versions. In
addition, both macros now cast the value to the specified type, as is
done in the original macros.

Closes #8522

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

Maksim Kaitmazian authored 1 year ago and Dmitry Ivanov committed 1 year ago

PG protocol run queries by calling dispatch_query, which stores the
response in port_c contained in box_function_ctx structure.

NO_DOC=exports
NO_CHANGELOG=exports
NO_TEST=exports

Maksim Kaitmazian authored 1 year ago and Dmitry Ivanov committed 1 year ago

This function is used to determine the authentication method when
authenticating a postgres client.

part of picodata/picodata/sbroad!292

NO_DOC=exports
NO_CHANGELOG=exports
NO_TEST=exports

These exports allow us to perform user authentication and
network communication.

part of picodata/picodata/sbroad!292

NO_DOC=exports
NO_CHANGELOG=exports
NO_TEST=exports

Georgy Moshkin authored 1 year ago and Maksim Kaitmazian committed 1 year ago

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Better late than never.

Unfortunately, Centos 7 provides only openssl 1.0.2 (at lest if we
disregard epel), so we can't build the bundled libldap & libsasl2.
"Okay", one might think, "we can link against the distro's libs".
Well, turns out libldap 2.4, which is what we have to deal with in
that case, doesn't have ldap_connect!

Luckily, we don't have to connect explicitly. According to man pages:

```
ldap_init() acts just like ldap_open(), but does not open a connection
to the LDAP server.  The actual connection open will occur when the
first operation is attempted.

ldap_initialize()  acts  like ldap_init()...
```

This is still true for libldap up to and including version 2.6.

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

This is needed to fix Ninja, which requires them to
build a dependency graph.

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Now it's possible to specify the desired authentication method during
user creation via `auth_type`, e.g.

```lua
box.schema.user.create('mickey', { auth_type = 'chap-sha1',
                                   password = 'foobar' })
```

Furthermore, authentication methods may now specify that they don't
require password to create stored authentication info. This is used
in LDAP authentication (`auth_type = 'ldap'`):

```lua
box.schema.user.create('mickey', { auth_type = 'ldap' })
```

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

This authentication method doesn't store any secrets; instead,
we delegate the whole auth to a pre-configured LDAP server. In
the method's implementation, we connect to the LDAP server and
perform a BIND operation which checks user's credentials.

Usage example:

```lua
-- Set the default auth method to LDAP and create a new user.
-- NOTE that we still have to provide a dummy password; otherwise
-- box.schema.user.create will setup an empty auth data.
box.cfg({auth_type = 'ldap'})
box.schema.user.create('demo', { password = '' })

-- Configure LDAP server connection URL and DN format string.
os = require('os')
os.setenv('TT_LDAP_URL', 'ldap://localhost:1389')
os.setenv('TT_LDAP_DN_FMT', 'cn=$USER,ou=users,dc=example,dc=org')

-- Authenticate using the LDAP authentication method via net.box.
conn = require('net.box').connect(uri, {
    user = 'demo',
    password = 'password',
    auth_type = 'ldap',
})
```

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

This is required for LDAP authentication, because we need
username to format the corresponding DN.

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

box.schema.user.passwd doesn't change the password for the current
user because new password is passed instead of the user name.

NO_CHANGELOG=fix an unreleased bug
NO_DOC=fix an unreleased bug

It fixes the following assertion
```bash
tarantool: ./src/lib/core/crypt.c:84: md5_encrypt:
Assertion `password_len + salt_len > 0' failed.
```
caused by the following code
```lua
box.cfg{auth_type='md5'}
box.schema.user.password("")
```

NO_CHANGELOG=fix an unreleased feature
NO_DOC=fix an unreleased feature