Denis Smirnov authored 1 year ago and Dmitry Ivanov committed 3 months ago

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

fix(build): Prevent sasl's makefile rules from staining vendor

Turns out Cyrus SASL doesn't use AM_MAINTAINER_MODE or anything to
prevent autotools from re-configuring the project at build time.
This patch "fixes" that by maiming autotools' paths in Makefile.

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

fix(build): add missing headers to libsasl

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

This patch fixes the following problem at build time:

```
cyrus-sasl-2.1.28/lib/saslutil.c:280:3: error: implicit declaration of function 'time' [-Wimplicit-function-declaration]
280 |   time(&now);
    |   ^~~~
cyrus-sasl-2.1.28/lib/saslutil.c:66:1: note: 'time' is defined in header '<time.h>'; this is probably fixable by adding '#include <time.h>'
 65 | #include "saslint.h"
+++ |+#include <time.h>
 66 | #include <saslutil.h>
cyrus-sasl-2.1.28/lib/saslutil.c: In function 'getranddata':
cyrus-sasl-2.1.28/lib/saslutil.c:364:41: error: implicit declaration of function 'clock' [-Wimplicit-function-declaration]
364 |             ret[1] ^= (unsigned short) (clock() & 0xFFFF);
    |                                         ^~~~~
cyrus-sasl-2.1.28/lib/saslutil.c:364:41: note: 'clock' is defined in header '<time.h>'; this is probably fixable by adding '#include <time.h>'
cyrus-sasl-2.1.28/lib/saslutil.c: In function 'get_fqhostname':
cyrus-sasl-2.1.28/lib/saslutil.c:563:42: warning: comparison of integer expressions of different signedness: 'size_t' {aka 'long unsigned int'} and 'int' [-Wsign-compare]
563 |         || strlen (result->ai_canonname) > namelen -1) {
    |                                          ^
make[9]: *** [Makefile:643: saslutil.lo] Error 1
```

Georgy Moshkin authored 2 years ago and Dmitry Ivanov committed 3 months ago

- tarantool is now a static library
- main(argc, argv) -> tarantool_main(argc, argv, cb, cb_data)
- callback is called before running lua script

Georgy Moshkin authored 1 year ago and Dmitry Ivanov committed 3 months ago

Closes #9237

Add exports for fiber_set_name_n, fiber_name, fiber_id, fiber_csw &
fiber_find.

Also make fiber_set_joinable, fiber_set_ctx & fiber_get_ctx interpret
NULL as the current fiber.

@TarantoolBot document
Title: add basic fiber api to ffi exports.

5 basic functions can now be used via ffi api, which were previously
only accessible via lua api: fiber_set_name_n, fiber_name, fiber_id,
fiber_csw & fiber_find.

fiber_set_joinable now interprets NULL as current fiber.

Georgy Moshkin authored 1 year ago and Dmitry Ivanov committed 3 months ago

NO_DOC=internal change
NO_TEST=internal change

Dmitry Rodionov authored 1 year ago and Dmitry Ivanov committed 3 months ago

Rename the members as well. Keep uint16_t in box_check_acess_space
as it is for other exported functions

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Dmitry Rodionov authored 1 year ago and Dmitry Ivanov committed 3 months ago

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Co-authored-by: Yaroslav Dynnikov <yaroslav.dynnikov@gmail.com>

Georgy Moshkin authored 1 year ago and Dmitry Ivanov committed 3 months ago

Introduce API for opening a read view over the specified spaces and
indexes and creating iterators over the index read views.

NO_DOC=picodata patch

test-run.py sets OMP_NUM_THREADS which causes this test to misbehave:
it doesn't expect to see a warning regarding the use of this env var
as the first log line.

Alternatively, we could update test-run's submodule, but in practice
this breaks other tests e.g. replication-luatest/linearizable_test.lua.

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

Ilya Verbin authored 1 year ago and Dmitry Ivanov committed 3 months ago

OpenMP is no longer used since commit 4f617b70 ("box: introduce
memtx_sort_threads config parameter"). All dependencies on libgomp
should be removed.

Follow-up #7689

NO_DOC=build
NO_TEST=build

Nikolay Shirokovskiy authored 1 year ago and Dmitry Ivanov committed 3 months ago

Closes #3389
Closes #7689
Closes #4646

@TarantoolBot document
Title: new box.cfg parameter memtx_sort_threads

The parameter sets the number of threads used to sort keys of secondary
indexes on loading memtx database. The parameter cannot be changed
dynamically (as it does not make sense).

Maximum value is 256, minimum is 1. Default is to use all available cores.

Usage example:
```
box.cfg{memtx_sort_threads=4}
```

Nikolay Shirokovskiy authored 1 year ago and Dmitry Ivanov committed 3 months ago

The algorithm runs sort in multiple threads and does not use OpenMP. It
has better threads utilization right from the beginning but probably
a worse constant than parallel qsort. See details in code comments.

Besides sort is not performed in calling thread but instead in spawned
worker threads. Calling thread yields waiting for worker threads to
finish. Exception is small data size, in this case sorting is executed
in calling thread saving time on spawning a thread. This should speed up
test execution. This is existing behaviour of qsort_arg but data size
threshold is reduced from 128000 to 1024.

Part of #3389

NO_CHANGELOG=internal
NO_DOC=internal

Georgy Moshkin authored 1 year ago and Dmitry Ivanov committed 3 months ago

Introduce fully temporary spaces: same as data-temporary space but with
temporary metadata. Basically temporary spaces now do not exist on
restart and do not exist on replicas. They can also be created, altered
and deleted when box.cfg.read_only = true.

To avoid conflicts with spaces created on replicas, the temporary
space ids by default start in a special range starting at
BOX_SPACE_ID_TEMPORARY_MIN.

Temporary spaces currently do not support several features e.g.
foreign key references (to and from), functional indexes, sql sequences,
sql triggers, etc. This may change in the future.

Implementing temporary spaces requires temporary tuples to be
inserted into system spaces: tuples which are neither replicated or
persisted. This mostly done in on_replace_dd_* triggers by dropping the
txn->stmt->row.

Closes #8323

@TarantoolBot document
Title: Introduce fully temporary spaces with temporary metadata

Temporary spaces are now data-temporary spaces with temporary metadata.
Created by specifying { type = "temporary" } in the options.
Temporary spaces will not exist upon server restart and will not
exist on replicas. They can also be created in read-only mode.

Georgy Moshkin authored 1 year ago and Dmitry Ivanov committed 3 months ago

A tiny preparatory commit for meta-temporary spaces

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Georgy Moshkin authored 2 years ago and Dmitry Ivanov committed 3 months ago

Move code that handles txn row counters into a separate function in
preparation of meta-temporary spaces introduction.

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Georgy Moshkin authored 1 year ago and Dmitry Ivanov committed 3 months ago

Everywhere where we refer to temporary spaces we now say data-temporary.
This is because temporary spaces were never truly temporary because
their definitions would still be persisted and replicated and they
couldn't be created on read-only replicas. In a following commit we will
introduce a new fully temporary type of spaces, which will be just
called 'temporary', so this commit signifies this terminology change.

NO_DOC=renaming
NO_CHANGELOG=renaming
NO_TEST=renaming

Georgy Moshkin authored 2 years ago and Dmitry Ivanov committed 3 months ago

Introduces a new field `type` to the space definition. Currently it can
only be "normal" or "data-temporary". It is backwards compatible with
temporary=true.

@TarantoolBot document
Title: Introduce space field type

A new space definition field "type" can now be used to specify the type
of the space. Usage: box.schema.create_space("s", { type = "normal" }).
Currently only 2 types are supported: "normal" & "data-temporary", which
is equivalent to { temporary = true }. Old-style { temporary = true } is
still supported, but only one option either 'temporary' or 'type' may be
specified at the same time.

Space type "temporary" will be introduced in a later commit.
In the future options "local", "synchronous", etc. may also be
supported.

NO_TEST=will be tested in the following commit

Aleksandr Lyapunov authored 1 year ago and Dmitry Ivanov committed 3 months ago

The function update_view_references is called when an SQL view
is created or dropped. The goal of this function is to modify
(increment or decrement) view_ref_count member of spaces that
the view references.

There were a several issues that deserves to be refactored:
* By design in case of error it left the job partially done, so
  some space references were modified while some other - not.
  Although there was no bug since special steps were made in case
  of error, this pattern is inconvenient and should be avoided.
* In case of error the failing space name was returned via special
  argument which is not flexible and even requires allocation.
* Another argument - suppress_error - has actually never
  suppressed any error because the only case when an error could
  occur is creation of a view, which used suppress_error = false.
* Fail of that function was not actually covered with tests.

So this commit:
* Makes the function to do all or nothing.
* Forces the function to set diag by itself in case of error.
* Removes suppress_error argument while adding several asserts.\
* Adds a small test that fulfills coverage.

NO_DOC=refactoring
NO_CHANGELOG=reafactoring

Aleksandr Lyapunov authored 1 year ago and Dmitry Ivanov committed 3 months ago

By design a newly created SrcList object contains one element
with NULL name. That was confusing and led to strange NULL checks
in a list that could not contain NULL names.

Fix it by clearing the list before usage.

NO_DOC=refactoring
NO_CHANGELOG=reafactoring
NO_TEST=refactoring

Aleksandr Lyapunov authored 1 year ago and Dmitry Ivanov committed 3 months ago

Since we panic on OOM now, no OOM error handling is needed now.
Fix both internals of the function and how it is used in alter.

NO_DOC=refactoring
NO_CHANGELOG=reafactoring
NO_TEST=refactoring

Ilya Verbin authored 2 years ago and Dmitry Ivanov committed 3 months ago

Now a field can be assigned a default value in the space format. When a new
tuple is inserted into a space, and some of the fields contain null values,
those fields will be filled with their respective default values.

Closes #8157

@TarantoolBot document
Title: Document default field values
Product: Tarantool
Since: 3.0
Root document: https://www.tarantool.io/en/doc/latest/reference/reference_lua/box_space/format/

The format clause contains, for each field, a definition within braces:
`{name='...',type='...'[,is_nullable=...][,default=...]}`, where:

* the optional `default` value contains a default value for the field.
  Its type must be compatible with the field type. If default value is set,
  it is applied regardless of whether `is_nullable` is true or false.

Example:

```lua
tarantool> box.space.tester:format{
         > {name = 'id', type = 'unsigned'},
         > {name = 'name', type = 'string', default = 'Noname'},
         > {name = 'pass', type = 'string'},
         > {name = 'shell', type = 'string', default = '/bin/sh'}}
---
...

tarantool> box.space.tester:insert{1000, nil, 'qwerty'}
---
- [1000, 'Noname', 'qwerty', '/bin/sh']
...
```

Ilya Verbin authored 1 year ago and Dmitry Ivanov committed 3 months ago

It encapsulates the logic that helps to build a new MsgPack array by
concatenating tuple fields from various locations. The idea is to
postpone memory allocation and copying until the finalization.

Needed for #8157

NO_DOC=internal
NO_CHANGELOG=internal

Ilya Verbin authored 2 years ago and Dmitry Ivanov committed 3 months ago

In order to avoid collision with the upcoming core default_value.

Also rename tuple_field::default_value_expr to
tuple_field::sql_default_value_expr.

Part of #8157

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Maksim Kaitmazian authored 1 year ago and Dmitry Ivanov committed 3 months ago

PG protocol run queries by calling dispatch_query, which stores the
response in port_c contained in box_function_ctx structure.

NO_DOC=exports
NO_CHANGELOG=exports
NO_TEST=exports

NO_DOC=internal
NO_CHANGELOG=internal
NO_TEST=internal

Denis Smirnov authored 1 year ago and Dmitry Ivanov committed 3 months ago

box_auth_data_prepare() method declared to return a tuple while in
reality it returned a region allocated message pack string. Fixed.

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Denis Smirnov authored 1 year ago and Dmitry Ivanov committed 3 months ago

1. Current commit introduces 'box_auth_data_prepare()' to prepare
   a data string for any supported authentication methods.
2. The user name argument is refactored in the auth methods: the
   null-terminated string is replaced with an address range approach.
   Now Rust users don't need to re-allocate username with CString.
3. Password length type was set to uint32_t (previously it was size_t,
   int, uint32_t for different functions). Tarantool uses murmur3a,
   so all the hashed strings should be up to 32 bit long.

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Denis Smirnov authored 1 year ago and Dmitry Ivanov committed 3 months ago

Introduce new methods:

1. box_user_id_by_name - get the user identifier by name;
2. box_effective_user_id - get current effective user
   identifier;
3. box_session_user_id - get current session user identifier;
4. box_session_su - change current session user;

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Maksim Kaitmazian authored 1 year ago and Dmitry Ivanov committed 3 months ago

This function is used to determine the authentication method when
authenticating a postgres client.

part of picodata/picodata/sbroad!292

NO_DOC=exports
NO_CHANGELOG=exports
NO_TEST=exports

Maksim Kaitmazian authored 1 year ago and Dmitry Ivanov committed 3 months ago

These exports allow us to perform user authentication and
network communication.

part of picodata/picodata/sbroad!292

NO_DOC=exports
NO_CHANGELOG=exports
NO_TEST=exports

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Now it's possible to specify the desired authentication method during
user creation via `auth_type`, e.g.

```lua
box.schema.user.create('mickey', { auth_type = 'chap-sha1',
                                   password = 'foobar' })
```

Furthermore, authentication methods may now specify that they don't
require password to create stored authentication info. This is used
in LDAP authentication (`auth_type = 'ldap'`):

```lua
box.schema.user.create('mickey', { auth_type = 'ldap' })
```

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Unfortunately, Centos 7 provides only openssl 1.0.2 (at lest if we
disregard epel), so we can't build the bundled libldap & libsasl2.
"Okay", one might think, "we can link against the distro's libs".
Well, turns out libldap 2.4, which is what we have to deal with in
that case, doesn't have ldap_connect!

Luckily, we don't have to connect explicitly. According to man pages:

```
ldap_init() acts just like ldap_open(), but does not open a connection
to the LDAP server.  The actual connection open will occur when the
first operation is attempted.

ldap_initialize()  acts  like ldap_init()...
```

This is still true for libldap up to and including version 2.6.

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

This authentication method doesn't store any secrets; instead,
we delegate the whole auth to a pre-configured LDAP server. In
the method's implementation, we connect to the LDAP server and
perform a BIND operation which checks user's credentials.

Usage example:

```lua
-- Set the default auth method to LDAP and create a new user.
-- NOTE that we still have to provide a dummy password; otherwise
-- box.schema.user.create will setup an empty auth data.
box.cfg({auth_type = 'ldap'})
box.schema.user.create('demo', { password = '' })

-- Configure LDAP server connection URL and DN format string.
os = require('os')
os.setenv('TT_LDAP_URL', 'ldap://localhost:1389')
os.setenv('TT_LDAP_DN_FMT', 'cn=$USER,ou=users,dc=example,dc=org')

-- Authenticate using the LDAP authentication method via net.box.
conn = require('net.box').connect(uri, {
    user = 'demo',
    password = 'password',
    auth_type = 'ldap',
})
```

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

This is required for LDAP authentication, because we need
username to format the corresponding DN.

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Maksim Kaitmazian authored 1 year ago and Dmitry Ivanov committed 3 months ago

box.schema.user.passwd doesn't change the password for the current
user because new password is passed instead of the user name.

NO_CHANGELOG=fix an unreleased bug
NO_DOC=fix an unreleased bug

Maksim Kaitmazian authored 1 year ago and Dmitry Ivanov committed 3 months ago

It fixes the following assertion
```bash
tarantool: ./src/lib/core/crypt.c:84: md5_encrypt:
Assertion `password_len + salt_len > 0' failed.
```
caused by the following code
```lua
box.cfg{auth_type='md5'}
box.schema.user.password("")
```

NO_CHANGELOG=fix an unreleased feature
NO_DOC=fix an unreleased feature