BREAKING CHANGE!:
1. remove sql_prepare from the export list;
2. introduce sql_prepare_ext.

The sql_prepare symbol previously included the tarantool port as
an output parameter. However, this structure was inconvenient for
libraries using the C API, as they primarily required just the
statement ID. To address this issue, the sql_prepare symbol was
replaced with the sql_prepare_ext symbol.

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch

Previously, users with multiple connections to tarantool instance
couldn't share prepared statements across sessions. They had to
manually call prepare in each session before execution.

This commit automates this process for the exported version of
SQL prepared statement execution (sql_execute_prepared_ext symbol).
Original Lua execution keeps the old behavior for backward
compatibility.

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

BREAKING CHANGE!:
1. sql_bind_list_decode - removed
2. sql_execute_prepared_ext - new arguments
3. sql_prepare_and_execute_ext - exported

There were several reasons to refactor the API.
1. sql_bind_list_decode (decodes message pack parameters into
   internal C bind structure) is very difficult to use without
   memory leaks (as it allocates results on fiber()->gc).
2. sql_execute_prepared_ext missed vdbe step limit in parameters
   and used the default value.
3. Sometimes SQL queries don't fit into prepared statement cache
   and the user still wants to execute them via a slow pass with
   full compilations from the query text. That was the reason to
   export sql_prepare_and_execute_ext symbol.

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

Also remove CI pipeline for fuzz_until, as running it in CI is not planned

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

- add cli flag whether to wait for 2x coverage
- add timestamps to log lines
- remove dictionary passing as it is not needed when
  corpus already exist

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

- add corpus for xrow_decode_error and xrow_header_decode
- update corpus for xrow_decode_call and xrow_decode_dml
  for increased coverage

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

NO_DOC=bug fix
NO_TEST=bug fix
NO_CHANGELOG=bug fix

Add new transport for IPROTO connections: ssl.
SSL transport may be configured with (at client and server side):
- certificate (mandatory at server side)
- private key (mandatory at server side)
- password for PK
- certificate authorities (for peer certificate verification)
- cipher list

SSL transport also can be used in replication and net.box mechanisms.

@TarantoolBot document
Title: add IPROTO traffic encryption.

New ssl transport allows
creating a secure connection between two IPROTO peers.
TLS protocol using and openssl v1.1 or later required.

To configure traffic encryption, you need to set the
special URI parameters for a particular connection.
The parameters can be set for the following box.cfg
options and nex.box method:
- box.cfg.listen – on the server side.
- box.cfg.replication–on the client side.
- net_box_object.connect()–on the client side.

Дмитрий Кибирев authored 1 year ago and Yaroslav Dynnikov committed 1 year ago

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

Egor Ivkov authored 1 year ago and Егор Ивков committed 1 year ago

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

godzie44 authored 1 year ago and Дмитрий Кольцов committed 1 year ago

Currently, DDL transaction may abort other in-progress
transactions (this behavior added in #8f4be322 - avoid sharing
(ergo phantom reads) metadata object for different
transactions in MVCC mode).
This behavior makes no sense if the MVCC is
disabled (cause these conflicts never happen).

Closes #47

NO_CHANGELOG=internal
NO_DOC=internal
NO_TEST=internal

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

(cherry picked from commit 05b62fe0)

Current commit is a part of the SQL procedures track for Picodata.
It introduces several features:
- the reserved id range in the _func space, that can be only consumed
  implicitly;
- an additional "id" option for function creation with Lua (to set
  the function identifier implicitly);
- helper function to generate function ids either in reserved or default
  ranges

NO_DOC=internal
NO_CHANGELOG=internal

Dmitry Ivanov authored 1 year ago and Дмитрий Кольцов committed 1 year ago

This reverts commit c8594fbd.

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

Dmitry Ivanov authored 1 year ago and Дмитрий Кольцов committed 1 year ago

This reverts commit 9bde48a8.

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

Dmitry Ivanov authored 1 year ago and Дмитрий Кольцов committed 1 year ago

This reverts commit 4643a26a.

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

Dmitry Rodionov authored 1 year ago and Дмитрий Кольцов committed 1 year ago

This patch makes schema_object_type to be exported.

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Dmitry Rodionov authored 1 year ago and Дмитрий Кольцов committed 1 year ago

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Georgy Moshkin authored 1 year ago and Дмитрий Кольцов committed 1 year ago

NO_CHANGELOG=bug fix for unreleased feature
NO_DOC=bug fix for unreleased feature

Feodor Alexandrov authored 1 year ago and Дмитрий Кольцов committed 1 year ago

Picodata is built statically, and name crypto clashes between
`src/lib/crypto` and libcrypto from `openssl`. To resolve the
ambiguity this patch renames tarantool library to `tcrypto`.

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Georgy Moshkin authored 1 year ago and Дмитрий Кольцов committed 1 year ago

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Georgy Moshkin authored 1 year ago and Дмитрий Кольцов committed 1 year ago

Closes #9237

Add exports for fiber_set_name_n, fiber_name, fiber_id, fiber_csw &
fiber_find.

Also make fiber_set_joinable, fiber_set_ctx & fiber_get_ctx interpret
NULL as the current fiber.

@TarantoolBot document
Title: add basic fiber api to ffi exports.

5 basic functions can now be used via ffi api, which were previously
only accessible via lua api: fiber_set_name_n, fiber_name, fiber_id,
fiber_csw & fiber_find.

fiber_set_joinable now interprets NULL as current fiber.

Dmitry Rodionov authored 1 year ago and Дмитрий Кольцов committed 1 year ago

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Co-authored-by: Yaroslav Dynnikov <yaroslav.dynnikov@gmail.com>

Georgy Moshkin authored 1 year ago and Дмитрий Кольцов committed 1 year ago

Introduce API for opening a read view over the specified spaces and
indexes and creating iterators over the index read views.

NO_DOC=picodata patch

Denis Smirnov authored 1 year ago and Дмитрий Кольцов committed 1 year ago

box_auth_data_prepare() method declared to return a tuple while in
reality it returned a region allocated message pack string. Fixed.

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Dmitry Ivanov authored 1 year ago and Дмитрий Кольцов committed 1 year ago

test-run.py sets OMP_NUM_THREADS which causes this test to misbehave:
it doesn't expect to see a warning regarding the use of this env var
as the first log line.

Alternatively, we could update test-run's submodule, but in practice
this breaks other tests e.g. replication-luatest/linearizable_test.lua.

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

Nikolay Shirokovskiy authored 1 year ago and Дмитрий Кольцов committed 1 year ago

Closes #3389
Closes #7689
Closes #4646

@TarantoolBot document
Title: new box.cfg parameter memtx_sort_threads

The parameter sets the number of threads used to sort keys of secondary
indexes on loading memtx database. The parameter cannot be changed
dynamically (as it does not make sense).

Maximum value is 256, minimum is 1. Default is to use all available cores.

Usage example:
```
box.cfg{memtx_sort_threads=4}
```

Nikolay Shirokovskiy authored 1 year ago and Дмитрий Кольцов committed 1 year ago

The algorithm runs sort in multiple threads and does not use OpenMP. It
has better threads utilization right from the beginning but probably
a worse constant than parallel qsort. See details in code comments.

Besides sort is not performed in calling thread but instead in spawned
worker threads. Calling thread yields waiting for worker threads to
finish. Exception is small data size, in this case sorting is executed
in calling thread saving time on spawning a thread. This should speed up
test execution. This is existing behaviour of qsort_arg but data size
threshold is reduced from 128000 to 1024.

Part of #3389

NO_CHANGELOG=internal
NO_DOC=internal

Denis Smirnov authored 1 year ago and Дмитрий Кольцов committed 1 year ago

1. Current commit introduces 'box_auth_data_prepare()' to prepare
   a data string for any supported authentication methods.
2. The user name argument is refactored in the auth methods: the
   null-terminated string is replaced with an address range approach.
   Now Rust users don't need to re-allocate username with CString.
3. Password length type was set to uint32_t (previously it was size_t,
   int, uint32_t for different functions). Tarantool uses murmur3a,
   so all the hashed strings should be up to 32 bit long.

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Denis Smirnov authored 1 year ago and Дмитрий Кольцов committed 1 year ago

Introduce new methods:

1. box_user_id_by_name - get the user identifier by name;
2. box_effective_user_id - get current effective user
   identifier;
3. box_session_user_id - get current session user identifier;
4. box_session_su - change current session user;

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Georgy Moshkin authored 1 year ago and Дмитрий Кольцов committed 1 year ago

Introduce fully temporary spaces: same as data-temporary space but with
temporary metadata. Basically temporary spaces now do not exist on
restart and do not exist on replicas. They can also be created, altered
and deleted when box.cfg.read_only = true.

To avoid conflicts with spaces created on replicas, the temporary
space ids by default start in a special range starting at
BOX_SPACE_ID_TEMPORARY_MIN.

Temporary spaces currently do not support several features e.g.
foreign key references (to and from), functional indexes, sql sequences,
sql triggers, etc. This may change in the future.

Implementing temporary spaces requires temporary tuples to be
inserted into system spaces: tuples which are neither replicated or
persisted. This mostly done in on_replace_dd_* triggers by dropping the
txn->stmt->row.

Closes #8323

@TarantoolBot document
Title: Introduce fully temporary spaces with temporary metadata

Temporary spaces are now data-temporary spaces with temporary metadata.
Created by specifying { type = "temporary" } in the options.
Temporary spaces will not exist upon server restart and will not
exist on replicas. They can also be created in read-only mode.

Georgy Moshkin authored 1 year ago and Дмитрий Кольцов committed 1 year ago

Everywhere where we refer to temporary spaces we now say data-temporary.
This is because temporary spaces were never truly temporary because
their definitions would still be persisted and replicated and they
couldn't be created on read-only replicas. In a following commit we will
introduce a new fully temporary type of spaces, which will be just
called 'temporary', so this commit signifies this terminology change.

NO_DOC=renaming
NO_CHANGELOG=renaming
NO_TEST=renaming