Kurdakov Alexander authored 1 year ago and Dmitry Rodionov committed 1 year ago

Add to console lua api function that overrides sql executor

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Feodor Alexandrov authored 1 year ago and Dmitry Rodionov committed 1 year ago

Picodata is built statically, and name crypto clashes between
`src/lib/crypto` and libcrypto from `openssl`. To resolve the
ambiguity this patch renames tarantool library to `tcrypto`.

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Dmitry Ivanov authored 1 year ago and Dmitry Rodionov committed 1 year ago

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Maksim Kaitmazian authored 1 year ago and Dmitry Rodionov committed 1 year ago

In the picodata-support branch, the tarantool.a library is built
instead of a tarantool executable. So tests cannot be launched because
of the lack of the executable. This commit adds this executable.

NO_DOC=just an executable is added
NO_TEST=just an executable is added
NO_CHANGELOG=just an executable is added

Georgy Moshkin authored 1 year ago and Dmitry Rodionov committed 1 year ago

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Georgy Moshkin authored 2 years ago and Dmitry Rodionov committed 1 year ago

Was broken because `tarantool_free` checks if the current process is the
main one and not the child, which was forked at some point (at what
point?). This check was implemented by saving the original process's id
in the static variable master_pid, which got set when the code got
loaded the first time into memory.

So we broke this when we started forking the process in picodata, which
resulted in `master_pid` being set to the pid of the picodata's
"supervisor" process, which doesn't even enter tarantool runtime.

The fix is simple - save master_pid first thing when running
tarantool_main.

Georgy Moshkin authored 2 years ago and Dmitry Rodionov committed 1 year ago

- tarantool is now a static library
- main(argc, argv) -> tarantool_main(argc, argv, cb, cb_data)
- callback is called before running lua script

Closes #9237

Add exports for fiber_set_name_n, fiber_name, fiber_id, fiber_csw &
fiber_find.

Also make fiber_set_joinable, fiber_set_ctx & fiber_get_ctx interpret
NULL as the current fiber.

@TarantoolBot document
Title: add basic fiber api to ffi exports.

5 basic functions can now be used via ffi api, which were previously
only accessible via lua api: fiber_set_name_n, fiber_name, fiber_id,
fiber_csw & fiber_find.

fiber_set_joinable now interprets NULL as current fiber.

NO_DOC=internal change
NO_TEST=internal change

Rename the members as well. Keep uint16_t in box_check_acess_space
as it is for other exported functions

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Co-authored-by: Yaroslav Dynnikov <yaroslav.dynnikov@gmail.com>

This reverts commit 0d8458d4.

Alexey Protsenko authored 1 year ago and Yaroslav Dynnikov committed 1 year ago

Add a trigger to run tarantool-module CI tests when a new docker image
is released (on tag).

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

Introduce API for opening a read view over the specified spaces and
indexes and creating iterators over the index read views.

NO_DOC=picodata patch

box_auth_data_prepare() method declared to return a tuple while in
reality it returned a region allocated message pack string. Fixed.

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

test-run.py sets OMP_NUM_THREADS which causes this test to misbehave:
it doesn't expect to see a warning regarding the use of this env var
as the first log line.

Alternatively, we could update test-run's submodule, but in practice
this breaks other tests e.g. replication-luatest/linearizable_test.lua.

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

Ilya Verbin authored 1 year ago and Dmitry Ivanov committed 1 year ago

OpenMP is no longer used since commit 4f617b70 ("box: introduce
memtx_sort_threads config parameter"). All dependencies on libgomp
should be removed.

Follow-up #7689

NO_DOC=build
NO_TEST=build

Nikolay Shirokovskiy authored 1 year ago and Dmitry Ivanov committed 1 year ago

Closes #3389
Closes #7689
Closes #4646

@TarantoolBot document
Title: new box.cfg parameter memtx_sort_threads

The parameter sets the number of threads used to sort keys of secondary
indexes on loading memtx database. The parameter cannot be changed
dynamically (as it does not make sense).

Maximum value is 256, minimum is 1. Default is to use all available cores.

Usage example:
```
box.cfg{memtx_sort_threads=4}
```

Nikolay Shirokovskiy authored 1 year ago and Dmitry Ivanov committed 1 year ago

The algorithm runs sort in multiple threads and does not use OpenMP. It
has better threads utilization right from the beginning but probably
a worse constant than parallel qsort. See details in code comments.

Besides sort is not performed in calling thread but instead in spawned
worker threads. Calling thread yields waiting for worker threads to
finish. Exception is small data size, in this case sorting is executed
in calling thread saving time on spawning a thread. This should speed up
test execution. This is existing behaviour of qsort_arg but data size
threshold is reduced from 128000 to 1024.

Part of #3389

NO_CHANGELOG=internal
NO_DOC=internal

1. Current commit introduces 'box_auth_data_prepare()' to prepare
   a data string for any supported authentication methods.
2. The user name argument is refactored in the auth methods: the
   null-terminated string is replaced with an address range approach.
   Now Rust users don't need to re-allocate username with CString.
3. Password length type was set to uint32_t (previously it was size_t,
   int, uint32_t for different functions). Tarantool uses murmur3a,
   so all the hashed strings should be up to 32 bit long.

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Дмитрий Кибирев authored 1 year ago and Yaroslav Dynnikov committed 1 year ago

Add the temporary path for macos CI.

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Introduce new methods:

1. box_user_id_by_name - get the user identifier by name;
2. box_effective_user_id - get current effective user
   identifier;
3. box_session_user_id - get current session user identifier;
4. box_session_su - change current session user;

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Introduce fully temporary spaces: same as data-temporary space but with
temporary metadata. Basically temporary spaces now do not exist on
restart and do not exist on replicas. They can also be created, altered
and deleted when box.cfg.read_only = true.

To avoid conflicts with spaces created on replicas, the temporary
space ids by default start in a special range starting at
BOX_SPACE_ID_TEMPORARY_MIN.

Temporary spaces currently do not support several features e.g.
foreign key references (to and from), functional indexes, sql sequences,
sql triggers, etc. This may change in the future.

Implementing temporary spaces requires temporary tuples to be
inserted into system spaces: tuples which are neither replicated or
persisted. This mostly done in on_replace_dd_* triggers by dropping the
txn->stmt->row.

Closes #8323

@TarantoolBot document
Title: Introduce fully temporary spaces with temporary metadata

Temporary spaces are now data-temporary spaces with temporary metadata.
Created by specifying { type = "temporary" } in the options.
Temporary spaces will not exist upon server restart and will not
exist on replicas. They can also be created in read-only mode.

A tiny preparatory commit for meta-temporary spaces

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Move code that handles txn row counters into a separate function in
preparation of meta-temporary spaces introduction.

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Everywhere where we refer to temporary spaces we now say data-temporary.
This is because temporary spaces were never truly temporary because
their definitions would still be persisted and replicated and they
couldn't be created on read-only replicas. In a following commit we will
introduce a new fully temporary type of spaces, which will be just
called 'temporary', so this commit signifies this terminology change.

NO_DOC=renaming
NO_CHANGELOG=renaming
NO_TEST=renaming

Introduces a new field `type` to the space definition. Currently it can
only be "normal" or "data-temporary". It is backwards compatible with
temporary=true.

@TarantoolBot document
Title: Introduce space field type

A new space definition field "type" can now be used to specify the type
of the space. Usage: box.schema.create_space("s", { type = "normal" }).
Currently only 2 types are supported: "normal" & "data-temporary", which
is equivalent to { temporary = true }. Old-style { temporary = true } is
still supported, but only one option either 'temporary' or 'type' may be
specified at the same time.

Space type "temporary" will be introduced in a later commit.
In the future options "local", "synchronous", etc. may also be
supported.

NO_TEST=will be tested in the following commit

Aleksandr Lyapunov authored 1 year ago and Georgy Moshkin committed 1 year ago

The function update_view_references is called when an SQL view
is created or dropped. The goal of this function is to modify
(increment or decrement) view_ref_count member of spaces that
the view references.

There were a several issues that deserves to be refactored:
* By design in case of error it left the job partially done, so
  some space references were modified while some other - not.
  Although there was no bug since special steps were made in case
  of error, this pattern is inconvenient and should be avoided.
* In case of error the failing space name was returned via special
  argument which is not flexible and even requires allocation.
* Another argument - suppress_error - has actually never
  suppressed any error because the only case when an error could
  occur is creation of a view, which used suppress_error = false.
* Fail of that function was not actually covered with tests.

So this commit:
* Makes the function to do all or nothing.
* Forces the function to set diag by itself in case of error.
* Removes suppress_error argument while adding several asserts.\
* Adds a small test that fulfills coverage.

NO_DOC=refactoring
NO_CHANGELOG=reafactoring

Aleksandr Lyapunov authored 1 year ago and Georgy Moshkin committed 1 year ago

By design a newly created SrcList object contains one element
with NULL name. That was confusing and led to strange NULL checks
in a list that could not contain NULL names.

Fix it by clearing the list before usage.

NO_DOC=refactoring
NO_CHANGELOG=reafactoring
NO_TEST=refactoring

Aleksandr Lyapunov authored 1 year ago and Georgy Moshkin committed 1 year ago

Since we panic on OOM now, no OOM error handling is needed now.
Fix both internals of the function and how it is used in alter.

NO_DOC=refactoring
NO_CHANGELOG=reafactoring
NO_TEST=refactoring

Aleksandr Lyapunov authored 1 year ago and Georgy Moshkin committed 1 year ago

There must be a couple of rules:
* foreign key from non-temporary space to temporary space must be
  forbidden since after restart all existing links will be broken.
* foreign key from non-local space to local space must be forbidden
  on any replica all existing can be broken.

This patch implements the rules.

Closes #8936

NO_DOC=bugfix

Vladimir Davydov authored 1 year ago and Georgy Moshkin committed 1 year ago

To achieve that, we bypass the read-only check for the _truncate system
space in box_process1() and perform it in the on_replace system trigger
instead, when we know which space is truncated.

Note, we have to move the check for insertion of a new record into the
_truncate system space before the read-only check in the on_replace
trigger callback; this is needed for initial recovery with a non-empty
_truncate space to work. While we are at it, let's use recovery_state to
make the check explicit.

Closes #5616

@TarantoolBot document
Title: Mention that temp and local spaces can be truncated in ro mode

DML operations on temporary and local spaces can be performed even if
the instance is in the read-only mode, but DDL operations (such as
`alter`) are forbidden in this case[^1]. Technically, `truncate` is
a DDL operation so initially it was forbidden as well. However, it
should be safe to perform this operation on a temporary or local space
because logically it only modifies the data stored in the space (like
DML) and it isn't replicated (see tarantool/tarantool#4263). So starting
from Tarantool 2.11.1 we allow users to truncate temporary spaces in the
read-only mode.

[^1]: https://www.tarantool.io/en/doc/latest/concepts/replication/repl_architecture/#replication-local

Andrey Saranchin authored 1 year ago and Georgy Moshkin committed 1 year ago

Currently, _schema.max_id is used to generate sequentially growing
space ids. The main drawback of this approach is that generated space id
can be not unique if one mixes implicit and explicit space ids. Let's use
actual maximal space id to generate a new one, and scan for free id if
overflow happened.

Closes #8036

NO_DOC=bugfix