NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

Ilya Verbin authored 2 years ago and Дмитрий Кольцов committed 1 year ago

Now a field can be assigned a default value in the space format. When a new
tuple is inserted into a space, and some of the fields contain null values,
those fields will be filled with their respective default values.

Closes #8157

@TarantoolBot document
Title: Document default field values
Product: Tarantool
Since: 3.0
Root document: https://www.tarantool.io/en/doc/latest/reference/reference_lua/box_space/format/

The format clause contains, for each field, a definition within braces:
`{name='...',type='...'[,is_nullable=...][,default=...]}`, where:

* the optional `default` value contains a default value for the field.
  Its type must be compatible with the field type. If default value is set,
  it is applied regardless of whether `is_nullable` is true or false.

Example:

```lua
tarantool> box.space.tester:format{
         > {name = 'id', type = 'unsigned'},
         > {name = 'name', type = 'string', default = 'Noname'},
         > {name = 'pass', type = 'string'},
         > {name = 'shell', type = 'string', default = '/bin/sh'}}
---
...

tarantool> box.space.tester:insert{1000, nil, 'qwerty'}
---
- [1000, 'Noname', 'qwerty', '/bin/sh']
...
```

Ilya Verbin authored 1 year ago and Дмитрий Кольцов committed 1 year ago

It encapsulates the logic that helps to build a new MsgPack array by
concatenating tuple fields from various locations. The idea is to
postpone memory allocation and copying until the finalization.

Needed for #8157

NO_DOC=internal
NO_CHANGELOG=internal

Georgy Moshkin authored 1 year ago and Maksim Kaitmazian committed 1 year ago

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

This authentication method doesn't store any secrets; instead,
we delegate the whole auth to a pre-configured LDAP server. In
the method's implementation, we connect to the LDAP server and
perform a BIND operation which checks user's credentials.

Usage example:

```lua
-- Set the default auth method to LDAP and create a new user.
-- NOTE that we still have to provide a dummy password; otherwise
-- box.schema.user.create will setup an empty auth data.
box.cfg({auth_type = 'ldap'})
box.schema.user.create('demo', { password = '' })

-- Configure LDAP server connection URL and DN format string.
os = require('os')
os.setenv('TT_LDAP_URL', 'ldap://localhost:1389')
os.setenv('TT_LDAP_DN_FMT', 'cn=$USER,ou=users,dc=example,dc=org')

-- Authenticate using the LDAP authentication method via net.box.
conn = require('net.box').connect(uri, {
    user = 'demo',
    password = 'password',
    auth_type = 'ldap',
})
```

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

box.schema.user.passwd doesn't change the password for the current
user because new password is passed instead of the user name.

NO_CHANGELOG=fix an unreleased bug
NO_DOC=fix an unreleased bug

It fixes the following assertion
```bash
tarantool: ./src/lib/core/crypt.c:84: md5_encrypt:
Assertion `password_len + salt_len > 0' failed.
```
caused by the following code
```lua
box.cfg{auth_type='md5'}
box.schema.user.password("")
```

NO_CHANGELOG=fix an unreleased feature
NO_DOC=fix an unreleased feature

part of picodata/tarantool#21

NO_CHANGELOG=refactoring
NO_DOC=refactoring

- Add a configurable non-negative
session parameter "sql_vdbe_max_steps"
-- max number of opcodes that Vdbe
is allowed to execute for sql query.

- Default value can be specified in box.cfg.
If not set via box.cfg, default value
is 45000. Value 0 means that no
checks for number of executed Vdbe
opcodes will be made.

- Add the third argument to box.execute
function, that allows to specify options
for query execution. The only option
supported: sql_vdbe_max_steps. Usage
example:

```
box.execute([[select * from t]], {}, {{sql_vdbe_max_steps = 1000}})
```

part of picodata/picodata/sbroad!461

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch

It prevents password sniffing and avoids storing passwords on the
server in plain text but provides no protection if an attacker
manages to steal the password hash from the server.

Usage example:
```lua
-- Enable the md5 authentication method for all new users.
box.cfg({auth_type = 'md5'})

-- Reset existing user passwords to use the md5 authentication method.
box.schema.user.passwd('alice', 'topsecret')

-- Authenticate using the md5 authentication method via net.box.
conn = require('net.box').connect(uri, {
    user = 'alice',
    password = 'topsecret',
    -- Specifying the authentication method isn't strictly necessary:
    -- by default the client will use the method set in the remote
	-- server config (box.cfg.auth_type)
    auth_type = 'md5',
})
```

part of picodata/picodata/sbroad!377

@TarantoolBot document
Title: md5 authentication method

See the commit message.

User name is usually used as a salt for user password in order to
avoid password repeating.
For instance, postgres md5 authentication stores passwords as
md5("password", "user"), so that the same passwords are represented by
different hashes.

part of picodata/picodata/sbroad!377

@TarantoolBot document
Title: Document updated `box.schema.user.password` declaration.

Since auth methods can use user name for hashing, user name is
added to argument list of `box.schema.user.password`.

NO_TEST=there are no methods that use user name

Denis Smirnov authored 1 year ago and Дмитрий Кольцов committed 1 year ago

Tuple hash calculation tests for the C API were incorrect. Thanks
to the full pipeline with DEBUG build we detected the problem and
fixed it.

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch

Denis Smirnov authored 1 year ago and Дмитрий Кольцов committed 1 year ago

Picodata supports cluster-wide SQL and needs some predictable
method to calculate tuple hashes for the bucket ids. Method
should be available for Lua, C and Rust users. It was decided
to expose a murmur3 hash calculation method of the key_def module.

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch

godzie44 authored 1 year ago and Дмитрий Кольцов committed 1 year ago

Introduced a new type of cbus pipe - lcpipe. The current pipe in the
cbus - cpipe, has a number of limitations, first of all - the cpipe
cannot be used from the 3rd party threads, cpipe only works as a channel
between two cords. That why lcpipe is needed. Its main responsibility -
create channel between any thread and tarantool cord.

Internally lcpipe is a cpipe, but:
- on flush triggers removed, cause triggers use thread-local mem-pool,
this is not possible on a third party thread
- producer event loop removed, cause there is no libev event loop in
third party thread

Also, lcpipe interface is exported to the outside world.

NO_DOC=core feature

NO_DOC=disable feedback
NO_TEST=disable feedback

Vladimir Davydov authored 1 year ago and Дмитрий Кольцов committed 1 year ago

The index directory is created on demand since commit c00ba8e7
("xlog: make log directory if needed") and removed when it becomes
empty. There's no need to create it when an index is created anymore.

Follow-up #8441

NO_DOC=bugfix

Denis Smirnov authored 2 years ago and Дмитрий Кольцов committed 1 year ago

When vinyl space is dropped, its files are left on the file system
until GC removes them. At the moment GC removes only run files,
but not the root directory. These empty directories are never
removed and occupy 4KB on ext-family file systems each. In a case
of many dropped vinyl spaces it can become a serious disk space
and inode leak. Current commit makes gc always remove root directory
if there are no runs in it.

Closes #8441

NO_DOC=bugfix

Due to inconsistency of Tarantool type casting while using strict
data types as "double" or "unsigned" it is needed
to use "number" data type in a whole bunch of cases.
However "number" may contain "decimal" that will be serialized into
string by JSON builtin module.

This commit adds "encode_decimal_as_number" parameter to json.cfg{}.
That forces to encode `decimal` as JSON number to force type
consistency in JSON output.
Use with catious - most of JSON parsers assume that number is restricted
to float64.

NO_DOC=we do not host doc

Denis Smirnov authored 2 years ago and Дмитрий Кольцов committed 1 year ago

Previously,

select "t1"."a" from (select "a" from "t") as "t1";

returned a result column name `t1` instead of `t1.a` because of
incorrect work of a dequoting function. The reason was that
previously sqlDequote() function finished its work when found the
first closing quote.

Old logic worked for simple selects where the column name doesn't
contain an explicit scan name ("a" -> a).
But for the sub-queries results sqlDequote() finished its work right
after the scan name ("t1"."a" -> t1). Now the function continues its
deqouting till it gets the null terminator at the end of the string.

Closes #7063

NO_DOC=don't change any public API, only a bug fix

Co-authored-by: Mergen Imeev <imeevma@gmail.com>

Denis Smirnov authored 2 years ago and Дмитрий Кольцов committed 1 year ago

Actually there is no reason to throw an error and make a user
manually recreate prepared statement when it expires. A much more
user friendly way is to recreate it under hood when statement's
schema version differs from the box one.

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Denis Smirnov authored 2 years ago and Дмитрий Кольцов committed 1 year ago

Problem description.

When we prepare a statement with parameters in the result columns
(for example box.prepare('select ?')) Tarantool has no information
about the type of the output column and set it to default boolean.
Then, on the execution phase, the type would be recalculated during
the parameter binding.

Tarantool expects that there is no way for parameter to appear in the
result tuple other than exactly be mentioned in the final projection.
But it is incorrect - we can easily propagate parameter from the inner
part of the join. For example

box.prepare([[select COLUMN_1 from t1 join (values (?)) as t2 on true]])

In this case column COLUMN_1 in the final projection is not a
parameter, but a "reference" to it and its type depends on the
parameter from the inner part of the join. But as Tarantool
recalculates only binded parameters in the result projection,
it doesn't change the default boolean metadata type of the COLUMN_1
and the query fails on comparison with the actual type of the tuple.

Solution.
As we don't want to patch Vdbe to make COLUMN_1 refer inner parameter,
it was decided to make a simple workaround: change the default
column type from BOOLEAN to ANY for parameters. It fixes the
comparison with the actual tuple type (we do not fail), but in some
cases get ANY column in the results where we would like to have
explicitly defined type. Also NULL parameters would also have ANY
type, though Tarantool prefers to have BOOLEAN in this case.

Closes https://github.com/tarantool/tarantool/issues/7283

NO_DOC=bug fix

In some cases unsafe extension decoding was done without bound and type
checks: add necessary checks.

Closes tarantool/security#73

NO_DOC=bugfix

(cherry picked from commit 1de6a071)

If the 'after' key is less than the search key in case of ge/gt or
greater than the search key in case of le/lt, the iterator either
crashes (vinyl) or returns invalid result (memtx). This happens because
the engine implementation doesn't expect an invalid 'after' key.
Let's fix this by raising an error at the top level in case the 'after'
key doesn't meet the search criteria.

Closes #8403
Closes #8404

NO_DOC=bug fix
NO_CHANGELOG=unreleased

(cherry picked from commit c561202d)

Currently, if the position isn't compatible with the index, we raise
an error like "Invalid key part count ...". From this error it's
difficult to figure out whether it's for the given iterator position of
for the search key. Let's always raise ER_ITERATOR_POSITION in this
case. Later on we'll use stacked diag to add extra error info.

Needed for #8403
Needed for #8404

NO_DOC=bug fix
NO_CHANGELOG=unreleased

(cherry picked from commit 81d43c17)

We need to compare a tuple position with a search key in select() and
pairs() to make sure the tuple position meets the search criteria. The
problem is that we strip the MessagePack header from the position while
key_compare() takes keys with headers. Let's make key_compare take keys
without headers like the rest of comparator functions. Since in Vinyl we
often need to compare keys with headers, we also add vy_key_compare()
helper function.

Needed for #8403
Needed for #8404

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

(cherry picked from commit 41b8a012)

Regular testing routine on macOS/M1 platforms will be enabled back in
the last patch of the series, but there are some failures unrelated to
the changes made within the previous commits. Hence it was decided to
add the skip_if directive to app-luatest/fiber_parent_backtrace_test.lua
for now.

Relates to tarantool/tarantool-qa#309

NO_DOC=tests
NO_CHANGELOG=tests

(cherry picked from commit f1fc70f4)

Regular testing routine on macOS/M1 platforms will be enabled back in
the last patch of the series, but there are some failures unrelated to
the changes made within the previous commits. Hence it was decided to
add the skipcond unit to app/fiber.test.lua for now.

Relates to tarantool/tarantool-qa#308

NO_DOC=tests
NO_CHANGELOG=tests

(cherry picked from commit dfa2f128)

To improve customer experience it was decided to disable JIT engine on
Tarantool startup for macOS builds. Either way, JIT will be aboard as a
result of the changes and more adventurous users will be able to enable
it via <jit.on> in their code.

Furthermore, for convenient maintenance of JIT default behaviour CMake
configuration option "LUAJIT_JIT_STATUS" is introduced.

Closes #8252

NO_DOC=no behaviour changes

(cherry picked from commit ae0db476)

Since luatest provides <skip_if> helper, .skipcond file is not necessary
anymore and is removed.

Needed for #8252

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

(cherry picked from commit 3548afdb)

As a result of the patch, all required files are moved to app-luatest
subdirectory. The test itself is rewritten via <justrun> and <treegen>
helpers to make it more clear and easy to maintain.

Since luatest provides <skip_if> helper, .skipcond file is not necessary
anymore and is removed.

Needed for #8252

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

(cherry picked from commit 2369b8d4)

As a result of the patch, all required files are moved to app-luatest
subdirectory. introducing the following layout:

  app-luatest/
  |- <name>_test.lua
  |- lib/
     |- CMakeLists.txt
     |- <libname>.c

The test itself is rewritten via <justrun> and <treegen> helpers to make
it more clear and easy to maintain.

Since luatest provides <skip_if> helper, .skipcond file is not necessary
anymore and is removed.

Needed for #8252

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

(cherry picked from commit 455a862c)

If the tuple pointed to by 'after' isn't in the space, eq pagination
would skip it because of the broken 'equals' flag update in the memtx
iterator.

Closes #8373

NO_DOC=bug fix
NO_CHANGELOG=unreleased

(cherry picked from commit bf275ed5)

The `log_level' configuration parameter can be set as a number or a string.
When it is a string, cfg_geti() returns 0. Use log_default->level instead,
which is initialized earlier during box_init_say().

Closes #8287

NO_DOC=bugfix
NO_CHANGELOG=minor bug

(cherry picked from commit 41ead021)

This patch fixes an assertion or segmentation error if a FOREIGN KEY or
CHECK constraint is declared before the first column.

Closes #8392

NO_DOC=bugfix of the bug added in the current release
NO_CHANGELOG=bugfix of the bug added in the current release

We use coio_connect() to connect the replica to a remote peer. It
implies no timeout, and does a non-blocking connect() to the peer and
then waits for the socket to become writable indefinitely.

When the remote peer changes its IP address, connect() might try
connecting to the old address for as long as ~ 2 minutes (given the
default tcp_syn_retries value of 6).

This blocks replica from trying to reconnect to the updated address and
is pretty inconvenient.

Let's use coio_connect_timeout() instead and use
replication_disconnect_timeout() as a timeout, like everywhere else in
master-replica communication.

Closes #7294

NO_DOC=bugfix

(cherry picked from commit 0486a489)

Add missing `server:drop()` to digest_crc32_recording_test.lua to avoid
running tarantool processes after the test.

NO_DOC=test fix
NO_TEST=test fix
NO_CHANGELOG=test fix

(cherry picked from commit 5cbb0daf)

The test fails with

 fail | 2023-03-01 15:54:30.550 [3724975] main/103/server_instance.lua
 F> can't initialize storage: unlink, called on fd 63,
 aka unix/:(socket), peer of unix/:(socket): Address already in use

We fixed a similar issue in commit 3d3e9dea ("test: fix flaky
box-luatest/gh_7917_log_row_on_recovery_error_test") by using unique
instance names. Let's do the same here.

Fixes commit b1095c1c ("memtx: fix force recovery handling").
Follow-up #7974

NO_DOC=test
NO_CHANGELOG=test

(cherry picked from commit 1490228a)

Force recovery needs to follow the following logic: any unsuccessful system
space request must make recovery fail (including failure to decode an xrow,
when we are not sure we have finished processing system space request). If
the request is a non-insert one (e.g., raft or synchro) or addresses a user
space, it means we have finished processing system space requests, and from
this moment force recovery can be enabled — change the behaviour
accordingly.

We assume the request order in the snapshot is the following:
1. system space requests;
2. user space requests;
3. non-insert requests (e.g., raft or synchro).

Refactor the force recovery logic: add a enumeration to track snapshot
recovery state and add a new diagnostic for the case when the snapshot
contains has no system spaces.

Closes #7974

NO_DOC=bugfix

(cherry picked from commit b1095c1c)

Follows up #8363

NO_CHANGELOG=see previous commit
NO_DOC=added a test

(cherry picked from commit cccd1f9a)

Patch fixes a bug when body in response couldn't be decoded:

NO_WRAP
```
tarantool> httpc = require('http.client').new()
tarantool> response = httpc:get('https://jsonplaceholder.typicode.com/todos/1'

)
tarantool> response:decode()

---
- error: 'builtin/http.client.lua:301: attempt to index field ''decoders'' (a nil
    value)'
...
```
NO_WRAP

Now response object contains table with decoders defined by user in his
http client instance. We hide this table on response serialization by
adding underscore because decoders there is not a part of API.

Reported-by: Alexander Turenko <alexander.turenko@tarantool.org>

Fixes #8363

NO_DOC=bugfix

(cherry picked from commit 83168b25)