Maksim Kaitmazian authored 1 year ago and Dmitry Ivanov committed 1 year ago

This function is used to determine the authentication method when
authenticating a postgres client.

part of picodata/picodata/sbroad!292

NO_DOC=exports
NO_CHANGELOG=exports
NO_TEST=exports

These exports allow us to perform user authentication and
network communication.

part of picodata/picodata/sbroad!292

NO_DOC=exports
NO_CHANGELOG=exports
NO_TEST=exports

Georgy Moshkin authored 1 year ago and Maksim Kaitmazian committed 1 year ago

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Unfortunately, Centos 7 provides only openssl 1.0.2 (at lest if we
disregard epel), so we can't build the bundled libldap & libsasl2.
"Okay", one might think, "we can link against the distro's libs".
Well, turns out libldap 2.4, which is what we have to deal with in
that case, doesn't have ldap_connect!

Luckily, we don't have to connect explicitly. According to man pages:

```
ldap_init() acts just like ldap_open(), but does not open a connection
to the LDAP server.  The actual connection open will occur when the
first operation is attempted.

ldap_initialize()  acts  like ldap_init()...
```

This is still true for libldap up to and including version 2.6.

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Now it's possible to specify the desired authentication method during
user creation via `auth_type`, e.g.

```lua
box.schema.user.create('mickey', { auth_type = 'chap-sha1',
                                   password = 'foobar' })
```

Furthermore, authentication methods may now specify that they don't
require password to create stored authentication info. This is used
in LDAP authentication (`auth_type = 'ldap'`):

```lua
box.schema.user.create('mickey', { auth_type = 'ldap' })
```

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

This authentication method doesn't store any secrets; instead,
we delegate the whole auth to a pre-configured LDAP server. In
the method's implementation, we connect to the LDAP server and
perform a BIND operation which checks user's credentials.

Usage example:

```lua
-- Set the default auth method to LDAP and create a new user.
-- NOTE that we still have to provide a dummy password; otherwise
-- box.schema.user.create will setup an empty auth data.
box.cfg({auth_type = 'ldap'})
box.schema.user.create('demo', { password = '' })

-- Configure LDAP server connection URL and DN format string.
os = require('os')
os.setenv('TT_LDAP_URL', 'ldap://localhost:1389')
os.setenv('TT_LDAP_DN_FMT', 'cn=$USER,ou=users,dc=example,dc=org')

-- Authenticate using the LDAP authentication method via net.box.
conn = require('net.box').connect(uri, {
    user = 'demo',
    password = 'password',
    auth_type = 'ldap',
})
```

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

This is required for LDAP authentication, because we need
username to format the corresponding DN.

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

box.schema.user.passwd doesn't change the password for the current
user because new password is passed instead of the user name.

NO_CHANGELOG=fix an unreleased bug
NO_DOC=fix an unreleased bug

It fixes the following assertion
```bash
tarantool: ./src/lib/core/crypt.c:84: md5_encrypt:
Assertion `password_len + salt_len > 0' failed.
```
caused by the following code
```lua
box.cfg{auth_type='md5'}
box.schema.user.password("")
```

NO_CHANGELOG=fix an unreleased feature
NO_DOC=fix an unreleased feature

part of picodata/tarantool#21

NO_CHANGELOG=refactoring
NO_DOC=refactoring

Previous commit caused CI `pack` job
to fail on some linux distros. This
commit fixes the warnings from compiler.

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

- Add a configurable non-negative
session parameter "sql_vdbe_max_steps"
-- max number of opcodes that Vdbe
is allowed to execute for sql query.

- Default value can be specified in box.cfg.
If not set via box.cfg, default value
is 45000. Value 0 means that no
checks for number of executed Vdbe
opcodes will be made.

- Add the third argument to box.execute
function, that allows to specify options
for query execution. The only option
supported: sql_vdbe_max_steps. Usage
example:

```
box.execute([[select * from t]], {}, {{sql_vdbe_max_steps = 1000}})
```

part of picodata/picodata/sbroad!461

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch

It prevents password sniffing and avoids storing passwords on the
server in plain text but provides no protection if an attacker
manages to steal the password hash from the server.

Usage example:
```lua
-- Enable the md5 authentication method for all new users.
box.cfg({auth_type = 'md5'})

-- Reset existing user passwords to use the md5 authentication method.
box.schema.user.passwd('alice', 'topsecret')

-- Authenticate using the md5 authentication method via net.box.
conn = require('net.box').connect(uri, {
    user = 'alice',
    password = 'topsecret',
    -- Specifying the authentication method isn't strictly necessary:
    -- by default the client will use the method set in the remote
	-- server config (box.cfg.auth_type)
    auth_type = 'md5',
})
```

part of picodata/picodata/sbroad!377

@TarantoolBot document
Title: md5 authentication method

See the commit message.

User name is usually used as a salt for user password in order to
avoid password repeating.
For instance, postgres md5 authentication stores passwords as
md5("password", "user"), so that the same passwords are represented by
different hashes.

part of picodata/picodata/sbroad!377

@TarantoolBot document
Title: Document updated `box.schema.user.password` declaration.

Since auth methods can use user name for hashing, user name is
added to argument list of `box.schema.user.password`.

NO_TEST=there are no methods that use user name

Calling a `TRASH` macro after calling the `free`
function dereferences the pointer to the already
freed memory.

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Gleb Kashkin authored 2 years ago and Дмитрий Кольцов committed 1 year ago

There used to be a rare error when failed to connect via tarantoolctl to
listening cartridge console. It was caused by unclear
console.local_print() contract. Starting from gh-7031 fix, the function
assumed string-only arguments, while in some cases cdata error was
passed.

Now console.local_print() prints all non-string arguments as is, without
modifying potential local_eos.

Closes #8374

NO_DOC=bugfix
NO_TEST=very hard to test

Denis Smirnov authored 1 year ago and Дмитрий Кольцов committed 1 year ago

Picodata supports cluster-wide SQL and needs some predictable
method to calculate tuple hashes for the bucket ids. Method
should be available for Lua, C and Rust users. It was decided
to expose a murmur3 hash calculation method of the key_def module.

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch

godzie44 authored 1 year ago and Дмитрий Кольцов committed 1 year ago

Introduced a new type of cbus pipe - lcpipe. The current pipe in the
cbus - cpipe, has a number of limitations, first of all - the cpipe
cannot be used from the 3rd party threads, cpipe only works as a channel
between two cords. That why lcpipe is needed. Its main responsibility -
create channel between any thread and tarantool cord.

Internally lcpipe is a cpipe, but:
- on flush triggers removed, cause triggers use thread-local mem-pool,
this is not possible on a third party thread
- producer event loop removed, cause there is no libev event loop in
third party thread

Also, lcpipe interface is exported to the outside world.

NO_DOC=core feature

NO_DOC=disable feedback
NO_TEST=disable feedback

Vladimir Davydov authored 2 years ago and Дмитрий Кольцов committed 1 year ago

The index directory is created on demand since commit c00ba8e7
("xlog: make log directory if needed") and removed when it becomes
empty. There's no need to create it when an index is created anymore.

Follow-up #8441

NO_DOC=bugfix

Denis Smirnov authored 2 years ago and Дмитрий Кольцов committed 1 year ago

When vinyl space is dropped, its files are left on the file system
until GC removes them. At the moment GC removes only run files,
but not the root directory. These empty directories are never
removed and occupy 4KB on ext-family file systems each. In a case
of many dropped vinyl spaces it can become a serious disk space
and inode leak. Current commit makes gc always remove root directory
if there are no runs in it.

Closes #8441

NO_DOC=bugfix

NO_DOC=core feature
NO_TEST=no Lua API
NO_CHANGELOG=bugfix

Due to inconsistency of Tarantool type casting while using strict
data types as "double" or "unsigned" it is needed
to use "number" data type in a whole bunch of cases.
However "number" may contain "decimal" that will be serialized into
string by JSON builtin module.

This commit adds "encode_decimal_as_number" parameter to json.cfg{}.
That forces to encode `decimal` as JSON number to force type
consistency in JSON output.
Use with catious - most of JSON parsers assume that number is restricted
to float64.

NO_DOC=we do not host doc

Denis Smirnov authored 2 years ago and Дмитрий Кольцов committed 1 year ago

Actually there is no reason to throw an error and make a user
manually recreate prepared statement when it expires. A much more
user friendly way is to recreate it under hood when statement's
schema version differs from the box one.

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Denis Smirnov authored 2 years ago and Дмитрий Кольцов committed 1 year ago

Problem description.

When we prepare a statement with parameters in the result columns
(for example box.prepare('select ?')) Tarantool has no information
about the type of the output column and set it to default boolean.
Then, on the execution phase, the type would be recalculated during
the parameter binding.

Tarantool expects that there is no way for parameter to appear in the
result tuple other than exactly be mentioned in the final projection.
But it is incorrect - we can easily propagate parameter from the inner
part of the join. For example

box.prepare([[select COLUMN_1 from t1 join (values (?)) as t2 on true]])

In this case column COLUMN_1 in the final projection is not a
parameter, but a "reference" to it and its type depends on the
parameter from the inner part of the join. But as Tarantool
recalculates only binded parameters in the result projection,
it doesn't change the default boolean metadata type of the COLUMN_1
and the query fails on comparison with the actual type of the tuple.

Solution.
As we don't want to patch Vdbe to make COLUMN_1 refer inner parameter,
it was decided to make a simple workaround: change the default
column type from BOOLEAN to ANY for parameters. It fixes the
comparison with the actual tuple type (we do not fail), but in some
cases get ANY column in the results where we would like to have
explicitly defined type. Also NULL parameters would also have ANY
type, though Tarantool prefers to have BOOLEAN in this case.

Closes https://github.com/tarantool/tarantool/issues/7283

NO_DOC=bug fix

godzie44 authored 2 years ago and Дмитрий Кольцов committed 1 year ago

sql: add sql_execute_prepared_ext function, same as sql_execute_prepared but without `region` parameter
closes #2

NO_DOC=minor
NO_TEST=minor

godzie44 authored 2 years ago and Дмитрий Кольцов committed 1 year ago

- add box_tuple_data_offset function (return offset of the messagePack encoded data from the beginning of the tuple)
- add more export functions

closes #1

NO_DOC=build
NO_TEST=build

In some cases unsafe extension decoding was done without bound and type
checks: add necessary checks.

Closes tarantool/security#73

NO_DOC=bugfix

(cherry picked from commit 1de6a071)

This patch addresses coverity complain 1535241.

Follow-up #8047

NO_TEST=nit
NO_CHANGELOG=nit
NO_DOC=nit

(cherry picked from commit 089cbfa9)

If the 'after' key is less than the search key in case of ge/gt or
greater than the search key in case of le/lt, the iterator either
crashes (vinyl) or returns invalid result (memtx). This happens because
the engine implementation doesn't expect an invalid 'after' key.
Let's fix this by raising an error at the top level in case the 'after'
key doesn't meet the search criteria.

Closes #8403
Closes #8404

NO_DOC=bug fix
NO_CHANGELOG=unreleased

(cherry picked from commit c561202d)

Currently, if the position isn't compatible with the index, we raise
an error like "Invalid key part count ...". From this error it's
difficult to figure out whether it's for the given iterator position of
for the search key. Let's always raise ER_ITERATOR_POSITION in this
case. Later on we'll use stacked diag to add extra error info.

Needed for #8403
Needed for #8404

NO_DOC=bug fix
NO_CHANGELOG=unreleased

(cherry picked from commit 81d43c17)

We need to compare a tuple position with a search key in select() and
pairs() to make sure the tuple position meets the search criteria. The
problem is that we strip the MessagePack header from the position while
key_compare() takes keys with headers. Let's make key_compare take keys
without headers like the rest of comparator functions. Since in Vinyl we
often need to compare keys with headers, we also add vy_key_compare()
helper function.

Needed for #8403
Needed for #8404

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

(cherry picked from commit 41b8a012)

To improve customer experience it was decided to disable JIT engine on
Tarantool startup for macOS builds. Either way, JIT will be aboard as a
result of the changes and more adventurous users will be able to enable
it via <jit.on> in their code.

Furthermore, for convenient maintenance of JIT default behaviour CMake
configuration option "LUAJIT_JIT_STATUS" is introduced.

Closes #8252

NO_DOC=no behaviour changes

(cherry picked from commit ae0db476)

If the tuple pointed to by 'after' isn't in the space, eq pagination
would skip it because of the broken 'equals' flag update in the memtx
iterator.

Closes #8373

NO_DOC=bug fix
NO_CHANGELOG=unreleased

(cherry picked from commit bf275ed5)

Refactor logging of dangerous `select` call out of `check_select_safety`
and add it to `box.internal` so that it can be reused for read views.

Change all occurrences of 'dangerous' to 'long' to be consistent with the
log message.

Needed for tarantool/tarantool-ee#211

NO_CHANGELOG=<refactoring>
NO_DOC=<refactoring>
NO_TEST=<refactoring>

(cherry picked from commit 4221a983)

The `log_level' configuration parameter can be set as a number or a string.
When it is a string, cfg_geti() returns 0. Use log_default->level instead,
which is initialized earlier during box_init_say().

Closes #8287

NO_DOC=bugfix
NO_CHANGELOG=minor bug

(cherry picked from commit 41ead021)

This patch fixes an assertion or segmentation error if a FOREIGN KEY or
CHECK constraint is declared before the first column.

Closes #8392

NO_DOC=bugfix of the bug added in the current release
NO_CHANGELOG=bugfix of the bug added in the current release

We use coio_connect() to connect the replica to a remote peer. It
implies no timeout, and does a non-blocking connect() to the peer and
then waits for the socket to become writable indefinitely.

When the remote peer changes its IP address, connect() might try
connecting to the old address for as long as ~ 2 minutes (given the
default tcp_syn_retries value of 6).

This blocks replica from trying to reconnect to the updated address and
is pretty inconvenient.

Let's use coio_connect_timeout() instead and use
replication_disconnect_timeout() as a timeout, like everywhere else in
master-replica communication.

Closes #7294

NO_DOC=bugfix

(cherry picked from commit 0486a489)

Force recovery needs to follow the following logic: any unsuccessful system
space request must make recovery fail (including failure to decode an xrow,
when we are not sure we have finished processing system space request). If
the request is a non-insert one (e.g., raft or synchro) or addresses a user
space, it means we have finished processing system space requests, and from
this moment force recovery can be enabled — change the behaviour
accordingly.

We assume the request order in the snapshot is the following:
1. system space requests;
2. user space requests;
3. non-insert requests (e.g., raft or synchro).

Refactor the force recovery logic: add a enumeration to track snapshot
recovery state and add a new diagnostic for the case when the snapshot
contains has no system spaces.

Closes #7974

NO_DOC=bugfix

(cherry picked from commit b1095c1c)