- If you prepare and execute statement with params
in the projection and then unprepare the
statement, byte counter may show the wrong value
or even overflow.
- The problem is that when we compile sql
statement, we set parameter type to 'any'.
But when we execute the statement we set parameter
type to actual type. Then we use this type in
calculation of estimated of sql cache entry size.
This leads to different estimated sizes of cache
entry during prepare and during unprepare after
statement was executed
- Fix this by resetting type to 'any' after
executing the statement

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

The following commit introduced a tautological if expression:

```gitcommit
sql: introduce structs assembling DDL arguments during parsing (ba56b145fafaa3)
```

Due to the changes in commit

```gitcommit
sql: refactor memory allocation system (cb777136dd7a0c)
```

the allocations in sql expressions became infallible, which means
that we may safely fix static analysis warnings by dropping the
tautological comparison altogether.

Original patch by Feodor Alexandrov.

Dmitry Rodionov authored 1 year ago and Дмитрий Кольцов committed 1 year ago

Rename the members as well. Keep uint16_t in box_check_acess_space
as it is for other exported functions

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Georgy Moshkin authored 1 year ago and Дмитрий Кольцов committed 1 year ago

Introduce fully temporary spaces: same as data-temporary space but with
temporary metadata. Basically temporary spaces now do not exist on
restart and do not exist on replicas. They can also be created, altered
and deleted when box.cfg.read_only = true.

To avoid conflicts with spaces created on replicas, the temporary
space ids by default start in a special range starting at
BOX_SPACE_ID_TEMPORARY_MIN.

Temporary spaces currently do not support several features e.g.
foreign key references (to and from), functional indexes, sql sequences,
sql triggers, etc. This may change in the future.

Implementing temporary spaces requires temporary tuples to be
inserted into system spaces: tuples which are neither replicated or
persisted. This mostly done in on_replace_dd_* triggers by dropping the
txn->stmt->row.

Closes #8323

@TarantoolBot document
Title: Introduce fully temporary spaces with temporary metadata

Temporary spaces are now data-temporary spaces with temporary metadata.
Created by specifying { type = "temporary" } in the options.
Temporary spaces will not exist upon server restart and will not
exist on replicas. They can also be created in read-only mode.

Georgy Moshkin authored 2 years ago and Дмитрий Кольцов committed 1 year ago

Introduces a new field `type` to the space definition. Currently it can
only be "normal" or "data-temporary". It is backwards compatible with
temporary=true.

@TarantoolBot document
Title: Introduce space field type

A new space definition field "type" can now be used to specify the type
of the space. Usage: box.schema.create_space("s", { type = "normal" }).
Currently only 2 types are supported: "normal" & "data-temporary", which
is equivalent to { temporary = true }. Old-style { temporary = true } is
still supported, but only one option either 'temporary' or 'type' may be
specified at the same time.

Space type "temporary" will be introduced in a later commit.
In the future options "local", "synchronous", etc. may also be
supported.

NO_TEST=will be tested in the following commit

Aleksandr Lyapunov authored 1 year ago and Дмитрий Кольцов committed 1 year ago

By design a newly created SrcList object contains one element
with NULL name. That was confusing and led to strange NULL checks
in a list that could not contain NULL names.

Fix it by clearing the list before usage.

NO_DOC=refactoring
NO_CHANGELOG=reafactoring
NO_TEST=refactoring

Aleksandr Lyapunov authored 1 year ago and Дмитрий Кольцов committed 1 year ago

Since we panic on OOM now, no OOM error handling is needed now.
Fix both internals of the function and how it is used in alter.

NO_DOC=refactoring
NO_CHANGELOG=reafactoring
NO_TEST=refactoring

Ilya Verbin authored 2 years ago and Дмитрий Кольцов committed 1 year ago

In order to avoid collision with the upcoming core default_value.

Also rename tuple_field::default_value_expr to
tuple_field::sql_default_value_expr.

Part of #8157

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Arseniy Volynets authored 1 year ago and Дмитрий Кольцов committed 1 year ago

Previous commit caused CI `pack` job
to fail on some linux distros. This
commit fixes the warnings from compiler.

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Arseniy Volynets authored 1 year ago and Дмитрий Кольцов committed 1 year ago

- Add a configurable non-negative
session parameter "sql_vdbe_max_steps"
-- max number of opcodes that Vdbe
is allowed to execute for sql query.

- Default value can be specified in box.cfg.
If not set via box.cfg, default value
is 45000. Value 0 means that no
checks for number of executed Vdbe
opcodes will be made.

- Add the third argument to box.execute
function, that allows to specify options
for query execution. The only option
supported: sql_vdbe_max_steps. Usage
example:

```
box.execute([[select * from t]], {}, {{sql_vdbe_max_steps = 1000}})
```

part of picodata/picodata/sbroad!461

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch

Denis Smirnov authored 2 years ago and Дмитрий Кольцов committed 1 year ago

Problem description.

When we prepare a statement with parameters in the result columns
(for example box.prepare('select ?')) Tarantool has no information
about the type of the output column and set it to default boolean.
Then, on the execution phase, the type would be recalculated during
the parameter binding.

Tarantool expects that there is no way for parameter to appear in the
result tuple other than exactly be mentioned in the final projection.
But it is incorrect - we can easily propagate parameter from the inner
part of the join. For example

box.prepare([[select COLUMN_1 from t1 join (values (?)) as t2 on true]])

In this case column COLUMN_1 in the final projection is not a
parameter, but a "reference" to it and its type depends on the
parameter from the inner part of the join. But as Tarantool
recalculates only binded parameters in the result projection,
it doesn't change the default boolean metadata type of the COLUMN_1
and the query fails on comparison with the actual type of the tuple.

Solution.
As we don't want to patch Vdbe to make COLUMN_1 refer inner parameter,
it was decided to make a simple workaround: change the default
column type from BOOLEAN to ANY for parameters. It fixes the
comparison with the actual tuple type (we do not fail), but in some
cases get ANY column in the results where we would like to have
explicitly defined type. Also NULL parameters would also have ANY
type, though Tarantool prefers to have BOOLEAN in this case.

Closes https://github.com/tarantool/tarantool/issues/7283

NO_DOC=bug fix

According to ANSI, EXISTS is a predicate that tests a given subquery and
returns true if it returns more than 0 rows, false otherwise. However,
after 2a720d11, EXISTS worked correctly only if there were exactly 0
or 1 rows, and in all other cases it gave an error. This patch makes
EXITS work properly.

Closes #8676

NO_DOC=bugfix

(cherry picked from commit a5e498d1)

This patch removes some deprecated code. This code had no user-visible
effect, but caused problems when running the test with ASAN enabled.

Closes #8761

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

(cherry picked from commit d63a4bf2)

Regular region implementation supports allocations of size 0 with no
extra efforts. It returns a non-NULL pointer in this case. However in
case of ASAN friendly implementation it will require a special care for
this case. Instead let's avaid allocations if size 0 for region.

Also use xregion_ macros for allocations. Our current policy is to panic
on OOM on runtime allocations.

Part of tarantool/tarantool#7327

NO_TEST=internal
NO_CHANGELOG=internal
NO_DOC=internal

(cherry picked from commit 8159347d)

This patch replaces region_*() functions with xregion_*() functions.

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

(cherry picked from commit 1ba84fe3)

This patch removes the 'size' argument from macros, as it was only used
to set an error on failure, which is not possible for x* versions. In
addition, both macros now cast the value to the specified type, as is
done in the original macros.

Closes #8522

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

(cherry picked from commit ae02f0cd)

This patch fixes SQL memory leaks found by static analyzers and SQL
fuzzer.

Part of tarantool/security#120

NO_DOC=fix for memleak
NO_TEST=fix for memleak
NO_CHANGELOG=fix for memleak

(cherry picked from commit cd173ce5)

Before this patch, if an index was created due to a column's UNIQUE
constraint or a column's PRIMARY KEY constraint before adding a
collation, and if the column's fieldno was not equal to the index's
position in space->index, the collation would not be assigned to the
index.

Also, this patch fixes an assertion in debug build for the case when an
index with more that one field was created before a collation was added.

Closes #9229

NO_DOC=bugfix

(cherry picked from commit 65608d87)

If non-terminal symbol is referenced in C code then destructor for
expression is not called. Thus we don't need to duplicate. Otherwise we
got a memory leak.

See https://www.sqlite.org/cgi/src/doc/trunk/doc/lemon.html#destructor

Close #9159

NO_DOC=bugfix
NO_TEST=tested by debug ASAN CI (to be turned on)

(cherry picked from commit 36ef3fb4)

With this option enabled (new), the multiresults returned by a stored C
function via iproto aren't wrapped in the additional msgpack array (old).

Due to new behaviour some renames are performed:
* `port_c_dump_msgpack()` -> `port_c_dump_msgpack_wrapped()`, since this
  is dump format with additional msgpack array encoded.
* `port_c_dump_msgpack16()` -> `port_c_dump_msgpack()`, since this
  format is now the default new format of a msgpack dump.

The behaviour of the C port msgpack dumping depends on the
`c_func_iproto_multireturn` option:
* uses `port_c_dump_msgpack()` if set to true (new),
* uses `port_c_dump_msgpack_wrapped()` otherwise (old).

Needed for #4799

@TarantoolBot document
Title: Document `c_func_iproto_multireturn` compat option

Please create a documentation page for the new compat option:
https://tarantool.io/compat/c_func_iproto_multireturn

In the new behaviour, the multiresults returned by a stored C function
via iproto aren't wrapped in the additional msgpack array (old).

```
tarantool> compat.c_func_iproto_multireturn = 'old'
---
...

tarantool> net_box.connect(box.cfg.listen):call('myclib.cfunc')
---
- [true, -1]
...

tarantool> compat.c_func_iproto_multireturn = 'new'
---
...

tarantool> net_box.connect(box.cfg.listen):call('myclib.cfunc')
---
- true
- -1
...

```

The new behaviour is consistent with the local call of the function
via `box.func`:

```
tarantool> box.func['myclib.cfunc']:call()
---
- true
- -1
...

```

Assume you have a stored C function that returns values like the
following:

```c
char *position = mp_encode_bool(buffer, true);
box_return_mp(ctx, buffer, position);
/* ... */
position = mp_encode_int(buffer, -1);
box_return_mp(ctx, buffer, position);
```

If you want to preserve the format of the returned array for your C
functions, when the `c_func_iproto_multireturn` option is set to "new",
you should add the additional wrapping, like the following:

```c
char *position = mp_encode_array(buffer_with_results, n_results);
position = mp_encode_bool(position, true);
/* ... */
position = mp_encode_int(position, -1);
box_return_mp(ctx, buffer_with_results, position);
```

The amount of `box_return_mp()` calls indicates the number of values to
be returned.

Also, you should update its usage via `box.func` if there is any.

(cherry picked from commit 96ee6d9b)

This patch fixes an issue in generate_column_metadata(). Prior to this
patch, the number of variable-only expressions was counted incorrectly
when temporary memory was allocated on region to store their positions.
However, although this allocation was incorrect, this did not lead to
any problems due to the specifics of the region allocations.

This patch fixes this by removing the temporary memory allocation.

Closes #8763

NO_DOC=no user-visible changes
NO_TEST=no user-visible changes
NO_CHANGELOG=no user-visible changes

(cherry picked from commit d4f143ad)

Currently on tuple encoding failure we raise Lua error. In many placess
the error is not handled in Lua C code and we get misc leaks. Let's
instead pass error as return value.

Note that generally speaking encoding code can raise an error on OOM.
Which will lead to leak again. Hopefully application will be killed by
OOM killer instead. Other then that we expect no more errors in the
code. If code calls a user defined callback then pcall is used (see
lua_field_inspect_ucdata for example). So the turn from raising errors
to returning error code seems the right direction.

Closes #7939

NO_DOC=bugfix

(cherry picked from commit 9f9142d6)

This patch adds a check that sqlXPrintf() does not fail in the built-in
SQL function printf(). There are two possible problems: the result might
get too large, or there might be an integer overflow because internally
int values are converted to size_t.

Closes #tarantool/security#122

NO_DOC=bugfix

(cherry picked from commit 13159230)

This patch fixes problems with INSERT INTO ... SELECT FROM optimization.
These problems appeared after 6b8acd8f, where the check became redundant,
but was not updated. Two problems arose:
1) an assertion or segmentation fault when optimization was used and the
source space does not have an index;
2) optimization can be used even if the indexes are incompatible.

The second problem does not result in changes that are user-visible, so
there is no test.

Closes #8661

NO_DOC=bugfix

(cherry picked from commit 039f714d)

This patch limits the amount of allocated memory in the built-in
REPLACE() function.

Follow-up tarantool/security#119

NO_DOC=bugfix
NO_CHANGELOG=already exists

(cherry picked from commit be3034ab)

This patch replaces the type for some int and uint32_t values with
size_t to avoid problems with integer overflow.

Closes tarantool/security#119

NO_DOC=bugfix

(cherry picked from commit 60a187d7)

Currently, _schema.max_id is used to generate sequentially growing
space ids. The main drawback of this approach is that generated space id
can be not unique if one mixes implicit and explicit space ids. Let's use
actual maximal space id to generate a new one, and scan for free id if
overflow happened.

Closes #8036

NO_DOC=bugfix

(cherry picked from commit 697123d0)

This patch fixes incorrect conversion of an integer greater than
INT64_MAX or less than 0 to decimal during SQL arithmetic operations.

Closes #8460

NO_DOC=bugfix

(cherry picked from commit 1e660dcf)

This patch prohibits the use of ARRAY, MAP and INTERVAL in ORDER BY.
In addition, GROUP BY now also checks the types of the arguments when
building the VDBE.

Closes #6668

NO_DOC=bugfix

(cherry picked from commit 0ead015b)

This patch makes SQL to support collations for the ANY type.

Closes #8070

NO_DOC=ANY already supports collations in BOX.

(cherry picked from commit da8336ce)

In some cases unsafe extension decoding was done without bound and type
checks: add necessary checks.

Closes tarantool/security#73

NO_DOC=bugfix

(cherry picked from commit 1de6a071)

This patch fixes an assertion or segmentation error if a FOREIGN KEY or
CHECK constraint is declared before the first column.

Closes #8392

NO_DOC=bugfix of the bug added in the current release
NO_CHANGELOG=bugfix of the bug added in the current release

This patch introduces new sql_seq_scan_default compat option. This
compat option allows to set default value for sql_seq_scan session
setting. Note that sql_seq_scan_default compat option only affects
sessions during initialization. This means that you should set
sql_seq_scan_default before running box.cfg{} or creating a new session.

Closes #8096

NO_DOC=Already exists

This patch fixes an issue with checking the result of sql_get_coll_seq()
in sql_expr_coll(). This fix only changes the error if the collation
combination is invalid because sql_get_coll_seq() sets the is_aborted
flag and error will be thrown in any case.

Closes tarantool/security#80

NO_DOC=change of returned error in rare case
NO_CHANGELOG=change of returned error in rare case

Some of the SQL modules have not been used for a long time. This patch
drops these modules.

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Fixed pthread-related CMake checks. The checks code is built with
`-pedantic-errors` and it leads to errors of the following kind on
clang 15:

```
<...>/CMakeFiles/CMakeScratch/TryCompile-78KaOK/src.c:4:17: error: a
    function declaration without a prototype is deprecated in all
    versions of C [-Werror,-Wstrict-prototypes]
        int main() { pthread_setname_np(pthread_self(), ""); }
                ^
                 void
```

Fixed a warning in the SQL code (it's an error in Debug build):

```
<...>/src/box/sql/vdbeaux.c:170:13: error: variable 'n' set but not used
    [-Werror,-Wunused-but-set-variable]
        static int n = 0;
```

Fixed several warnings from lemon.c of the following kind:

```
<...>/extra/lemon.c:173:6: warning: a function declaration without a
    prototype is deprecated in all versions of C and is treated as a
    zero-parameter prototype in C2x, conflicting with a subsequent
    definition [-Wdeprecated-non-prototype]
void FindRulePrecedences();
     ^
<...>/extra/lemon.c:766:6: note: conflicting prototype is here
void FindRulePrecedences(struct lemon *xp)
```

See also https://github.com/tarantool/small/issues/57

Fixes #8110

NO_DOC=build fix
NO_TEST=build fix

This patch removes unused variables that were not caught by the compiler
due to MAYBE_UNUSED or conversion to void.

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

This patch introduces new keyword SEQSCAN and new restrictions on
SELECTs. These restrictions are disabled by default.

Closes #7747

@TarantoolBot document
Title: SEQSCAN

Now scanning SELECT will not run and will throw an error if the new
SEQSCAN keyword is not used for scanned spaces. This change only affects
SELECT and does not affect UPDATE and DELETE. A SELECT is recognized as
a scanning SELECT if `EXPLAIN QUERY PLAN SELECT ...` indicates that the
SELECT `scans` rather than `searches`.

For example, if we have spaces created with these queries:
```
CREATE TABLE t(i INT PRIMARY KEY, a INT);
CREATE TABLE s(i INT PRIMARY KEY, a INT);
```

Then these queries will throw an error:
```
SELECT * FROM t;
SELECT * FROM t WHERE a > 1;
SELECT * FROM t WHERE i + 1 = 5;
SELECT * FROM t, s;
SELECT * FROM t JOIN s;
```

And these will not:
```
SELECT * FROM t WHERE i > 1;
SELECT * FROM SEQSCAN t;
SELECT * FROM SEQSCAN t WHERE i + 1 = 5;
SELECT * FROM SEQSCAN t, SEQSCAN s;
SELECT * FROM SEQSCAN t JOIN SEQSCAN s;
```

Scanning can be allowed or disallowed by default. To do this, a new
session setting is introduced: `sql_seq_scan`. The default value for
setting is `true`, i.e. scanning is allowed. When set to `false`, the
scanning SELECTs will throw a `scanning is not allowed` error.

This patch removes code that was used to implement the SQL check
constraint as they are now replaced by BOX constraint. Also, the syntax
for enabling/disabling check constraints has been removed as BOX
constraints do not support this feature.

Follow-up #6986

NO_DOC=Already introduced.
NO_CHANGELOG=Already introduced.

This patch removes code that was used to implement the SQL foreign key
as they are now replaced by BOX foreign keys.

Follow-up #6986

NO_DOC=Refactoring.
NO_TEST=Refactoring
NO_CHANGELOG=Refactoring.