Arseniy Volynets authored 7 months ago and Dmitry Ivanov committed 3 months ago

- If you prepare and execute statement with params
in the projection and then unprepare the
statement, byte counter may show the wrong value
or even overflow.
- The problem is that when we compile sql
statement, we set parameter type to 'any'.
But when we execute the statement we set parameter
type to actual type. Then we use this type in
calculation of estimated of sql cache entry size.
This leads to different estimated sizes of cache
entry during prepare and during unprepare after
statement was executed
- Fix this by resetting type to 'any' after
executing the statement

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch

Arseniy Volynets authored 7 months ago and Dmitry Ivanov committed 3 months ago

BREAKING CHANGE!:
1. add session id argument to sql_prepare_ext
2. introduce sql_unprepare_ext function.
This function removes prepared stmt using
given session id.

In picodata SQL, we may prepare stmt in one
session and unprepare it in some
other session, which does not know in
what session the statement was prepared. Now
sql_prepare_ext returns not only statement id,
but also a session id. This way statement can
be unprepared from other session using
sql_unprepare_ext.

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch

Denis Smirnov authored 8 months ago and Dmitry Ivanov committed 3 months ago

BREAKING CHANGE!:
1. remove sql_prepare from the export list;
2. introduce sql_prepare_ext.

The sql_prepare symbol previously included the tarantool port as
an output parameter. However, this structure was inconvenient for
libraries using the C API, as they primarily required just the
statement ID. To address this issue, the sql_prepare symbol was
replaced with the sql_prepare_ext symbol.

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch

Denis Smirnov authored 8 months ago and Dmitry Ivanov committed 3 months ago

Previously, users with multiple connections to tarantool instance
couldn't share prepared statements across sessions. They had to
manually call prepare in each session before execution.

This commit automates this process for the exported version of
SQL prepared statement execution (sql_execute_prepared_ext symbol).
Original Lua execution keeps the old behavior for backward
compatibility.

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Denis Smirnov authored 8 months ago and Dmitry Ivanov committed 3 months ago

Previously, sql_prepare_and_execute and sql_execute_prepared
functions didn't follow a convention to keep output parameters
(the port to be exact) at the end of the argument list.

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Denis Smirnov authored 8 months ago and Dmitry Ivanov committed 3 months ago

BREAKING CHANGE!:
1. sql_bind_list_decode - removed
2. sql_execute_prepared_ext - new arguments
3. sql_prepare_and_execute_ext - exported

There were several reasons to refactor the API.
1. sql_bind_list_decode (decodes message pack parameters into
   internal C bind structure) is very difficult to use without
   memory leaks (as it allocates results on fiber()->gc).
2. sql_execute_prepared_ext missed vdbe step limit in parameters
   and used the default value.
3. Sometimes SQL queries don't fit into prepared statement cache
   and the user still wants to execute them via a slow pass with
   full compilations from the query text. That was the reason to
   export sql_prepare_and_execute_ext symbol.

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

Arseniy Volynets authored 1 year ago and Dmitry Ivanov committed 3 months ago

- Add a configurable non-negative
session parameter "sql_vdbe_max_steps"
-- max number of opcodes that Vdbe
is allowed to execute for sql query.

- Default value can be specified in box.cfg.
If not set via box.cfg, default value
is 45000. Value 0 means that no
checks for number of executed Vdbe
opcodes will be made.

- Add the third argument to box.execute
function, that allows to specify options
for query execution. The only option
supported: sql_vdbe_max_steps. Usage
example:

```
box.execute([[select * from t]], {}, {{sql_vdbe_max_steps = 1000}})
```

part of picodata/picodata/sbroad!461

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch

Denis Smirnov authored 2 years ago and Dmitry Ivanov committed 3 months ago

Actually there is no reason to throw an error and make a user
manually recreate prepared statement when it expires. A much more
user friendly way is to recreate it under hood when statement's
schema version differs from the box one.

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

After this patch, the list of auto-increment IDs will be reset before
executing the prepared statement.

Closes #6422

NO_DOC=bugfix

Needed for #6896

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Function sql_column_to_messagepack() has almost the same functionality
as sql_vdbe_mem_encode_tuple(). Due to this it is not necessary to
have sql_column_to_messagepack(), so it is removed in this commit.

Part of #5818

sql_value_type() and mem_mp_type() do the same thing: return messagePack
type corresponding to value stored in memory cell. However,
sql_value_type() operates on opaque API wrapper - sql_value*. To avoid
duplicating code let's invoke mem_mp_type() in sql_value_type().
At once, let's account that mp_type now can be not only _BIN, but also
_ARRAY and _MAP - this fact will be used when we introduce arrays and
maps in SQL.

Port_tuple is exactly the same as port_c, but is not able to store
raw MessagePack. In theory it sounds like port_tuple should be a
bit simpler and therefore faster, but in fact it is not.
Microbenchmarks didn't reveal any difference. So port_tuple is no
longer needed, all its functionality is covered by port_c.

Follow up #4641

Before this patch prepared statements didn't reset bound values after
its execution. As a result, if during next execution cycle not all
parameters were provided, cached values would appear. For instance:

prep = box.prepare('select :a, :b, :c')
prep:execute({{[':a'] = 1}, {[':b'] = 2}, {[':c'] = 3}}
-- [1, 2, 3]
prep:execute({{[':a'] = 1}, {[':b'] = 2}})
-- [1, 2, 3]

However, expected result for the last query should be [1, 2, NULL].
Let's fix it and always reset all binding values before next execution.

Closes #4825

The patch fixes a bug for the commit b0f588f6 where statistics on
box.execute was collected twice. This happened because
sql_prepare_and_execute called sql_execute under the hood, so there's
no need to do rmean_collect in both of them.

Follow-up #4756

Calling prepare and execute did not update corresponding request statistics
in the box.stat table. This happened because methods that collect stats were
never called where they should have been.

Closes #4756

This patch introduces support of prepared statements in IProto
protocol. To achieve this new IProto command is added - IPROTO_PREPARE
(key is 0x13). It is sent with one of two mandatory keys:
IPROTO_SQL_TEXT (0x40 and assumes string value) or IPROTO_STMT_ID (0x43
and assumes integer value). Depending on body it means to prepare or
unprepare SQL statement: IPROTO_SQL_TEXT implies prepare request,
meanwhile IPROTO_STMT_ID - unprepare.  Also to reply on PREPARE request a
few response keys are added: IPROTO_BIND_METADATA (0x33 and contains
parameters metadata of type map) and IPROTO_BIND_COUNT (0x34 and
corresponds to the count of parameters to be bound).

Part of #2592

This patch introduces local prepared statements. Support of prepared
statements in IProto protocol and netbox is added in the next patch.

Prepared statement is an opaque instance of SQL Virtual Machine. It can
be executed several times without necessity of query recompilation. To
achieve this one can use box.prepare(...) function. It takes string of
SQL query to be prepared; returns extended set of meta-information
including statement's ID, parameter's types and names, types and names
of columns of the resulting set, count of parameters to be bound.  Lua
object representing result of :prepare() invocation also features two
methods - :execute() and :unprepare(). They correspond to
box.execute(stmt.stmt_id) and box.unprepare(stmt.stmt_id), i.e.
automatically substitute string of prepared statement to be executed.
Statements are held in prepared statement cache - for details see
previous commit.  After schema changes all prepared statement located in
cache are considered to be expired - they must be re-prepared by
separate :prepare() call (or be invalidated with :unrepare()).

Two sessions can share one prepared statements. But in the current
implementation if statement is executed by one session, another is
not able to use it and will compile it from scratch and than execute.

SQL cache memory limit is regulated by box{sql_cache_size} which can be
set dynamically. However, it can be set to the value which is less than
the size of current free space in cache (since otherwise some statements
can disappear from cache).

Part of #2592

Result set serialization formats of DQL and DML queries are different:
the last one contains number of affected rows and optionally list of
autoincremented ids; the first one comprises all meta-information
including column names of resulting set and their types. What is more,
serialization format is going to be different for execute and prepare
requests. So let's introduce separate member to struct port_sql
responsible for serialization format to be used.

Note that C standard specifies that enums are integers, but it does not
specify the size. Hence, let's use simple uint8 - mentioned enum are
small enough to fit into it.

What is more, prepared statement finalization is required only for
PREPARE-AND-EXECUTE requests. So let's keep flag indicating required
finalization as well.

Needed for #2592

Let's follow unified naming rules for SQL high level API which
manipulates on statements objects. To be more precise, let's use
'sql_stmt_' prefix for interface functions operating on statement
handles.

sql_prepare() is going not only to compile statement, but also to save it
to the prepared statement cache. So we'd better rename sqlPrepare()
which is static wrapper around sql_prepare() and make it non-static.
Where it is possible let's use sql_stmt_compile() instead of sql_prepare().

Needed for #2592

- Removed saveSqlFlag as argument from sqlPrepare(). It was used to
  indicate that its caller is sql_prepare_v2() not sql_prepare().
  Since in previous commit we've left only one version of this function
  let's remove this flag at all.

- Removed struct db from list of sql_prepare() arguments. There's one
  global database handler and it can be obtained by sql_get() call.
  Hence, it makes no sense to pass around this argument.

Needed for #3292

There are two versions of the same function (sql_prepare()) which are
almost identical. Let's keep more relevant version sql_prepare_v2() but
rename it to sql_prepare() in order to avoid any mess.

Needed for #3292

Each column of result set features its name span (in full metadata
mode). For instance:

SELECT x + 1 AS add FROM ...;

In this case real name (span) of resulting set column is "x + 1"
meanwhile "add" is its alias. This patch extends metadata with
member which corresponds to column's original expression.
It is worth mentioning that in most cases span coincides with name, so
to avoid overhead and sending the same string twice, we follow the rule
that if span is encoded as MP_NIL then its value is the same as name.
Also note that span is always presented in full metadata mode.

Closes #4407

@TarantoolBot document
Title: extended SQL metadata

Before this patch metadata for SQL DQL contained only two fields:
name and type of each column of result set. Now it may contain
following properties:
 - collation (in case type of resulting set column is string and
              collation is different from default "none");
   is encoded with IPROTO_FIELD_COLL (0x2) key in IPROTO_METADATA map;
   in msgpack is encoded as string and held with MP_STR type;
 - is_nullable (in case column of result set corresponds to space's
                field; for expressions like x+1 for the sake of
                simplicity nullability is omitted);
   is encoded with IPROTO_FIELD_IS_NULLABLE key (0x3) in IPROTO_METADATA;
   in msgpack is encoded as boolean and held with MP_BOOL type;
   note that absence of this field implies that nullability is unknown;
 - is_autoincrement (is set only for autoincrement column in result
                     set);
   is encoded with IPROTO_FIELD_IS_AUNTOINCREMENT (0x4) key in IPROTO_METADATA;
   in msgpack is encoded as boolean and held with MP_BOOL type;
 - span (is always set in full metadata mode; it is an original
   expression forming result set column. For instance:
   SELECT a + 1 AS x; -- x is a name, meanwhile a + 1 is a span);
   is encoded with IPROTO_FIELD_SPAN (0x5) key in IPROTO_METADATA map;
   in msgpack is encoded as string and held with MP_STR type OR
   as NIL with MP_NIL type. The latter case indicates that span
   coincides with name. This simple optimization allows to avoid
   sending the same string twice.

This extended metadata is send only when PRAGMA full_metadata is
enabled. Otherwise, only basic (name and type) metadata is processed.

If result set contains column which features attached sequence
(AUTOINCREMENT in terms of SQL) then meta-information will contain
corresponding field ('is_autoicrement' : boolean) in response.

Part of #4407

If member of result set is (solely) column identifier, then metadata
will contain its corresponding field nullability as boolean property.
Note that indicating nullability for other expressions (like x + 1)
may make sense but it requires derived nullability calculation which in
turn seems to be overkill (at least in scope of current patch).

Part of #4407

If resulting set column is of STRING type and features collation (no
matter explicit or implicit) different from "none", then metadata will
contain its name.

This patch also introduces new pragma: full_metadata. By default it is
not set. If it is turned on, then optional metadata (like collation) is
pushed to Lua stack. Note that via IProto protocol always full metadata
is send, but its decoding depends on session SQL settings.

Part of #4407

Closes #2200
Closes #4113
Closes #2233

@TarantoolBot document
Title: The box.internal.sql_function_create is forbidden
Legacy mechanism box.internal.sql_function_create to make some
Lua function available in SQL is forbidden now.

To make some function available in SQL you need to use
box.schema.func.create() mechanism: you need to specify
1) function language and language-specific options(e.g. you
   are able to define a persistent Lua function)
2) whether this function is_deterministic or not: deterministic
   functions allows to generate more efficient SQL VDBE bytecode
   so you better specify it when it is true
3) the function returns type: a Tarantool type string describes
   a type of value returned by function
4) param_list - a table of Tarantool's types strings desccribe
   function argument types
5) exports - a table of Tarantool's frontends where this function
   should be available ('LUA' by default). You need to specify
   {'LUA', 'SQL'} to make function available both in SQL requests
   and visible in box.func folder

Example:
-- Case1: C function
-- function1.so has int divide() symbol
box.schema.func.create("function1.divide", {language = 'C',
			returns = 'number',
			param_list = {'number', 'number'},
			is_deterministic = true,
			exports = {'LUA', 'SQL'}})
box.execute('SELECT "function1.divide"(6, 3)')
- metadata:
  - name: '"function1.divide"(6, 3)'
    type: number
  rows:
  - [2]
box.schema.func.drop("function1.divide")

-- Case2: Persistent Lua function
box.schema.func.create("SUMMARIZE", {language = 'LUA',
			returns = 'number',
			body = 'function (a, b) return a + b end',
			param_list = {'number', 'number'},
			is_deterministic = true,
			exports = {'LUA', 'SQL'}})
box.execute('SELECT summarize(1, 2)')
- metadata:
  - name: summarize(1, 2)
    type: number
  rows:
  - [3]
box.schema.func.drop("summarize")

Moreover there is a special predefined Lua function LUA that
allows to evaluate a custom Lua expressions in SQL.
You need to pass a string in form "return ...." to LUA function
that returns more than one value of any type.

Example:
box.execute('SELECT lua(\'return 1 + 1\')')
- metadata:
  - name: lua('return 1 + 1')
    type: any
  rows:
  - [2]
box.execute('SELECT lua(\'return box.cfg.memtx_memory\')')
- metadata:
  - name: lua('return box.cfg.memtx_memory')
    type: any
  rows:
  - [268435456]

This patch allows to operate on integer values in range [2^63, 2^64 - 1]
It means that:
 - One can use literals from 9223372036854775808 to 18446744073709551615
 - One can pass values from mentioned range to bindings
 - One can insert and select values from mentioned range

Support of built-in functions and operators has been introduced in
previous patches.

Closes #3810
Part of #4015

As it was stated in the previous commit message, we are going to support
operations on unsigned values. Since unsigned and signed integers have
different memory representations, to provide correct results of
arithmetic operations we should be able to tell whether value is signed
or not.
This patch introduces new type of value placed in VDBE memory cell -
MEM_UInt. This flag means that value is integer and greater than zero,
hence can be fitted in range [0, 2^64 - 1]. Such approach would make
further replacing MEM_* flags with MP_ format types quite easy: during
decoding and encoding msgpack we assume that negative integers have
MP_INT type and positive - MP_UINT. We also add and refactor several
auxiliary helpers to operate on integers. Note that current changes
don't add ability to operate on unsigned integers - it is still
unavailable.

Needed for #3810
Needed for #4015

Re-factor box_lua_call and box_lua_eval so that they don't take
call_request. This approach is more scalable: in case of a
functional index, the user expects to see a tuple with field
names so we should be able to pass not only raw msgpack, but
also a tuple to a Lua call so we need an universal way to pass
arguments to _call methods.

To pass a tuple msgpack introduced a new port_msgpack: the port
class with dump_lua method.
A new method get_msgpack returns a content of a port as a
msgpack data. The lifecycle of the returned value is
implementation-specific: it may either be returned directly from
the port, in which case the data will stay alive as long as the
port is alive, or it may be allocated on the fiber()->gc, in
which case the caller is responsible for cleaning up.

Needed for #4182, #1260

Removing this error/status code is part of getting rid of the SQL
error system.

After this patch, all errors in VDBE will be set using diag_set().

Closes #4074

This patch introduces basic facilities to operate on boolean type:
boolean literals "true" and "false" where true > false; alias to null -
unknown; column type "BOOLEAN" and shortcut "BOOL"; opportunity to
insert and select boolean values from table; OR and AND predicates
accept boolean arguments; CAST operation involving boolean type;
comparison between boolean values (including VDBE sorter routines).

Part of #3648

This patch provides straightforward refactoring replacing enum sql_type
with enum mp_type. Note that we use msgpack format instead of field_type
since it is more suitable when dealing with NULLs.

This patch changes the name of the function used in the port_sql
methods and the name of the member of the result of the execution.

Follow up #3505

This patch creates the method dump_lua() for port_sql and uses it
in the new function box.execute(). The function box.execute()
replaces box.sql.execute() in the next patch.

Part of #3505

Exported sql_bind structure, sql_bind_decode, sql_bind_column
and sql_bind routines in separate module bind.h. We need SQL
bindings in further pathes with check constraints. Also, bind
encapsulated in execute.c prevent us from implementation of a Lua
part of forthcoming box.execute().

Needed for #3691, #3505

This patch creates port_sql implementation for the port. This will
allow us to dump sql responses to obuf or to Lua. Also this patch
defines methods dump_msgpack() and destroy() of port_sql.

Part of #3505

Currently, functions sql_execute() and sql_prepare_and_execute()
set the ER_SQL_EXECUTE code for all errors that occur during the
execution of a SQL command. This is considered incorrect because
some of these errors may have their own error code. After this
patch, only errors without an error code will have the error code
ER_SQL_EXECUTE.

Part of #3505