Arseniy Volynets authored 7 months ago and Dmitry Ivanov committed 3 months ago

- If you prepare and execute statement with params
in the projection and then unprepare the
statement, byte counter may show the wrong value
or even overflow.
- The problem is that when we compile sql
statement, we set parameter type to 'any'.
But when we execute the statement we set parameter
type to actual type. Then we use this type in
calculation of estimated of sql cache entry size.
This leads to different estimated sizes of cache
entry during prepare and during unprepare after
statement was executed
- Fix this by resetting type to 'any' after
executing the statement

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch

This patch introduces new keyword SEQSCAN and new restrictions on
SELECTs. These restrictions are disabled by default.

Closes #7747

@TarantoolBot document
Title: SEQSCAN

Now scanning SELECT will not run and will throw an error if the new
SEQSCAN keyword is not used for scanned spaces. This change only affects
SELECT and does not affect UPDATE and DELETE. A SELECT is recognized as
a scanning SELECT if `EXPLAIN QUERY PLAN SELECT ...` indicates that the
SELECT `scans` rather than `searches`.

For example, if we have spaces created with these queries:
```
CREATE TABLE t(i INT PRIMARY KEY, a INT);
CREATE TABLE s(i INT PRIMARY KEY, a INT);
```

Then these queries will throw an error:
```
SELECT * FROM t;
SELECT * FROM t WHERE a > 1;
SELECT * FROM t WHERE i + 1 = 5;
SELECT * FROM t, s;
SELECT * FROM t JOIN s;
```

And these will not:
```
SELECT * FROM t WHERE i > 1;
SELECT * FROM SEQSCAN t;
SELECT * FROM SEQSCAN t WHERE i + 1 = 5;
SELECT * FROM SEQSCAN t, SEQSCAN s;
SELECT * FROM SEQSCAN t JOIN SEQSCAN s;
```

Scanning can be allowed or disallowed by default. To do this, a new
session setting is introduced: `sql_seq_scan`. The default value for
setting is `true`, i.e. scanning is allowed. When set to `false`, the
scanning SELECTs will throw a `scanning is not allowed` error.

After this patch, variable names will have to follow the rules defined
for identifiers in SQL. Essentially, this means that a digit can no
longer be used as the first character of a bind variable name.

Part of #4763

NO_DOC=Will be added later.
NO_CHANGELOG=Will be added later.

Prior to this, the patch functions MIN(), MAX(), LEAST() and GREATEST()
showed SCALAR as result types in metadata. However, in reality, the type
of the result could be any scalar type. After this patch, the type of
the result will always be the same as the type in the metadata. Also,
for MIN() and MAX() functions, the type of the result will be the same
as the type of the argument. For the LEAST() and GREATEST() functions,
the result type will be the same as the type of the arguments if all
arguments are of the same type, or it will be NUMBER if all arguments
are of numeric types, or it will be SCALAR.

Part of #6105

ER_TUPLE_FOUND message shows only space and index, let's also show old
tuple and new tuple.

This commit changes error message in code and in tests. Test sql/checks
and sql-tap/aler remain the same due to problems in showing their old
and new tuples in error message.

Closes #5567

Name resulting columns generated by an expression or <VALUES>
construction by the "COLUMN_N" pattern.

Closes #3962

@TarantoolBot document
Title: Column naming in SQL

Now, every auto generated column is named by the "COLUMN_N"
pattern, where N is the number of generated column in a query
(starting  from 1). Auto generated column is a column in a query
result generated by an expression or a column from <VALUES>
construction.

Examples:
```
box.execute("VALUES(1, 2, 3);")
---
- metadata:
  - name: COLUMN_1
    type: integer
  - name: COLUMN_2
    type: integer
  - name: COLUMN_3
    type: integer
  rows:
  - [1, 2, 3]
...
box.execute("SELECT * FROM (VALUES (1+1, 1+1));")
---
- metadata:
  - name: COLUMN_1
    type: integer
  - name: COLUMN_2
    type: integer
  rows:
  - [2, 2]
...
box.execute("SELECT 1+1, 1+1;")
---
- metadata:
  - name: COLUMN_1
    type: integer
  - name: COLUMN_2
    type: integer
  rows:
  - [2, 2]
...
```

Here, the expression "mycol + 1" generates a new column, so that
it is the first auto generated resulting column will be named as
"COLUMN_1".
```
tarantool> CREATE TABLE test (mycol INT PRIMARY KEY);
---
- row_count: 1
...

tarantool> SELECT mycol, mycol + 1 FROM test;
---
- metadata:
  - name: MYCOL
    type: integer
  - name: COLUMN_1
    type: integer
  rows: []
...
```
Note that you can use generated names already within the query,
e.g. in <ORDER BY> clause.
```
tarantool> SELECT mycol, mycol + 1 FROM test ORDER BY column_1;
---
- metadata:
  - name: MYCOL
    type: integer
  - name: COLUMN_1
    type: integer
  rows: []
...
```

It should also be noted that if you use column names similar to
the "COLUMN_N" pattern, you can get the same names as a result:

```
tarantool> CREATE TABLE test (column_1 SCALAR PRIMARY KEY);
---
- row_count: 1
...

tarantool> INSERT INTO test VALUES(1);
---
- row_count: 1
...

tarantool> SELECT column_1, column_1 + 1 FROM test;
---
- metadata:
  - name: COLUMN_1
    type: scalar
  - name: COLUMN_1
    type: scalar
  rows:
  - [1, 2]
...
```

The patch fixes a bug for the commit b0f588f6 where statistics on
box.execute was collected twice. This happened because
sql_prepare_and_execute called sql_execute under the hood, so there's
no need to do rmean_collect in both of them.

Follow-up #4756

Calling prepare and execute did not update corresponding request statistics
in the box.stat table. This happened because methods that collect stats were
never called where they should have been.

Closes #4756

This patch introduces support of prepared statements in IProto
protocol. To achieve this new IProto command is added - IPROTO_PREPARE
(key is 0x13). It is sent with one of two mandatory keys:
IPROTO_SQL_TEXT (0x40 and assumes string value) or IPROTO_STMT_ID (0x43
and assumes integer value). Depending on body it means to prepare or
unprepare SQL statement: IPROTO_SQL_TEXT implies prepare request,
meanwhile IPROTO_STMT_ID - unprepare.  Also to reply on PREPARE request a
few response keys are added: IPROTO_BIND_METADATA (0x33 and contains
parameters metadata of type map) and IPROTO_BIND_COUNT (0x34 and
corresponds to the count of parameters to be bound).

Part of #2592

This patch removes control pragmas. They are not needed now, after
the introduction of the _session_settings system space.

Closes #4511

Pragma "count_changes" forces the INSERT, REPLACE, DELETE, and
UPDATE statements to return the number of changed rows as a
result set. This is not necessary, as these statements return the
number of changed rows as metadata.

Part of #4511

When it comes for huge queries, it may turn out to be useful to see
exact position of occurred error. Hence, let's now display line and
position within line near which syntax error takes place. Note that it
can be done only during parsing process (since AST can be analysed only
after its construction is completed), so most of semantic errors still
don't contain position. A few errors have been reworked to match new
formatting patterns.

First iteration of this patch is implemented by @romanhabibov

Closes #2611

Type was displayed in error messages, was returned in
meta headers, and a type string is a result of
typeof() SQL function.

Typeof() always returns lower case type string; meta
contained upper case type; error messages contained
both.

It was necessary to choose one case for everything,
and the lower one was chosen. It allows not to break
typeof() function which actually might be used by
someone.

This patch does two things: renames existing scalar min/max
functions and reserves names for them in NoSQL cache.

Moreover it is an important step to get rid of function's name
overloading required for replace FuncDef cache with Tarantool's
function cache.

Closes #4405
Needed for #2200, #4113, #2233

@TarantoolBot document
Title: Scalar functions MIN/MAX are renamed to LEAST/GREATEST

The MIN/MAX functions are typically used only as aggregate
functions in other RDBMS(MSSQL, Postgress, MySQL, Oracle) while
Tarantool's SQLite legacy code use them also in meaning
GREATEST/LEAST scalar function. Now it fixed.

It was decided that for all integer literals we would return "INTEGER"
type, not "UNSIGNED". Accidentally, after substitution of unsigned
binding value type was set to "UNSIGNED". Let's fix that and set
"INTEGER" type.

Current patch introduces new type available in SQL:
 - VARBINARY now is reserved keyword;
 - Allow to specify VARBINARY column and CAST type;
 - All literals which start from 'x' are assumed to be of this type;
 - There's no available implicit or explicit conversions between
   VARBINARY and other types;
 - Under the hood all values of VARBINARY type are stored as MP_BIN
   msgpack format type.

Closes #4206

Before this patch it was allowed to specify REAL/FLOAT/DOUBLE types
which matched with NUMBER NoSQL type in space format. However, NUMBER is
different from standard floating point types, since it is able to hold
integers in range [-2^63; 2^64-1] alongside with double precision
floating point values. Hence, to not confuse users it has been decided
to remove support of REAL/FLOAT/DOUBLE types from SQL grammar and use
instead original NUMBER type naming.

Currently, if an INSERT is executed inside SQL trigger and it
results in generated autoincrement identifiers, ones will be
displayed as a result of the statement. This is wrong, since we
are not able to divide IDs obtained into those that belong to the
table mentioned in the statement and those that do not belong to
this table. This has been fixed by adding a new argument to
OP_IdxInsert. In case the argument is not 0, recently generated
identifier is retrieved and saved into the list, which is held in
VDBE itself. Note that from now we don't save autoincremented
value to VDBE right in sequence_next() - this operation is moved
to OP_IdxInsert. So that, VDBE can be removed from struct txn.

For example:
box.execute('CREATE TABLE t1 (i INT PRIMARY KEY AUTOINCREMENT);')
box.execute('CREATE TABLE t2 (i INT PRIMARY KEY AUTOINCREMENT);')
box.execute('CREATE TRIGGER r AFTER INSERT ON t1 FOR EACH ROW '..
            'BEGIN INSERT INTO t2 VALUES (null); END')
box.execute('INSERT INTO t2 VALUES (100);')
box.execute('INSERT INTO t1 VALUES (NULL), (NULL), (NULL);')

Result should be:
---
- autoincrement_ids:
  - 1
  - 2
  - 3
  row_count: 3
...

Closes #4188

This patch allows to operate on integer values in range [2^63, 2^64 - 1]
It means that:
 - One can use literals from 9223372036854775808 to 18446744073709551615
 - One can pass values from mentioned range to bindings
 - One can insert and select values from mentioned range

Support of built-in functions and operators has been introduced in
previous patches.

Closes #3810
Part of #4015

Currently, in OP_Halt, you can get a SQL error other than
SQL_TARANTOOL_ERROR, for example, the SQL_CONSTRAINT error. After
this patch, all errors going through OP_Halt will have SQL error
code SQL_TARANTOOL_ERROR and have diag set.

Part of #4074

The error codes SQL_TARANTOOL_DELETE_FAIL,
SQL_TARANTOOL_ITERATOR_FAIL and SQL_TARANTOOL_INSERT_FAIL have
practically no functions, but their use can lead to incorrect
behavior. This patch replaces them with SQL_TARANTOOL_ERROR. This
will simplify the work with errors.

Before this patch, it was possible to create a trigger without FOR EACH
ROW clause, for example:

CREATE TRIGGER trg AFTER DELETE ON tbl BEGIN ; END;

In ANSI SQL, if trigger-timing-clause is not specified, FOR EACH
STATEMENT is used. Tarantool, however, did not support FOR EACH
STATEMENT and assumed FOR EACH ROW. This could break future
applications, once FOR EACH STATEMENT is added.

Thus, make FOR EACH ROW clause mandatory. Update tests.

No docs ticket since there is no docs for this feature yet :/ -
will document the fixed behaviour right away.

This patch changes the name of the function used in the port_sql
methods and the name of the member of the result of the execution.

Follow up #3505

Proper type of function's returning value is set during names resolution
(i.e. promotion from struct FuncDef to struct Expr, see
resolveExprStep()). Accidentally, it was set again during byte-code
generation for VDBE. What is more, in some cases it was set to a wrong
type. For instance, built-in function randomblob() returns value to be
encoded as MP_BIN, so its returning type is SCALAR. However, it was
reset to INTEGER (as its first argument). This patch simply removes this
second type promotion.

Column meta-information which is sent alongside execution result via
IProto protocol, contains string representation of column type.
In some cases, name of type is different from actual type of field. For
instance, if column has type SCALAR, string representation in
meta-information will be "BLOB"; for NUMBER NoSQL type - it will be
"NUMERIC"; for STRING - "TEXT". Instead of this mess, let's always return
exact name of underlying NoSQL type.

This patch defines parameters binding for SQL statements executed
through box.execute().

Closes #3401

This patch replaces box.sql.execute() by box.execute() in tests
and removes box.sql.execute() from code.

Closes #3505

Currently, functions sql_execute() and sql_prepare_and_execute()
set the ER_SQL_EXECUTE code for all errors that occur during the
execution of a SQL command. This is considered incorrect because
some of these errors may have their own error code. After this
patch, only errors without an error code will have the error code
ER_SQL_EXECUTE.

Part of #3505

Currently, if the count_changes pragma is enabled, the INSERT,
REPLACE and UPDATE statements will return the number of changes at
execution time. This patch sets an INTEGER type for this result.

Follow up #3832
Needed from #3505

This patch completely replaces sqlErrorMsg() with diag_set() and
removes sqlErrorMsg().

Closes #3965
Closes #3036

This patch reworks some of SQL semantic errors.

Part of #3965

This patch reworks SQL syntax errors. After this patch, these
error will be set as Tarantool errors.

Part of #3965

Currently, EXPLAIN and PRAGMA do not set the column types for the
result. This is incorrect, since any returned row must have a
column type. This patch defines the types for these columns.

Closes #3832

This patch reworks SQL errors of types "no such object" and
"object exists". After this patch, these error will be set as
Tarantool errors.

Part of #3965

Replace all usage of sqlite3_, sqlite, SQLite prefixes with simple sql_
All other occurrences of SQLite are substituted with SQL word.
SQL test suit is purified as well.

Also this patch resolves issue connected with wrong query plans during
select on spaces created from Lua: instead of index search in most cases
table scan was used. It appeared due to the fact that index was checked
on affinity compatibility with space format. So, if space is created
without affinity in format, indexes won't be used.
However, now all checks are related to field types, and as a result
query optimizer is able to choose correct index.

Closes #3886
Part of #3698

VDBE returns an error if LIMIT or OFFSET expressions are casted to
the negative integer value.
If expression in the LIMIT clause can't be converted into integer
without data loss the VDBE instead of SQLITE_MISMATCH returns
SQL_TARANTOOL_ERROR with message "Only positive integers are allowed
in the LIMIT clause". The same for OFFSET clause.

Closes #3467

Too many autogenerated ids leads to SEGFAULT. This problem
appeared because region was cleaned twice: once in
sqlite3VdbeHalt() and once in sqlite3VdbeDelete() which was
executed during sqlite3_finalize(). Autogenerated ids that were
saved there, were fetched after sqlite3VdbeHalt() and before
sqlite3_finalize(). In this patch region cleaning in
sqlite3VdbeHalt() has been removed.

Follow up #2618
Follow up #3199

Before this patch MSGPACK received using SELECT statement through
net.box was unpacked. Fixed in this patch.

In our SQL implementation REPLACE acts as DELETE + INSERT, so we should
account it as two row changes.

Needed for #2181