Due to inconsistency of Tarantool type casting while using strict
data types as "double" or "unsigned" it is needed
to use "number" data type in a whole bunch of cases.
However "number" may contain "decimal" that will be serialized into
string by JSON builtin module.

This commit adds "encode_decimal_as_number" parameter to json.cfg{}.
That forces to encode `decimal` as JSON number to force type
consistency in JSON output.
Use with catious - most of JSON parsers assume that number is restricted
to float64.

NO_DOC=we do not host doc

Denis Smirnov authored 2 years ago and Дмитрий Кольцов committed 1 year ago

Previously,

select "t1"."a" from (select "a" from "t") as "t1";

returned a result column name `t1` instead of `t1.a` because of
incorrect work of a dequoting function. The reason was that
previously sqlDequote() function finished its work when found the
first closing quote.

Old logic worked for simple selects where the column name doesn't
contain an explicit scan name ("a" -> a).
But for the sub-queries results sqlDequote() finished its work right
after the scan name ("t1"."a" -> t1). Now the function continues its
deqouting till it gets the null terminator at the end of the string.

Closes #7063

NO_DOC=don't change any public API, only a bug fix

Co-authored-by: Mergen Imeev <imeevma@gmail.com>

Denis Smirnov authored 2 years ago and Дмитрий Кольцов committed 1 year ago

Problem description.

When we prepare a statement with parameters in the result columns
(for example box.prepare('select ?')) Tarantool has no information
about the type of the output column and set it to default boolean.
Then, on the execution phase, the type would be recalculated during
the parameter binding.

Tarantool expects that there is no way for parameter to appear in the
result tuple other than exactly be mentioned in the final projection.
But it is incorrect - we can easily propagate parameter from the inner
part of the join. For example

box.prepare([[select COLUMN_1 from t1 join (values (?)) as t2 on true]])

In this case column COLUMN_1 in the final projection is not a
parameter, but a "reference" to it and its type depends on the
parameter from the inner part of the join. But as Tarantool
recalculates only binded parameters in the result projection,
it doesn't change the default boolean metadata type of the COLUMN_1
and the query fails on comparison with the actual type of the tuple.

Solution.
As we don't want to patch Vdbe to make COLUMN_1 refer inner parameter,
it was decided to make a simple workaround: change the default
column type from BOOLEAN to ANY for parameters. It fixes the
comparison with the actual tuple type (we do not fail), but in some
cases get ANY column in the results where we would like to have
explicitly defined type. Also NULL parameters would also have ANY
type, though Tarantool prefers to have BOOLEAN in this case.

Closes https://github.com/tarantool/tarantool/issues/7283

NO_DOC=bug fix

godzie44 authored 2 years ago and Дмитрий Кольцов committed 1 year ago

sql: add sql_execute_prepared_ext function, same as sql_execute_prepared but without `region` parameter
closes #2

NO_DOC=minor
NO_TEST=minor

godzie44 authored 2 years ago and Дмитрий Кольцов committed 1 year ago

- add box_tuple_data_offset function (return offset of the messagePack encoded data from the beginning of the tuple)
- add more export functions

closes #1

NO_DOC=build
NO_TEST=build

Also, remove unreleased/ entries.

NO_DOC=changelog
NO_TEST=changelog
NO_CHANGELOG=changelog

Two new categories 'testing' and 'tools' were introduced, clangelogs
updated to match the new criteria.

NO_DOC=changelog
NO_TEST=changelog
NO_CHANGELOG=changelog

NO_DOC=changelog
NO_TEST=changelog
NO_CHANGELOG=changelog

The essence is fix of a http client bug, libcurl bump is mere a method.

NO_DOC=changelog
NO_TEST=changelog
NO_CHANGELOG=changelog

We need to call `tx_accept_msg` in `tx_process_override` before we pass
message to the override handler. Unfortunately if handler response with
IPROTO_HANDLER_FALLBACK we call the builtin handler for message that
calls `tx_accept_msg` again which is not expected. Some actions of
this function are idempotent and some are not.

Let's make the function NOP if it called once again.

Closes #9345

NO_DOC=bugfix

(cherry picked from commit 21112b06)

Without checking the return value of lua_pcall()` in
`lua_field_inspect_ucdata()`, the error message itself is returned as a
serialized result. The result status of `lua_pcall()` is not ignored
now.

NO_DOC=bugfix

Closes #9396

(cherry picked from commit 98474f70)

Current split-brain detector implementation raises an error each time a
CONFIRM or ROLLBACK entry is received from the previous synchronous
transaction queue owner. It is assumed that the new queue owner must
have witnessed all the previous CONFIRMS. Besides, according to Raft,
ROLLBACK should never happen.

Actually there is a case when a CONFIRM from an old term is legal: it's
possible that during leader transition old leader writes a CONFIRM for
the same transaction that is confirmed by the new leader's PROMOTE. If
PROMOTE and CONFIRM lsns match there is nothing bad about such
situation.

Symmetrically, when an old leader issues a ROLLBACK with the lsn right
after the new leader's PROMOTE lsn, it is not a split-brain.

Allow such cases by tracking the last confirmed lsn for each synchronous
transaction queue owner and silently nopifying CONFIRMs with an lsn less
than the one recorded and ROLLBACKs with lsn greater than that.

Closes #9138

NO_DOC=bugfix

(cherry picked from commit ffa6ac15)

The fix is simple: look up the function in `box.func` by name and, if
found, execute its `call` method. The only tricky part is to avoid the
lookup before `box.cfg` is called because `box.func` is unavailable at
the time. We achieve that by checking `box.ctl.is_recovery_finished`.

Closes #9131

NO_DOC=bug fix

(cherry picked from commit e92a8e7b)

On Tarantool shutdown we destroy all the fibers in some sequence. We
don't require that all the fibers are finished before shutdown. So it
may turn out that we first destroy some alive fiber and then destroy
another alive fiber which joins the first one. Currently we have
use-after-free issue in this case because clearing `link` field of
the second fiber changes `wake` field of the first fiber.

Close #9406

NO_DOC=bugfix

(cherry picked from commit 2f7ec948)

Graceful shutdown is done in a special fiber which is started for
example on SIGTERM. So it can run concurrently with fiber executing
Tarantool init script. On init fiber exit we break event loop to pass
control back to the Tarantool initialization code. But we fail to run
event loop a bit more to finish graceful shutdown.

The test is a bit contrived. A more real world case is when Tarantool is
termintated during lingering box.cfg().

Close #9411

NO_DOC=bugfix

(cherry picked from commit 786eb2ac)

According to ANSI, EXISTS is a predicate that tests a given subquery and
returns true if it returns more than 0 rows, false otherwise. However,
after 2a720d11, EXISTS worked correctly only if there were exactly 0
or 1 rows, and in all other cases it gave an error. This patch makes
EXITS work properly.

Closes #8676

NO_DOC=bugfix

(cherry picked from commit a5e498d1)

* Mark CONV as non-weak, to prevent elimination of its side-effect.
* Fix ABC FOLD rule with constants.
* test: add test for conversions folding
* Add NaN check to IR_NEWREF.
* LJ_GC64: Fix lua_concat().
* test: introduce asserts assert_str{_not}_equal
* ci: enable codespell
* cmake: introduce target with codespell
* codehealth: fix typos
* tools: add cli flag to run profile dump parsers
* profilers: purge generation mechanism
* memprof: refactor symbol resolution
* sysprof: fix crash during FFUNC stream
* Fix last commit.
* Print errors from __gc finalizers instead of rethrowing them.
* x86/x64: Fix math.ceil(-0.9) result sign.
* test: fix flaky fix-jit-dump-ir-conv.test.lua
* IR_MIN/IR_MAX is non-commutative due to underlying FPU ops.
* Fix jit.dump() output for IR_CONV.
* Fix FOLD rule for x-0.
* FFI: Fix pragma push stack limit check and throw on overflow.
* Prevent compile of __concat with tailcall to fast function.
* Fix base register coalescing in side trace.
* Fix register mask for stack check in head of side trace.
* x64: Properly fix __call metamethod return dispatch.

Closes #8594
Closes #8767
Closes #9339
Part of #9145

NO_DOC=LuaJIT submodule bump
NO_TEST=LuaJIT submodule bump

* Revert "Update cur_L on exceptional path (arm)"
* Revert "arm64: fix cur_L restoration on error throw"
* Revert "Update cur_L on exceptional path"
* Revert "Fix cur_L tracking on exceptional path"
* Restore cur_L for specific Lua/C API use case.
* Fix Clang build.

Closes #6323
Part of #9145

NO_DOC=LuaJIT submodule bump
NO_TEST=LuaJIT submodule bump

The issue is fixed in the `small' submodule by the following commit:

* matras: fix matras_view::block_count overflow

Closes #3594

NO_TEST=The test requires more than 64 GB of RAM.

@TarantoolBot document
Title: Document maximum number of tuples in hash index
Product: Tarantool
Root document: https://www.tarantool.io/en/doc/latest/book/box/limitations/

Number of tuples in hash index: 2147483648

(cherry picked from commit 7eaa2d23)

The bug was fixed in the small library:
 - slab: fix NULL ptr deref in assertion in slab_get
   https://github.com/tarantool/small/commit/ef77efacd452cb90caea2caf22d266f791c95ec3
 - slab: fix uint32_t overflow in slab_capacity
   https://github.com/tarantool/small/commit/77203600a7c645d97bce56f901eec25de0b29d6e

The small library submodule was updated in commit ebafd684
("small: bump version").

Closes #9218

NO_DOC=changelog
NO_TEST=changelog

(cherry picked from commit 1fc6b4a1)

We already use this info in one of the test and going to use it more.

Part of #7327

@TarantoolBot document
Title: new tarantool.build.asan flag

It is `true` if `ENABLE_ASAN` build option is set and `false` otherwise.

(cherry picked from commit 23012356)

Configuring log modules work differently with log.cfg and box.cfg:
box.cfg{log_modules=...} overwrites the current config completely while
log.cfg{modules=...} overwrites the currently config only for the
specified modules. Let's fix this inconsistency by making log.cfg behave
exactly as box.cfg.

Closes #7962

NO_DOC=bug fix

(cherry picked from commit c13e59a5)

We install a signal handler that prints the stack trace on SIGSEGV,
SIGBUS, SIGILL, SIGFPE. The signal handler uses the current stack.
This works fine for most issues, but not for stack overflow, because
the latter makes the current stack unusable, leading to a crash in
the signal handler. Let's install an alternative signal stack in each
thread so that we can print the stack trace on stack overflow.

Note that we skip this for ASAN because it installs its own signal
stack. (Installing a custom stack would result in a crash.)

Closes #9222

NO_DOC=bug fix

(cherry picked from commit cb8e903b)

The motivation is to reduce time slip on Tarantool startup before
running init scripts. Internal ev time is set in fiber_init/ev_default_loop
and is not get updated until starting event loop. This causes
timeouts slip up to 0.3 in debug ASAN build in init script (see #9261).

Let's run event loop right at the beginning of the run_script_f before
executing any script. This way besides updating internal ev time we make
an explicit place of starting script event loop. Currently it is started
lazily when config script yields.

This will fix CI for PR https://github.com/tarantool/tarantool-ee/pull/572
for debug ASAN workflow.

We can also remove start_loop condition. It does not make sense now. It
was added in the commit 3a851430 ("Fix tarantool -e "os.exit()"
hang") but since then we start to stop event loop after handling
os.exit().

Also this fixes #9266. The issue is we don't have an event loop to run
on shutdown triggers if -e command line expression add such a trigger
and then call os.exit().

Follow-up #7327
Closes #9266

NO_DOC=bugfix

(cherry picked from commit 1fcfb8c2)

During building an index in background, some transaction can perform a dml
request that affects space size (e.g. a replace), but the size will remain
the same, because bsize is moved from the old space to the new space in
memtx_space_prepare_alter() prior to space_execute_dml(). Fix this issue by
calling space_finish_alter() in alter_space_do().
In fact, this patch partially reverts commit 9ec3b1a4 ("alter: zap
space_vtab::commit_alter").

NO_DOC=bugfix

Closes #9247

(cherry picked from commit 54a42186)

Before this patch, if an index was created due to a column's UNIQUE
constraint or a column's PRIMARY KEY constraint before adding a
collation, and if the column's fieldno was not equal to the index's
position in space->index, the collation would not be assigned to the
index.

Also, this patch fixes an assertion in debug build for the case when an
index with more that one field was created before a collation was added.

Closes #9229

NO_DOC=bugfix

(cherry picked from commit 65608d87)

In order to preserve transaction boundaries over replication, Tarantool
writes a global NOP row after the last transaction row, if this row
happens to be local. This is done to make sure that the is_commit flag,
which is set only in the last transaction row, reaches the replica. This
wouldn't happen if the last row was local.

This workaround works fine for transactions completely authored by one
instance: when both global and local rows come from operations of a
single master.

However, it's possible to append local rows to a remote master's
transaction on a replica. For example, one can use on_replace triggers
to write to replica's local space on each new transaction coming from
master.

In this case essentially a global NOP entry is added at the end of a
remote master's transaction. This leads to several problems.

First of all, this bumps replica's LSN, which is counter-intuitive,
given that the replica might even be read-only. Besides, in a star
topology this leads to master being unable to connect to the replica
later on due to their vclocks becoming incompatible.

Secondly, even if replication channel between master and replica is
bidirectional, it creates a new row which should be replicated from
replica to master, but at the same time is the last row of the master's
transaction. Once master receives this row, it breaks its connection to
replica due to transaction boundary violation (the last row of the
transaction is received without its beginning).

Adding a NOP row became extraneous since the previous commit, which made
relay find transaction boundaries by itself.

Closes #8958

NO_DOC=bugfix

(cherry picked from commit f5e52b2c)

If non-terminal symbol is referenced in C code then destructor for
expression is not called. Thus we don't need to duplicate. Otherwise we
got a memory leak.

See https://www.sqlite.org/cgi/src/doc/trunk/doc/lemon.html#destructor

Close #9159

NO_DOC=bugfix
NO_TEST=tested by debug ASAN CI (to be turned on)

(cherry picked from commit 36ef3fb4)

In this case we don't have knowledge how to downgrade correctly.

Close #9182

NO_DOC=bugfix

(cherry picked from commit cbe6a4da)

mp_compare_decimal_any_number erroneously assumed that any float or
double from which a decimal can't be created is either infinite or NaN.
This is not true. Any float greater than 1e38 can't fit into our decimal
representation. When such a float got compared to a decimal, an
assertion fired, which was wrong. Luckily, on release build the
comparison was correct. Only the assertion is wrong. Fix it.

Closes #8472

NO_DOC=bugfix

(cherry picked from commit f1b23896)

Our split-brain detection machinery relies among other things on all
nodes tracking the synchro queue confirmed lsn. This tracking was only
added together with the split-brain detection. Only the synchro queue
owner tracked the confirmed lsn before.

This means that after an upgrade all the replicas remember the latest
confirmed lsn as 0, and any PROMOTE/DEMOTE request from the queue owner
is treated as a split brain.

Let's fix this and only enable split-brain detection on the replica set
once the schema version is updated. Thanks to the synchro queue freeze
on restart, this can only happen after a new PROMOTE or DEMOTE entry is
written by one of the nodes, and thus the correct confirmed lsn
is propagated with this PROMOTE/DEMOTE to all the cluster members.

Closes #8996

NO_DOC=bugfix

(cherry picked from commit a844bd37)

With this option enabled (new), the multiresults returned by a stored C
function via iproto aren't wrapped in the additional msgpack array (old).

Due to new behaviour some renames are performed:
* `port_c_dump_msgpack()` -> `port_c_dump_msgpack_wrapped()`, since this
  is dump format with additional msgpack array encoded.
* `port_c_dump_msgpack16()` -> `port_c_dump_msgpack()`, since this
  format is now the default new format of a msgpack dump.

The behaviour of the C port msgpack dumping depends on the
`c_func_iproto_multireturn` option:
* uses `port_c_dump_msgpack()` if set to true (new),
* uses `port_c_dump_msgpack_wrapped()` otherwise (old).

Needed for #4799

@TarantoolBot document
Title: Document `c_func_iproto_multireturn` compat option

Please create a documentation page for the new compat option:
https://tarantool.io/compat/c_func_iproto_multireturn

In the new behaviour, the multiresults returned by a stored C function
via iproto aren't wrapped in the additional msgpack array (old).

```
tarantool> compat.c_func_iproto_multireturn = 'old'
---
...

tarantool> net_box.connect(box.cfg.listen):call('myclib.cfunc')
---
- [true, -1]
...

tarantool> compat.c_func_iproto_multireturn = 'new'
---
...

tarantool> net_box.connect(box.cfg.listen):call('myclib.cfunc')
---
- true
- -1
...

```

The new behaviour is consistent with the local call of the function
via `box.func`:

```
tarantool> box.func['myclib.cfunc']:call()
---
- true
- -1
...

```

Assume you have a stored C function that returns values like the
following:

```c
char *position = mp_encode_bool(buffer, true);
box_return_mp(ctx, buffer, position);
/* ... */
position = mp_encode_int(buffer, -1);
box_return_mp(ctx, buffer, position);
```

If you want to preserve the format of the returned array for your C
functions, when the `c_func_iproto_multireturn` option is set to "new",
you should add the additional wrapping, like the following:

```c
char *position = mp_encode_array(buffer_with_results, n_results);
position = mp_encode_bool(position, true);
/* ... */
position = mp_encode_int(position, -1);
box_return_mp(ctx, buffer_with_results, position);
```

The amount of `box_return_mp()` calls indicates the number of values to
be returned.

Also, you should update its usage via `box.func` if there is any.

(cherry picked from commit 96ee6d9b)

luarocks version updated to version 3.9.2

Closes #6597

NO_DOC=The engine has been updated, the functionality has not changed
NO_TEST=see NO_DOC

(cherry picked from commit 1dc8cd81)

Before this patch if one called `vinyl`, `sql`, `gc` and `memory`
functions from box.info() instance crashed. It's interesting that
`replication_anon` functions worked ok.
This patch fixes that crashes.

Closes #9173

NO_DOC=bugfix

(cherry picked from commit d85556c9)

* test: fix fix-mips64-spare-side-exit-patching
* test: fix `fillmcode()` generator helper
* MIPS: Fix "bad FP FLOAD" assertion.
* Handle table unsinking in the presence of IRFL_TAB_NOMM.
* Fix handling of instable types in TNEW/TDUP load forwarding.
* Fix predict_next() in parser (again).
* Always exit after machine code page protection change fails.

Part of #8825

NO_DOC=LuaJIT submodule bump
NO_TEST=LuaJIT submodule bump

If MP_ERROR map contains two MP_ERROR_STACK keys, then the second call to
`error_set_prev(effect, cur)' will crash, because `effect' is NULL, but
`err == NULL' is false, because it is assigned on the first iteration.
This patch raises an error if more than one MP_ERROR_STACK key is present.

NO_DOC=bugfix

Closes #9136

(cherry picked from commit 990aeee9)

The patch updates curl module to the version 8.3.0 [1] and updates a
CMake module for building curl library.

Changes in CMake module:

- Option `CURL_STATICLIB` is gone and replaced with `BUILD_STATIC_LIBS`.
- Option `CURL_USE_GNUTLS` was added and disabled by default.
- NSS library support was removed and option `CURL_USE_NSS` has been
  removed as well.
- Option `CMAKE_UNITY_BUILD` was added and disabled by default.
- Option `CURL_DISABLE_FORM_API` was added and disabled by default. It is
  in fact depends on `CURL_DISABLE_MIME`, but anyway disabled explicitly.

Changelog: https://curl.se/changes.html#8_3_0

1. https://github.com/curl/curl/releases/tag/curl-8_3_0

NO_DOC=libcurl submodule bump
NO_TEST=libcurl submodule bump

Fixes #9086

(cherry picked from commit 979b4adb)

Some downgrade operations are performed with disabled system space
triggers because they were prohibited recently (creation of SQL built-in
functions) or never allowed (dropping a system space). This works fine
on the instance running downgrade but apparently fails on replicas.

To fix this issue, let's disable the checks the operations that prevent
downgrade in the following scenarios:
 - in the fiber that is currently running a schema upgrade or downgrade;
 - in the applier fiber so that it can replicate changes done by upgrade
   or downgrade on the master;
 - during recovery so that DDL records written to the WAL can be
   replayed.

We already have all the necessary infrastructure in-place - we use it
for allowing DDL operations with an old schema for upgrade.

Closes #9049

NO_DOC=bug fix

(cherry picked from commit 71de4b2c)

NOTE: We don't have the commit that disables DDL operations with an old
schema in 2.11 so we have to backport bits of it from 3.0, see commit
97c2c9a4 ("box: disable DDL with old schema").

If `strlen(name)` is 1, `value_size` is 1, and `extra` is 0, then 15 bytes
are allocated for `struct error_field` in error_payload_prepare(). However,
the size of this structure is 16 because of the padding for the alignment.
Thus TRASH() in error_payload_destroy() writes 1 byte beyond the structure.

Closes #9098

NO_DOC=bugfix

(cherry picked from commit 454ffd13)

An attempt to print a dead fiber raised a fatal error, which is quite
unexpected. This patch updates __tostring metamethod of fiber_object so
that it pushes the "fiber: <fid> (dead)" string instead of the error.
The __serialize metamethod is patched similarly.

Closes #4265

NO_DOC=bugfix

(cherry picked from commit 3421a3bd)