NO_DOC=internal
NO_CHANGELOG=internal

NO_DOC=internal
NO_CHANGELOG=internal
NO_TEST=internal

It is a cherry-pick of the following sqlite commits:
- fca2355
- 648fe49

NO_DOC=internal
NO_CHANGELOG=internal
NO_TEST=internal

Georgy Moshkin authored 3 weeks ago and Denis Smirnov committed 3 weeks ago

NO_DOC=internal
NO_CHANGELOG=internal

Tarantool has a special token type TK_LINEFEED introduced in commit
ea958f41 ("sql: display line and position in syntax errors").
Original sqlite only has the TK_SPACE token, which means that when we
back-port any code which handles TK_SPACE in any way we must
also update it to handle TK_LINEFEED appropriately.

NO_DOC=internal
NO_CHANGELOG=internal

Based on the following sqlite commits:
- e4984a2
- 287fa17
- 7a606e1
- e33f6e7
- 683b0ff
- f607bec
- 7262ca9

NO_DOC=internal
NO_CHANGELOG=internal

Denis Smirnov authored 4 weeks ago and Dmitry Ivanov committed 4 weeks ago

Window function name resolution produced a panic on flattering
sub-queries. Fixed.

NO_DOC=internal
NO_CHANGELOG=internal

Denis Smirnov authored 1 month ago and Dmitry Ivanov committed 1 month ago

Cherry-pick of 287fa17b7 from sqlite

Prevent "UNBOUNDED FOLLOWING" from being used as the starting
boundary of a window-frame. And "UNBOUNDED PRECEDING" from being
used as the ending boundary.

NO_DOC=internal
NO_CHANGELOG=internal

Denis Smirnov authored 5 months ago and Dmitry Ivanov committed 1 month ago

Current commit backports the changes from SQLite commit
9592320b911e23d6ff3ce432e1da9a25773a4288. It introduces support for
SQL window functions in Tarantool, including:

- functions: row_number();
- aggregates: sum(), count(), avg(), min(), max().

NO_DOC=internal
NO_CHANGELOG=internal

NO_DOC=internal
NO_CHANGELOG=internal
NO_TEST=internal

This bug could potentially make it so the actual on_shutdown timeout
would be equal to the one specified by user multiplied by the number of
registered on_shutdown triggers.

NO_DOC=internal
NO_TEST=tested in picodata
NO_CHANGELOG=internal

Split-brain detector might be triggered by complicated online
cluster upgrades (e.g. "the quorum promote patch"), so we don't
want to write inconsistent state to disk.

If there's no inconsistent snapshot, the problem may go away
once we restart the node with a newer version of Tarantool.

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

This patch adds a new control knob to let the user temporarily
disable Tarantool's ability to create snapshots. This might come
in useful when performing an online cluster upgrade which is
known to bring nodes into an inconsistent state, e.g.
"the quorum promote patch".

Before upgrading, the user is supposed to run

```
box.cfg{checkpoint_enabled = false}
```

to temporarily disable both scheduled and manual snapshots.

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

Maksim Kaitmazian authored 3 months ago and Maksim Kaitmazian committed 1 month ago

ssl-luatest/replication_test.lua used to fail because there was no error
set in diag. Actually, the error was set, but in another fiber.
The erroneous scenario was as follows:

1. A fatal error occurs in one fiber, this fiber poisons iostream with
   SSL_IOSTREAM_POISON flag and reports the error using diag_set.
   This operation fails with an error reported.
2. Another fiber starts a new operation and discovers that
   SSL_IOSTREAM_POISON is set, so it returns IOSTREAM_ERROR early.
   However, the error was set only in another fiber, not in this one
   (errors are fiber local), so diag_raise fails on the assertion.

This commit resolves this problem by removing poisoning logic. If a
fatal error occurred, further errors will be reported by opensll,
not due to the flag.
It also adds a new flag SSL_SHUTDOWN_MUST_NOT_BE_CALLED,
which purpose is described in its comment.

Note that the same strategy is used in rust-openssl:
https://github.com/sfackler/rust-openssl/issues/2334

Closes picodata#890.

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

Previously, we'd preserve WAL directory lock during exec(), meaning that
stray child processes would not let us restart the instance:

```
E> ER_ALREADY_RUNNING: Failed to lock WAL directory /data and hot_standby mode is off
F> can't initialize storage: Failed to lock WAL directory /data and hot_standby mode is off
```

This patch fixes that by setting O_CLOEXEC for the fd used in flock().

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

The bug was discovered during the development of the quorum promote feature
-- basically, it causes qpromote_several_outstanding_promotes_test.lua
to fail spontaneously. Furthermore, we suspect that this is the underlying
cause of lost heartbeats and subsequent severe replication lags.

Long story short, whenever we want to send a message from the TX thread
back to a relay thread, we should first check if they are still
connected. Otherwise, we'll see
  * An assertion failure in debug, or
  * (Presumably) a relay hangup in release due to
    `if (status_msg->msg.route != NULL) return;`
    in relay_status_update() -> relay_check_status_needs_update().

The upstream is aware of this issue:
https://github.com/tarantool/tarantool/issues/9920

Backtrace:

```
__pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
0x00007f891f6a5463 in __pthread_kill_internal (threadid=<optimized out>, signo=6) at pthread_kill.c:78
0x00007f891f64c120 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
0x00007f891f6334c3 in __GI_abort () at abort.c:79
0x00007f891f6333df in __assert_fail_base (fmt=0x7f891f7c3c20 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
    assertion=assertion@entry=0x58560077b94b "loop() == pipe->producer", file=file@entry=0x58560077b935 "./src/lib/core/cbus.h", line=line@entry=224,
    function=function@entry=0x58560077b910 "void cpipe_push_input(cpipe*, cmsg*)") at assert.c:94
0x00007f891f644177 in __assert_fail (assertion=0x58560077b94b "loop() == pipe->producer", file=0x58560077b935 "./src/lib/core/cbus.h", line=224,
    function=0x58560077b910 "void cpipe_push_input(cpipe*, cmsg*)") at assert.c:103
0x0000585600328782 in cpipe_push_input (pipe=0x58563115dfc8, msg=0x58563115e028) at tarantool/src/lib/core/cbus.h:224
0x0000585600328802 in cpipe_push (pipe=0x58563115dfc8, msg=0x58563115e028) at tarantool/src/lib/core/cbus.h:241
0x000058560032a157 in tx_status_update (msg=0x58563115e028) at tarantool/src/box/relay.cc:629
0x0000585600468d4b in cmsg_deliver (msg=0x58563115e028) at tarantool/src/lib/core/cbus.c:553
0x0000585600469e50 in fiber_pool_f (ap=0x7f891e8129a8) at tarantool/src/lib/core/fiber_pool.c:64
0x00005856001be16a in fiber_cxx_invoke(fiber_func, typedef __va_list_tag __va_list_tag *) (f=0x585600469b82 <fiber_pool_f>, ap=0x7f891e8129a8)
    at tarantool/src/lib/core/fiber.h:1283
0x000058560045f495 in fiber_loop (data=0x0) at tarantool/src/lib/core/fiber.c:1085
0x0000585600745b8f in coro_init () at tarantool/third_party/coro/coro.c:108
```

Relevant frames:

```
0x0000585600328782 in cpipe_push_input (pipe=0x58563115dfc8, msg=0x58563115e028) at tarantool/src/lib/core/cbus.h:224
224             assert(loop() == pipe->producer);
(gdb)
0x0000585600328802 in cpipe_push (pipe=0x58563115dfc8, msg=0x58563115e028) at tarantool/src/lib/core/cbus.h:241
241             cpipe_push_input(pipe, msg);
(gdb)
0x000058560032a157 in tx_status_update (msg=0x58563115e028) at tarantool/src/box/relay.cc:629
629             cpipe_push(&status->relay->relay_pipe, msg);
(gdb)
0x0000585600468d4b in cmsg_deliver (msg=0x58563115e028) at tarantool/src/lib/core/cbus.c:553
553             msg->hop->f(msg);
(gdb)
0x0000585600469e50 in fiber_pool_f (ap=0x7f891e8129a8) at tarantool/src/lib/core/fiber_pool.c:64
64              cmsg_deliver(msg);
(gdb)
```

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

Previously, log_destroy would close log->fd even if it's one of the
standard streams. This behavior almost never makes sense, unless
one's trying to write a unix daemon which is obviously not the case.

Closing stderr has the following side effects:
- it breaks a reinit of the default logger;
- it inhibits ASan's final leak report;
- it causes a EPOLLHUP during a restart via `--entrypoint-fd` (Picodata).

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

This stores all of the include directories of box into a file
called box-include-args. We use this to generate certain FFI
bindings using rust-bindgen.

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

Apparently, clang 18 is not particularly happy about luajit's static
assert implementation:

```
In file included from tarantool/src/box/box.cc:38:
In file included from tarantool/src/lua/utils.h:47:
In file included from tarantool/third_party/luajit/src/lj_state.h:9:
tarantool/third_party/luajit/src/lj_obj.h:488:1: error: variable length arrays in C++ are a Clang extension; did you mean to use 'static_assert'? [-Werror,-Wvla-extension-static-assert]
  488 | LJ_STATIC_ASSERT(offsetof(Node, val) == 0);
      | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
tarantool/third_party/luajit/src/lj_def.h:370:71: note: expanded from macro 'LJ_STATIC_ASSERT'
  370 |   extern void LJ_ASSERT_NAME(__COUNTER__)(int STATIC_ASSERTION_FAILED[(cond)?1:-1])
      |                                                                       ^~~~~~~~~~~
tarantool/third_party/luajit/src/lj_obj.h:488:18: note: cast that performs the conversions of a reinterpret_cast is not allowed in a constant expression
  488 | LJ_STATIC_ASSERT(offsetof(Node, val) == 0);
      | ~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~
tarantool/src/trivia/util.h:273:33: note: expanded from macro 'offsetof'
  273 | #define offsetof(type, member) ((size_t) &((type *)0)->member)
      |                                 ^
tarantool/third_party/luajit/src/lj_def.h:370:72: note: expanded from macro 'LJ_STATIC_ASSERT'
  370 |   extern void LJ_ASSERT_NAME(__COUNTER__)(int STATIC_ASSERTION_FAILED[(cond)?1:-1])
      |
```

Luckily, we can just mute this.

NO_DOC=<nothing interesting here>
NO_TEST=<tested during build time>
NO_CHANGELOG=<nothing interesting here>

Denis Smirnov authored 6 months ago and Dmitry Ivanov committed 3 months ago

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

Denis Smirnov authored 6 months ago and Dmitry Ivanov committed 3 months ago

Picodata SQL now manages the Tarantool statement cache using a
dedicated SQL fiber that handles preparation and unpreparation of
statements based on an LRU eviction policy. Prepared statements
can be executed across different sessions by SQL clients.

Previously, when a client executed a prepared statement, it
increased the reference count in the statement cache and linked
the statement to the client's session. While this approach seemed
fine, it caused issues during eviction, as references to these
statements remained in client sessions, preventing proper eviction.

This commit addresses the issue by ensuring that if a statement is
added to the current session during execution, it is removed and
the session state is restored once execution is complete.

NO_DOC=internal
NO_CHANGELOG=internal

Denis Smirnov authored 7 months ago and Dmitry Ivanov committed 3 months ago

Temporary spaces, used for cluster-wide SQL data materialization,
were causing unnecessary netbox schema version bumps, leading to
schema downloading via netbox, excessive Lua garbage and GC blocks.
Since these tables are for internal SQL use, we don't need to inform
netbox clients about schema changes. We now maintain separate schema
versions: one for netbox clients and one for the internal prepared
statement cache.

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

Dmitry Rodionov authored 1 year ago and Dmitry Ivanov committed 3 months ago

* smoke test that verifies that cluster can be successfully bootstrapped
* test diverging limbo owner (litmus test)
  - doesn't work on fork master, works with promote chains
* test aba leader
  - two cases where elected leader fails to deliver promote to others
* attempt to model 2+2 quorum lowering case
* two cases with two outstanding promotes
* use new injections for particular xrow types

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

Co-authored-by: Dmitry Ivanov <ivadmi5@gmail.com>

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

Further reading:
* https://git.picodata.io/picodata/tarantool/-/merge_requests/175
* https://github.com/tarantool/tarantool/pull/10334

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

Dmitry Rodionov authored 1 year ago and Dmitry Ivanov committed 3 months ago

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

Co-authored-by: Dmitry Ivanov <ivadmi5@gmail.com>

Arseniy Volynets authored 7 months ago and Dmitry Ivanov committed 3 months ago

- If you prepare and execute statement with params
in the projection and then unprepare the
statement, byte counter may show the wrong value
or even overflow.
- The problem is that when we compile sql
statement, we set parameter type to 'any'.
But when we execute the statement we set parameter
type to actual type. Then we use this type in
calculation of estimated of sql cache entry size.
This leads to different estimated sizes of cache
entry during prepare and during unprepare after
statement was executed
- Fix this by resetting type to 'any' after
executing the statement

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch

Arseniy Volynets authored 7 months ago and Dmitry Ivanov committed 3 months ago

BREAKING CHANGE!:
1. add session id argument to sql_prepare_ext
2. introduce sql_unprepare_ext function.
This function removes prepared stmt using
given session id.

In picodata SQL, we may prepare stmt in one
session and unprepare it in some
other session, which does not know in
what session the statement was prepared. Now
sql_prepare_ext returns not only statement id,
but also a session id. This way statement can
be unprepared from other session using
sql_unprepare_ext.

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch

NO_DOC=internal
NO_CHANGELOG=internal
NO_TEST=internal

Дмитрий Кольцов authored 7 months ago and Dmitry Ivanov committed 3 months ago

Replace fiber.sleep with luatest.helpers.retrying
to make tests less flaky

NO_DOC=internal
NO_CHANGELOG=internal

godzie44 authored 1 year ago and Dmitry Ivanov committed 3 months ago

WAL extensions allows to add auxiliary information to each
write-ahead log record. WAL extensions configured by
`box.cfg.wal_ext` option. Currently, there is only one builtin
extension: `new_old`. `new_old` extension add information about
new and old tuples for ddl operations.

NO_DOC=internal
NO_CHANGELOG=internal

Denis Smirnov authored 8 months ago and Dmitry Ivanov committed 3 months ago

BREAKING CHANGE!:
1. remove sql_prepare from the export list;
2. introduce sql_prepare_ext.

The sql_prepare symbol previously included the tarantool port as
an output parameter. However, this structure was inconvenient for
libraries using the C API, as they primarily required just the
statement ID. To address this issue, the sql_prepare symbol was
replaced with the sql_prepare_ext symbol.

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch