Vladimir Davydov authored 3 months ago and Dmitry Ivanov committed 3 months ago

Once a statement is prepared to be committed to WAL, it becomes visible
(in the 'read-committed' isolation level) so it can be added to the
tuple cache. That's why if the statement is rolled back due to a WAL
error, we have to invalidate the cache. The problem is that the function
invalidating the cache (`vy_cache_on_write`) ignores the statement if
it's a DELETE judging that "there was nothing and there is nothing now".
This is apparently wrong for rollback. Fix it.

Closes #10879

NO_DOC=bug fix

(cherry picked from commit d64e29da2c323a4b4fcc7cf9fddb0300d5dd081f)

Vladimir Davydov authored 3 months ago and Dmitry Ivanov committed 3 months ago

A multikey index stores a tuple once per each entry of the indexed
array field, excluding duplicates. For example, if the array field
equals {1, 3, 2, 3}, the tuple will be stored three times. Currently,
when a tuple with duplicate multikey entries is inserted into a
transaction write set, duplicates are overwritten as if they belonged
to different statements. Actually, this is pointless: we could just as
well skip them without trying to add to the write set. Besides, this
may break the assumptions taken by various optimizations, resulting in
anomalies. Consider the following example:

```lua
local s = box.schema.space.create('test', {engine = 'vinyl'})
s:create_index('primary')
s:create_index('secondary', {parts = {{'[2][*]', 'unsigned'}}})
s:replace({1, {10, 10}})
s:update({1}, {{'=', 2, {10}}})
```

It will insert the following entries to the transaction write set
of the secondary index:

 1. REPLACE {10, 1} [overwritten by no.2]
 2. REPLACE {10, 1} [overwritten by no.3]
 3. DELETE {10, 1} [turned into no-op as REPLACE + DELETE]
 4. DELETE {10, 1} [overwritten by no.5]
 5. REPLACE {10, 1} [turned into no-op as DELETE + REPLACE]

(1-2 correspond to `replace()` and 3-5 to `delete()`)

As a result, tuple {1, {10}} will be lost forever.

Let's fix this issue by silently skipping duplicate multikey entries
added to a transaction write set. After the fix, the example above
will produce the following write set entries:

 1. REPLACE{10, 1} [overwritten by no.2]
 2. DELETE{10, 1} [turned into no-op as REPLACE + DELETE]
 3. REPLACE{10, 1} [committed]

(1 corresponds to `replace()` and 2-3 to `delete()`)

Closes #10869
Closes #10870

NO_DOC=bug fix

(cherry picked from commit 1869dce15d9a797391e45df75507078d91f1651e)

Vladimir Davydov authored 4 months ago and Dmitry Ivanov committed 3 months ago

A Vinyl read iterator scans all read sources (memory and disk levels)
even if it's executed in a read view from which most of the sources are
invisible. As a result, a long running scanning request may spend most
of the time skipping invisible statements. The situation is exacerbated
if the instance is experiencing a heavy write load because it would pile
up old statement versions in memory and force the iterator to skip over
them after each disk read.

Since the replica join procedure in Vinyl uses a read view iterator
under the hood, the issue is responsible for a severe performance
degradation of the master instance and the overall join procedure
slowdown when a new replica is joined to an instance running under
a heavy write load.

Let's fix this issue by making a read iterator skip read sources that
aren't visible from its read view.

Closes #10846

NO_DOC=bug fix

(cherry picked from commit 6a214e42e707b502022622866d898123a6f177f1)

Vladimir Davydov authored 4 months ago and Dmitry Ivanov committed 3 months ago

Statements executed in a transaction are first inserted into the
transaction write set and only when the transaction is committed, they
are applied to the LSM trees that store indexed keys in memory. If the
same key is updated more than once in the same transaction, the old
version is marked as overwritten in the write set and not applied on
commit.

Initially, write sets of different indexes of the same space were
independent: when a transaction was applied, we didn't have a special
check to skip a secondary index statement if the corresponding primary
index statement was overwritten because in this case the secondary
index statement would have to be overwritten as well. This changed when
deferred DELETEs were introduced in commit a6edd455 ("vinyl:
eliminate disk read on REPLACE/DELETE"). Because of deferred DELETEs,
a REPLACE or DELETE overwriting a REPLACE in the primary index write
set wouldn't generate DELETEs that would overwrite the previous key
version in write sets of the secondary indexes. If we applied such
a statement to the secondary indexes, it'd stay there forever because,
since there's no corresponding REPLACE in the primary index, a DELETE
wouldn't be generated on primary index compaction. So we added a special
instruction to skip a secondary index statement if the corresponding
primary index was overwritten, see `vy_tx_prepare()`. Actually, this
wasn't completely correct because we skipped not only secondary index
REPLACE but also DELETE. Consider the following example:

```lua
local s = box.schema.space.create('test', {engine = 'vinyl'})
s:create_index('primary')
s:create_index('secondary', {parts = {2, 'unsigned'}})

s:replace{1, 1}

box.begin()
s:update(1, {{'=', 2, 2}})
s:update(1, {{'=', 2, 3}})
box.commit()
```

UPDATEs don't defer DELETEs because, since they have to query the old
value, they can generate DELETEs immediately so here's what we'd have
in the transaction write set:

 1. REPLACE {1, 2} in 'test.primary' [overwritten by no.4]
 2. DELETE {1, 1} from 'test.secondary'
 3. REPLACE {1, 2} in 'test.secondary' [overwritten by no.5]
 4. REPLACE{1, 3} in 'test.primary'
 5. DELETE{1, 2} from 'test.secondary'
 6. REPLACE{1, 3} in 'test.secondary'

Statement no.2 would be skipped and marked as overwritten because of
the new check, resulting in {1, 1} never deleted from the secondary
index. Note, the issue affects spaces both with and without enabled
deferred DELETEs.

This commit fixes this issue by updating the check to only skip REPLACE
statements. It should be safe to apply DELETEs in any case.

There's another closely related issue that affects only spaces with
enabled deferred DELETEs. When we generate deferred DELETEs for
secondary index when a transaction is committed (we can do it if we find
the previous version in memory), we assume that there can't be a DELETE
in a secondary index write set. This isn't true: there can be a DELETE
generated by UPDATE or UPSERT. If there's a DELETE, we have nothing to
do unless the DELETE was optimized out (marked as no-op).

Both issues were found by `vinyl-luatest/select_consistency_test.lua`.

Closes #10820
Closes #10822

NO_DOC=bug fix

(cherry picked from commit 6a87c45deeb49e4e17ae2cc0eeb105cc9ee0f413)

Also, remove unreleased/ entries.

NO_DOC=changelog
NO_TEST=changelog
NO_CHANGELOG=changelog

Currently, we use raw index for count operation instead of
`box_index_count`. As a result, we skip a check if current transaction
can continue and we don't begin transaction in engine if needed. So, if
count statement is the first in a transaction, it won't be tracked by
MVCC since it wasn't notified about the transaction. The commit fixes
the mistake. Also, the commit adds a check if count was successful and
covers it with a test.

In order to backport the commit to 2.11, space name was wrapped with
quotes since it is in lower case and addressing such spaces with SQL
without quotes is Tarantool 3.0 feature. Another unsupported feature is
prohibition of data access in transactional triggers - it was used in a
test case so it was rewritten.

Closes #10825

NO_DOC=bugfix

(cherry picked from commit 0656a9231149663a0f13c4be7466d4776ccb0e66)

`vy_mem_insert()` and `vy_mem_insert_upsert()` increment the row count
statistic of `vy_mem` only if no statement is replaced, which is
correct, while `vy_lsm_commit()` increments the row count of `vy_lsm`
unconditionally. As a result, `vy_lsm` may report a non-zero statement
count (via `index.stat()` or `index.len()`) after a dump. This may
happen only with a non-unique multikey index, when the statement has
duplicates in the indexed array, and only if the `deferred_deletes`
option is enabled, because otherwise we drop duplicates when we form
the transaction write set, see `vy_tx_set()`. With `deferred_deletes`,
we may create a `txv` for each multikey entry at the time when we
prepare to commit the transaction, see `vy_tx_handle_deferred_delete()`.

Another problem is that `vy_mem_rollback_stmt()` always decrements
the row count, even if it didn't find the rolled back statement in
the tree. As a result, if the transaction with duplicate multikey
entries is rolled back on WAL error, we'll decrement the row count
of `vy_mem` more times than necessary.

To fix this issue, let's make the `vy_mem` methods update the in-memory
statistic of `vy_lsm`. This way they should always stay in-sync. Also,
we make `vy_mem_rollback_stmt()` skip updating the statistics in case
the rolled back statement isn't present in the tree.

This issue results in `vinyl-luatest/select_consistency_test.lua`
flakiness when checking `index.len()` after compaction. Let's make
the test more thorough and also check that `index.len()` equals
`index.count()`.

Closes #10751
Part of #10752

NO_DOC=bug fix

(cherry picked from commit e8810c555d4e6ba56e6c798e04216aa11efb5304)

This commit fixes some cases of upgrading schema from 1.6.9:

1. Fix updating empty password for users. In 1.6 credentials were array
   in _user, in 1.7.5 they became map.

2. Automatically update the format of user spaces. Format of system
   spaces have been properly fixed during upgrade to 1.7.5. However,
   commit 519bc82e ("Parse and validate space formats") introduced
   strict checking of format field in 1.7.6. So, the format of user
   spaces should be also fixed.

Back in 1.6 days, it was allowed to write anything in space format.
This commit only fixes valid uses of format:
    {name = 'a', type = 'number'}
    {'a', type = 'number'}
    {'a', 'num'}
    {'a'}

Invalid use of format (e.g. {{}}, or {{5, 'number'}} will cause error
anyway. User has to fix the format on old version and only after that
start a new one.

This commit also introduces the test, which checks, that we can
properly upgrade from 1.6.9 to the latest versions, at least in basic
cases.

Closes #10180

NO_DOC=bugfix

(cherry picked from commit f69e2ae488b3620e31f1a599d8fb78a66917dbfd)

When building an index in background, we create on_rollback triggers for
tuples inserted concurrently. The problem here is on_rollback trigger
has independent from `index` and `memtx_ddl_state` lifetime - it can be
called after the index was build (and `memtx_ddl_state` is destroyed)
and even after the index was altered. So, in order to avoid
use-after-free in on_rollback trigger, let's drop all on_rollback
triggers when the DDL is over. It's OK because all owners of triggers
are already prepared, hence, in WAL or replication queue (since we
build indexes in background only without MVCC so the transactions cannot
yield), so if they are rolled back, the same will happen to the DDL.

In order to delete on_rollback triggers, we should collect them into a
list in `memtx_ddl_state`. On the other hand, when the DML statement is
over (committed or rolled back), we should delete its trigger from the
list to prevent use-after-free. That's why the commit adds the on_commit
trigger to background build process.

Closes #10620

NO_DOC=bugfix

(cherry picked from commit d8d82dba4c884c3a7ad825bd3452d35627c7dbf4)

The commit bumps luafun to the new version with a bunch of bugfixes:
* Now `chain` works correctly with iterators without `param`.
* Now `drop_while` supports stateful iterators.
* The module is populated with missing `maximum_by` alias of `max_by`.
* Now `nth` and `length` work correctly with other luafun iterators.

Since our index iterators are stateful (can return different values
with the same `state` passed), the old `drop_while` implementation
didn't work well with them - it was skipping an extra element.
The bump resolves this issue.

Note that there are still methods that don't work correctly with
`index:pairs` - `cycle`, `head` and `is_null`.

Closes #6403

NO_DOC=bugfix

(cherry picked from commit ec758869f8364624efaff58bdd4ebc7c133ede0a)

* Fix typo.
* OSX/iOS: Always generate 64 bit non-FAT Mach-O object files.
* test: don't run JIT-based LuaJIT tests without JIT
* test: actualize <LuaJIT-tests/README.md>
* test: enable <misc/alias_alloc.lua> LuaJIT test
* test: refactor <alias_alloc.lua> LuaJIT test
* test: refactor <lang/coroutine.lua> LuaJIT test
* test: remove <misc/coro_yield.lua> LuaJIT test
* test: enable <misc/debug_gc.lua> LuaJIT test
* test: enable <misc/dualnum.lua> LuaJIT test
* test: refactor <lang/dualnum.lua> LuaJIT test
* test: remove <misc/fori_coerce.lua> LuaJIT test
* test: remove <misc/fori_dir.lua> LuaJIT test
* test: remove <misc/gc_rechain.lua> LuaJIT test
* test: enable <misc/gc_trace.lua> LuaJIT test
* test: refactor <trace/gc.lua> LuaJIT test
* test: enable <misc/gcstep.lua> LuaJIT test
* test: enable <misc/hook_active.lua> LuaJIT test
* test: enable <misc/hook_line.lua> LuaJIT test
* test: enable <misc/hook_norecord.lua> LuaJIT test
* test: enable <misc/hook_record.lua> LuaJIT test
* test: enable <misc/hook_top.lua> LuaJIT test
* test: enable <misc/jit_flush.lua> LuaJIT test
* test: remove <misc/loop_unroll.lua> LuaJIT test
* test: enable <misc/parse_comp.lua> LuaJIT test
* test: enable <misc/parse_esc.lua> LuaJIT test
* test: enable <misc/parse_misc.lua> LuaJIT test
* test: enable <misc/phi_conv.lua> LuaJIT test
* test: refactor <trace/phi/conv.lua> LuaJIT test
* test: enable <misc/recurse_deep.lua> LuaJIT test
* test: remove <misc/recurse_tail.lua> LuaJIT test
* test: enable <misc/stack_gc.lua> LuaJIT test
* test: refactor <lang/gc_stack.lua> LuaJIT test
* test: enable <misc/stack_purge.lua> LuaJIT test
* test: refactor <trace/stack_purge.lua> LuaJIT test
* test: enable <misc/stackov.lua> LuaJIT test
* test: enable <misc/stackovc.lua> LuaJIT test
* test: enable <misc/tcall_base.lua> LuaJIT test
* test: refactor <trace/tcall_base.lua> LuaJIT test
* test: enable <misc/tcall_loop.lua> LuaJIT test
* test: enable <misc/tonumber_scan.lua> LuaJIT test
* test: remove <misc/uclo.lua> LuaJIT test
* test: enable <misc/unordered_jit.lua> LuaJIT test
* test: enable <misc/wbarrier.lua> LuaJIT test
* test: enable <misc/wbarrier_jit.lua> LuaJIT test
* test: enable <misc/wbarrier_obar.lua> LuaJIT test
* test: update <LuaJIT-tests/README.md>
* test: off JIT for routines in <lang/stackov.lua>
* Limit number of string format elements to compile.
* Clear stack after print_jit_status() in CLI.
* FFI: Workaround for platform dlerror() returning NULL.
* test: move profilers tests to subdirectory
* test: rename <arm64-ccall-fp-convention.test.lua>
* cmake: introduce AppendTestEnvVar macro
* test: shrink LUA_PATH environment variable
* test: shrink LUA_CPATH and {DY}LD_LIBRARY_PATH
* test: skip flaky tests with enabled table bump
* test: set LD_PRELOAD only when necessary
* test: fix misclib-getmetrics-lapi.test.lua
* FFI: Fix __tostring metamethod access to enum cdata value.
* Fix limit check in narrow_conv_backprop().
* FFI: Drop finalizer table rehash after GC cycle.
* Drop unused function wrapper.
* FFI: Fix various issues in recff_cdata_arith.
* Add missing coercion when recording select(string, ...)
* Fix stack allocation after on-trace stack check.
* Restore state when recording __concat metamethod throws an error.
* Fix bit op coercion in DUALNUM builds.
* FFI: Fix 64 bit shift fold rules.
* Limit CSE for IR_CARG to fix loop optimizations.

Closes #10290
Closes #10199
Closes #9898
Part of #9398

NO_DOC=LuaJIT submodule bump
NO_TEST=LuaJIT submodule bump

Since we often search spaces, users, funcs and so on in internal caches
that have `read-committed` isolation level (prepared tuples are seen),
let's always allow to read prepared tuples of system spaces.

Another advantage of such approach is that we never handle MVCC when
working with system spaces, so after the commit they will behave in the
same way - prepared tuples will be seen. The only difference is that
readers of prepared rows will be aborted if the row will be rolled back.

By the way, the inconsistency between internal caches and system spaces
could lead to crash in some sophisticated scenarios - the commit fixes
this problem as well because now system spaces and internal caches are
synchronized.

Closes #10262
Closes tarantool/security#131

NO_DOC=bugfix

(cherry picked from commit b33f17b25de6bcbe3ebc236250976e4a0250e75e)

Yielding DDL operations acquire DDL lock so that the space cannot be
modified under its feet. However, there is a case when it actually can:
if a yielding DDL has started when there is another DDL is being
committed and it gets rolled back due to WAL error, `struct space`
created by rolled back DDL is deleted - and it's the space being altered
by the yielding DDL. In order to fix this problem, let's simply wait for
all previous alters to be committed.

We could use `wal_sync` to wait for all previous transactions to be
committed, but it is more complicated - we need to use `wal_sync` for
single instance and `txn_limbo_wait_last_txn` when the limbo queue has
an owner. Such approach has more pitfalls and requires more tests to
cover all cases. When relying on `struct alter_space` directly, all
situations are handled with the same logic.

Alternative solutions that we have tried:
1. Throw an error in the case when user tries to alter space when
   there is another non-committed alter. Such approach breaks applier
   since it applies rows asynchronously. Trying applier to execute
   operations synchronously breaks it even harder.
2. Do not use space in `build_index` and `check_format` methods. In this
   case, there is another problem: rollback order. We have to rollback
   previous alters firstly, and the in-progress one can be rolled back
   only after it's over. It breaks fundamental memtx invariant: rollback
   order must be reverse of replace order. We could try to use
   `before_replace` triggers for alter, but the patch would be bulky.

Closes #10235

NO_DOC=bugfix

(cherry picked from commit fee8c5dd6b16471739ed8512ba4137ff2e7274aa)

The reason is that the previous libcurl submodule update in commit
0919f390802f146852b462215327ef03e2730cfc ("third_party: update libcurl from
8.8.0 to 8.10.1") reveals the following regression:

NOWRAP
```c
$ tarantool -e "require('http.client').new():get('https://google.com') collectgarbage()"
tarantool: ./third_party/curl/lib/multi.c:3691: curl_multi_assign: Assertion `!(multi)' failed.
Aborted (core dumped)
```
NOWRAP

The stacktrace is the following:

NOWRAP
```c
<...>
#4  __assert_fail
#5  curl_multi_assign            // <- called by us
#6  curl_multi_sock_cb           // <- this is our callback
#7  Curl_multi_pollset_ev
#8  cpool_update_shutdown_ev
#9  cpool_discard_conn
#10 cpool_close_and_destroy_all
#11 Curl_cpool_destroy
#12 curl_multi_cleanup
#13 curl_env_finish              // <- destroy the multi handle
#14 httpc_env_finish
#15 luaT_httpc_cleanup
#16 lj_BC_FUNCC
#17 gc_call_finalizer
#18 gc_finalize
#19 gc_onestep
#20 lj_gc_fullgc
#21 lua_gc
#22 lj_cf_collectgarbage
#23 lj_BC_FUNCC
#24 lua_pcall
#25 luaT_call
#26 lua_main
#27 run_script_f
#28 fiber_cxx_invoke
#29 fiber_loop
#30 coro_init
```
NOWRAP

The multi handle is during the destroy, but our
`CURLMOPT_SOCKETFUNCTION` callback is invoked and the
`curl_multi_assign()` call (invoked to associate a libev watcher to the
given file descriptor) fails on the assertion. Everything is as
described in https://github.com/curl/curl/issues/15201.

The first bad libcurl's commit is [curl-8_10_0-4-g48f61e781][1], but
later it was fixed in [curl-8_10_1-241-g461ce6c61][2]. This commit
updates libcurl to this revision to fix the regression.

Adjusted build options in our build script:

* Added `CURL_DISABLE_IPFS=ON`: [curl-8_10_1-57-gce7d0d413][3]
* Added `CURL_TEST_BUNDLES=OFF`: [curl-8_10_1-67-g71cf0d1fc][4]
* Changed `ENABLE_WEBSOCKETS=OFF` to `CURL_DISABLE_WEBSOCKETS=ON`:
  [curl-8_10_1-130-gd78e129d5][5]

[1]: https://github.com/curl/curl/commit/48f61e781a01e6a8dbc4a347e280644b1c68ab6a
[2]: https://github.com/curl/curl/commit/461ce6c6160b86439ddd74c59541231ec9e8558e
[3]: https://github.com/curl/curl/commit/ce7d0d41378007eda676c83ad6b86c59870cc9f1
[4]: https://github.com/curl/curl/commit/71cf0d1fca9e1f53524e1545ef0c08d174458d80
[5]: https://github.com/curl/curl/commit/d78e129d50b2d190f1c1bde2ad1f62f02f152db0

NO_DOC=bugfix
NO_CHANGELOG=fixes an unreleased commit
NO_TEST=can't reproduce without https to add a test case, verified
        locally

(cherry picked from commit fbe6d0a0a40945c42609f5119a007b5c3980c232)

Updates curl module to the version 8.10.1. The new version brings
several new options, such as:

CURL_USE_RUSTLS - Enables Rustls for SSL/TLS. Added in commit
curl/curl@ed76a23fccc1 ("cmake: add rustls")

CURL_USE_WOLFSSH - Option to use wolfSSH. Added in commit
curl/curl@0d8fdd1c7421 ("cmake: add wolfSSH support")

CURL_USE_GSASL - Option to use libgsasl.
Added in commit curl/curl@66bf995d1cfc
("cmake: add CURL_USE_GSASL option with detection + CI test")

CURL_DISABLE_SHA512_256 - Disables SHA-512/256 hash algorithm.
Added in commit curl/curl@33629949488c
("build: add options to disable SHA-512/256 hash algo")

CURL_USE_LIBUV - Use libuv for event-based tests. Added in commit
curl/curl@f7d5f47059c3 ("cmake: add support for `CURL_USE_LIBUV` option")

Corrected http_client test according to curl commit curl/curl@b7e769dc872d
("vtls: stop offering alpn http/1.1 for http2-prior-knowledge")

Build file missed several options and paths not used in the current build.
List of these options was added into curl-excluded-options.txt for the
convenience of the following bumps.

Closes #10576

https://curl.se/changes.html#8_10_1
https://github.com/curl/curl/releases/tag/curl-8_10_1

NO_DOC=libcurl submodule bump
NO_TEST=libcurl submodule bump

(cherry picked from commit 0919f390802f146852b462215327ef03e2730cfc)

@Totktonada: the http_client test mentioned in the original commit
message doesn't present in the `release/2.11` branch.

Some curl options appear to missed by several previous curl bumps.
Here is a list of missed options with curl commits in which
they first appeared:

BUILD_EXAMPLES - Build libcurl examples.
Added in curl 8.8.0 release in commit curl/curl@dfdd978f7c60
("cmake: add `BUILD_EXAMPLES` option to build examples")

USE_ECH - Enables ECH support.
Added in curl 8.8.0 release in commit curl/curl@a362962b7289
("TLS: add support for ECH (Encrypted Client Hello)")

USE_HTTPSRR - Enables HTTPS RR support for ECH.
Added in curl 8.8.0 release in commit curl/curl@a362962b7289
("TLS: add support for ECH (Encrypted Client Hello)")

BUILD_STATIC_CURL - Builds curl executable with static libcurl.
Added in curl 8.3.0 release in commit curl/curl@1199308dbc90
("cmake: support building static and shared libcurl in one go")

CURL_DISABLE_NEGOTIATE_AUTH - Disables negotiate authentication.
Added in curl 8.3.0 release in commit curl/curl@e92edfbef644
("lib: add ability to disable auths individually")

CURL_DISABLE_SRP - Disables TLS-SRP support.
Added in curl 8.4.0 release in commit curl/curl@781242ffa44a
("cmake: detect TLS-SRP in OpenSSL/wolfSSL/GnuTLS")

NO_TEST=does not change tarantool behavior
NO_DOC=does not change tarantool behavior

(cherry picked from commit 97e3136ddc691f42cf0a5dff27881978b3c25d52)

This patch optimizes the process of collecting ACKs from replicas for
synchronous transactions. Before this patch, collecting confirmations
was slow in some cases. There was a possible situation where it was
necessary to go through the entire limbo again every time the next ACK
was received from the replica. This was especially noticeable in the
case of a large number of parallel synchronous requests.
For example, in the 1mops_write bench with parameters --fibers=6000
--ops=1000000 --transaction=1, performance increases by 13-18 times on
small clusters of 2-4 nodes and 2 times on large clusters of 31 nodes.

Closes #9917

NO_DOC=performance improvement
NO_TEST=performance improvement

(cherry picked from commit 4a866f64d64c610a3c8441835fee3d8dda5eca71)

Vinyl doesn't support altering the primary index of a non-empty space,
but the check forbidding this isn't entirely reliable - the DDL function
may yield to wait for pending WAL writes to finish after ensuring that
the space doesn't contain any tuples. If a new tuples is inserted into
the space in the meantime, the DDL operation will proceed rebuilding
the primary index and trigger a crash because the code is written on
the assumption that it's rebuilding a secondary index:

```
./src/box/vinyl.c:1572: vy_check_is_unique_secondary_one: Assertion `lsm->index_id > 0' failed.
```

Let's fix this by moving the check after syncing on WAL.

Closes #10603

NO_DOC=bug fix

(cherry picked from commit 955537b57c2aade58b7ca42501a9bbe50dd91f26)

Curl option BUILD_MISC_DOCS builds misc man pages and set ON by default.
Other documentation building options such as ENABLE_CURL_MANUAL and
BUILD_LIBCURL_DOCS was set OFF in BuildLibCurl.cmake.
I suppose this option has to be added in commit 7192bf66
("third_party: update libcurl from 8.7.0 to 8.8.0+patches") and set OFF.

Follows up #9885

NO_TEST=does not change tarantool behavior
NO_DOC=does not change tarantool behavior

(cherry picked from commit 1080995fa4083c4de6aa19e964b98f0ffb7e34c5)

`index.count()` may hang for too long in Vinyl if a substantial
consecutive hunk of the space is stored in memory. Let's add
a fiber slice check to it to prevent it from blocking the TX thread
for too long.

Closes #10553

NO_DOC=bug fix

(cherry picked from commit e19bca5a74e83d2521fe770f2a93c3e3d3ad4801)

The tuple cache doesn't store historical data. It stores only the newest
tuple versions, including prepared but not yet confirmed (committed but
not written to WAL) tuples. This means that transactions sent to a read
view shouldn't add any new chains to the cache because such a chain may
bypass a tuple invisible from the read view.

A transaction may be sent to a read view in two cases:

 1. If some other transactions updates data read by it.
 2. If the transaction is operating in the 'read-confirmed' isolation
    mode and skips an unconfirmed tuple while scanning the memory level.
    This was added in commit 588170a7 ("vinyl: implement transaction
    isolation levels").

The second point should be checked by the read iterator itself, and it
is indeed for the standard case when we scan the memory level before
reading the disk. However, there's the second case: if some other tuples
are inserted into the memory level while the read iterator was waiting
for a disk read to complete, it rescans the memory level and may skip
a new unconfirmed tuple that wasn't there the first time we scanned
the memory level. Currently, if this happens, it won't send itself to
a read view and may corrupt the cache by inserting a chain that skips
over the unconfirmed tuple. Fix this by adding the missing check.

While we are at it, let's simplify the code a bit by moving the check
inside `vy_read_iterator_scan_mem()`. It's okay because sending to
a read view a transaction that's already in the read view is handled
correctly by `vy_tx_send_to_read_view()`.

Closes #10558

NO_DOC=bug fix

(cherry picked from commit a3feee322e76a1e10ab874e63f17f97b6457b59d)

`vy_slice_stream_next()` clears the return value on failure. This isn't
expected by `vy_write_iterator_merge_step()`, which doesn't update
the source position in the `vy_wirte_iterator::src_heap` in this case.
As a result, an attempt to remove `end_of_key_src` from the heap in
`vy_write_iterator_build_history()` may crash as follows:

```
 # 1  0x572a2ecc21a6 in crash_collect+256
 # 2  0x572a2ecc2be2 in crash_signal_cb+100
 # 3  0x7cfef6645320 in __sigaction+80
 # 4  0x572a2eab16de in tuple_format+16
 # 5  0x572a2eab1a25 in vy_stmt_is_key+24
 # 6  0x572a2eab1be8 in vy_stmt_compare+89
 # 7  0x572a2eab1e37 in vy_entry_compare+74
 # 8  0x572a2eab2913 in heap_less+88
 # 9  0x572a2eab21e3 in vy_source_heap_sift_up+255
 # 10 0x572a2eab20b9 in vy_source_heap_update_node+54
 # 11 0x572a2eab25c1 in vy_source_heap_delete+249
 # 12 0x572a2eab4134 in vy_write_iterator_build_history+1497
 # 13 0x572a2eab4995 in vy_write_iterator_build_read_views+193
 # 14 0x572a2eab4ce6 in vy_write_iterator_next+380
 # 15 0x572a2eadd20b in vy_task_write_run+1132
 # 16 0x572a2eade6cf in vy_task_compaction_execute+124
 # 17 0x572a2eadfa8d in vy_task_f+445
 # 18 0x572a2e9ea143 in fiber_cxx_invoke(int (*)(__va_list_tag*), __va_list_tag*)+34
 # 19 0x572a2eccee7c in fiber_loop+219
 # 20 0x572a2f0aef18 in coro_init+120
```

Normally, a function shouldn't update the return value on failure so
let's fix `vy_slice_stream_next()`.

Closes #10555

NO_DOC=bug fix

(cherry picked from commit f1144c533b6c52c324ffe1cc4fcaeab1f2f6cd9f)

There is an option tz in `datetime.parse()`, it was added in
commit 3c403661 ("datetime, lua: date parsing functions").
The option is not documented, and the commit message says that
option `tz` is "Not yet implemented in this commit.".

The patch added tests and a doc request for this option.
The behaviour of the option `tz` is the same as with option
`tzoffset`:
- if timezone was not set in a parsed string then it is set to
  a value specified by `tz`
- if timezone was set in a parsed string then option `tz` is
  ignored

```
tarantool> date.parse("1970-01-01T01:00:00 MSK", { tz = 'Europe/Paris' })
---
- 1970-01-01T01:00:00 MSK
- 23
...

tarantool> date.parse("1970-01-01T01:00:00", { tz = 'Europe/Paris' })
---
- 1970-01-01T01:00:00 Europe/Paris
- 19
...
```

Follows up #6731
Fixes #10420

@TarantoolBot document
Title: Introduce option `tz` in `datetime.parse()`

The option `tz` is added in a function `datetime.parse()`.
The option set timezone to a passed value if it was not set in
a parsed string.

(cherry picked from commit c6bab23a6dc4f819167cbc78eb93859847a389ea)

The patch fixes a behaviour, when `datetime.parse()` ignores
`tzoffset` option if custom format is used.

Fixes #8333
Relates to #10420

NO_DOC=bugfix

(cherry picked from commit 04811e032f29afe0fa6206ef2c7a0f8434861830)

The patch forbids using non-integer values in datetime's `:set()`
for `year`, `month`, `day`, `hour`, `min`, `sec`, `usec`, `msec`,
`nsec` and `tzoffset` keys. `timestamp` can be double, and integer
values allowed in timestamp if `nsec`, `usec`, or `msecs`
provided. An error will be raised when a value of incorrect type
is passed.

Fixes #10391

@TarantoolBot document
Title: Update types of datetime values passed to SQL's `CAST();`

`CAST` can accept only integer values for `year`, `month`, `day`,
`hour`, `min`, `sec`, `usec`, `msec`, `nsec` and `tzoffset`.
`timestamp` can be integer or double.

(cherry picked from commit f57be571b5e4cc8d57c7e97c15b52df37ad6f12c)

The patch forbids using non-integer values in datetime's `:set()`
for `year`, `month`, `day`, `hour`, `min`, `sec`, `usec`, `msec`
and `nsec` keys. The type of `tzoffset` can be integer or string,
`timestamp` can be double, and integer values allowed in timestamp
if `nsec`, `usec`, or `msecs` provided. An error will be raised
when a value of incorrect type is passed.

Part of #10391

@TarantoolBot document
Title: Update types of values passed to `:set()` and parse()

`:set()` can accept only integer values for `year`, `month`,
`day`, `hour`, `min`, `sec`, `usec`, `msec` and `nsec`.
The type of `tzoffset` can be integer or string, `timestamp` can
be integer or double. `tzoffset` passed to `datetime.parse()`
can be integer or string.

(cherry picked from commit 6e77907baa3cbeebc79241cc0046a539a09e3f2c)

The patch forbids using non-integer values in datetime constructor
`datetime.new()` for `year`, `month`, `day`, `hour`, `min`, `sec`,
`usec`, `msec` and `nsec` keys. The type of `tzoffset` can be
integer or string, `timestamp` can be double, and integer values
allowed in timestamp if `nsec`, `usec`, or `msecs` provided.
An error will be raised when a value of incorrect type is passed.

Part of #10391

@TarantoolBot document
Title: Update types of values passed to `datetime.new()`

`datetime.new()` can accept only integer values for `year`,
`month`, `day`, `hour`, `min`, `sec`, `usec`, `msec` and `nsec`.
The type of `tzoffset` can be integer or string, `timestamp` can
be integer or double.

(cherry picked from commit cc9010a2b11477b2f16f2b2e168a6b9dcca2fb20)

When DDL happens, we remove statements of concurrent transactions
from MVCC. When removing statements, we set their `engine_savepoint` to
`NULL` so that they won't be rolled back because we've already handled
them.

However, we remove statements only from stories, and not all statements
can be accessed in this way. For example, when we have several delete
statements of one story, and one of them gets prepared, others are
unlinked. It leads to use-after-free, but it's read-only and doesn't
affect anything, so only ASAN can catch it. It happens when the
statement is being rolled back in `memtx_engine_rollback_statement`:
we check if `space->upgrade` is not `NULL` (space can be already deleted)
but this check affects instruction flow only if `stmt->new_tuple != NULL`
and in our case that's not so. Anyway, let's iterate over all statements
of all transactions and remove savepoints for ones related to the space
that is being invalidated. It takes more time, but anyway, we are doing
DDL that is heavy, so it doesn't really matter.

Along the way, the commit removes helper `memtx_tx_history_remove_stmt`
and all its helpers because they are not needed anymore. This helper
unlinks added story from history chain, links all its delete statements
to the previous story, if any, unlinks the statement from related stories
and sets `engine_savepoint` to `NULL`. Since we already do all of this
things except for unlinking statements from stories, let's simply call
`memtx_tx_story_unlink_added[deleted]_by` instead. This change makes the
code much more straightforward.

Closes #10146

NO_DOC=bugfix

(cherry picked from commit ac112b73192ad96271a02ee85dba3e9737fdaa9d)

We cannot tolerate index extent memory allocation failure on rollback.
At the same time it is not practical to reserve memory because a whole
index can easily be changed on rollback if read view is created before
rollback.

So in case of rollback and memtx memory OOM let's allocate outside the
memtx arena limited by quota.

Now part of the index can reside outside memtx arena. But regularly the
index changes will move this part back to the memtx arena. Until next
such situation of course.

Closes #10551

NO_DOC=bugfix

(cherry picked from commit 32ea713af0a4f27f9ae37bb767c21722ee8c6742)

Whenever a range is compacted, split, or coalesced, we log the range
boundaries. This gets really annoying if there's an index that has
a lot of key parts or contains binary strings. Let's lower the level
used for logging these events down to VERBOSE so that they are not
shown by default but can be enabled if needed.

Closes #10524

NO_DOC=bug fix

(cherry picked from commit 06fa83947b0b63c39732efba4c9d67578f113612)

The fix itself is in the small submodule which is bumped in the previous
commit.

Closes #10148

NO_DOC=bugfix

(cherry picked from commit e4ce9e111483a24d66e078f4f05679d309fcb94d)

`vy_page_stmt()` may fail (return NULL) if:
 - the statement is corrupted;
 - memory allocation for the statement fails;
 - the statement size exceeds `box.cfg.vinyl_max_tuple_size`.

If this happens `vy_page_find_key()` won't return an error. Instead,
it'll either point the caller to a wrong statement or claim that there's
no statement matching the key in this page. This may result in invalid
index selection results and, later on, a crash caused by inconsistencies
in the tuple cache. The issue was introduced by commit ac8ce023
("vinyl: factor out function to lookup key in page").

All of the three cases are actually very unlikely to happen in
production:
 - If a statement stored in a run file is corrupted, we'll probably fail
   to load the whole page due to failed checksums and never even get to
   `vy_page_stmt()`.
 - Statements are allocated with `malloc()`, which doesn't normally
   fail (instead the whole process would be terminated by OOM) .
 - Users don't tend to lower the tuple size limit after restart.

Still, let's fix the issue by implementing proper error handling for
`vy_page_find_key()`.

Closes #10512

NO_DOC=bug fix

(cherry picked from commit 9dbaa6a9bc0d65984b417f8a76aa8373b6125d16)

`ffi.C.tnt_iconv_open` returns pointer to `struct iconv`. In this case
`__gc` in metatable is not bound to the object.

Closes #10487
Part-of #10211

NO_TEST=covered by existing tests
NO_DOC=bugfix

(cherry picked from commit 105e6188ee6cc8de71ca2ab077f78f51be07559d)

The issue is we increment `page_count` only on page write. If we fail
for some reason before then page info `min_key` in leaked.

LSAN report for 'vinyl/recovery_quota.test.lua':

```
2024-07-05 13:30:34.605 [478603] main/103/on_shutdown vy_scheduler.c:1668 E> 512/0: failed to compact range (-inf..inf)

=================================================================
==478603==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 4 byte(s) in 1 object(s) allocated from:
    #0 0x5e4ebafcae09 in malloc (/home/shiny/dev/tarantool/build-asan-debug/src/tarantool+0x1244e09) (BuildId: 20c5933d67a3831c4f43f6860379d58d35b81974)
    #1 0x5e4ebb3f9b69 in vy_key_dup /home/shiny/dev/tarantool/src/box/vy_stmt.c:308:14
    #2 0x5e4ebb49b615 in vy_page_info_create /home/shiny/dev/tarantool/src/box/vy_run.c:257:23
    #3 0x5e4ebb48f59f in vy_run_writer_start_page /home/shiny/dev/tarantool/src/box/vy_run.c:2196:6
    #4 0x5e4ebb48c6b6 in vy_run_writer_append_stmt /home/shiny/dev/tarantool/src/box/vy_run.c:2287:6
    #5 0x5e4ebb72877f in vy_task_write_run /home/shiny/dev/tarantool/src/box/vy_scheduler.c:1132:8
    #6 0x5e4ebb73305e in vy_task_compaction_execute /home/shiny/dev/tarantool/src/box/vy_scheduler.c:1485:9
    #7 0x5e4ebb73e152 in vy_task_f /home/shiny/dev/tarantool/src/box/vy_scheduler.c:1795:6
    #8 0x5e4ebb01e0b1 in fiber_cxx_invoke(int (*)(__va_list_tag*), __va_list_tag*) /home/shiny/dev/tarantool/src/lib/core/fiber.h:1331:10
    #9 0x5e4ebc389ee0 in fiber_loop /home/shiny/dev/tarantool/src/lib/core/fiber.c:1182:18
    #10 0x5e4ebd3e9595 in coro_init /home/shiny/dev/tarantool/third_party/coro/coro.c:108:3

SUMMARY: AddressSanitizer: 4 byte(s) leaked in 1 allocation(s).
```

Closes #10489
Part-of #10211

NO_TEST=covered by existing tests
NO_DOC=bugfix

(cherry picked from commit 84101f60947dc9322b6bb31d2b3c536101c723c7)

Besides mentioned #10485 we also fix a similar memleak (updating user)
that introduced by the same commit 5b32bb7f ("alter: Refactor
access_check outside constructors").

Closes #10485
Part-of #10211

NO_TEST=covered by existing tests
NO_DOC=bugfix

(cherry picked from commit 84f10be00824348844c9e1997bd813b881836928)

Closes #10476
Part-of #10211

NO_TEST=covered by existing tests
NO_DOC=bugfix

(cherry picked from commit a4f4569286c2bc6c237656f63a93b5ff5913ad95)

Closes #10479
Part-of #10211

NO_TEST=covered by existing tests
NO_DOC=bugfix

(cherry picked from commit 3333040462069f064ab0eb01e0ae245e034950a6)

coio_connect_timeout() fallbacks to next address returned by
coio_getaddrinfo() if it cannot connect to the first one. In this case
it fails to free resources related to the first address in function
cleanup.

Closes #10482
Part-of #10211

NO_TEST=covered by existing tests
NO_DOC=bugfix

(cherry picked from commit 00124cea3df72b51b7100d337368db018f542779)

Closes #10480
Part-of #10211

NO_TEST=covered by existing tests
NO_DOC=bugfix

(cherry picked from commit b8d1ae4e8f833d255814f83af1352b698834d610)

If an index is dropped while a dump or compaction task is in progress
we must not write any information about it to the vylog when the task
completes otherwise there's a risk of getting a vylog recovery failure
in case the garbage collector manages to purge the index from the vylog.

We disabled logging on successful completion of a dump task quite a
while ago, in commit 29e2931c ("vinyl: fix race between compaction
and gc of dropped LSM"), and for compaction only recently, in commit
ae6a02eb ("vinyl: do not log dump if index was dropped"), but the
issue remains for a dump/compaction failure, when we log a discard
record for a run file we failed to write. These results in errors like:

```
ER_INVALID_VYLOG_FILE: Invalid VYLOG file: Run 6 deleted twice
```

or

```
ER_INVALID_VYLOG_FILE: Invalid VYLOG file: Run 5768 deleted but not registered
```

Let's fix these issues in exactly the same way as we fixed them for
successful dump/compaction completion - by skipping writing to vylog
in case the index is marked as dropped.

Closes #10452

NO_DOC=bug fix

(cherry picked from commit de59504c2bdb0369cdd27af892301f8515293fe1)