Excluded tuples actually have their own history chains in MVCC - such
chains consist of only one `memtx_story` containing excluded tuple
itself. Such chains should be skipped when counting invisible tuples
because they are not inserted to the index - that's what the commit
does.

Closes #10396

NO_DOC=bugfix

(cherry picked from commit 8947cb04f59423e2944d48b8a1effec2fb11b1db)

Currently, when starting an iterator in memtx tree on a range request,
we pass key from `start_data` to memtx MVCC. The problem is `start_data`
can contain pagination key that is extracted with `cmp_def`, but MVCC
performs all comparisons with `key_def`. Fortunately, first parts of
`cmp_def` is actually `key_def` of the index, so let's crop `start_data`
by passing `part_count` not greater than `key_def->part_count` to MVCC.

Closes #10448

NO_DOC=bugfix

(cherry picked from commit 0dca0076c0fdaee142020cdeddb031bc0e2238cb)

If the iterator type is EQ/REQ/LE/GE and the search key is exact (that
is, there may be at most one tuple matching the key in the index),
there's no need to scan disk levels if we found a statement for this
key in the memory level. We've had this optimization for ages but it
worked only for full keys in terms `cmp_def` (key definition extended
with primary key parts). Apparently, a lookup in a secondary index
performed by the user wouldn't match these criteria unless the secondary
index explicitly included all primary key parts.

This commit improves on that. Now, we enable the optimization if the
search key is **exact**. We consider a key **exact** if either of the
following conditions is true:

 - The key statement is a tuple (tuple has all key parts).
 - The key statement is a full key in terms of `cmp_def`.
 - The key statement is a full key in terms of `key_def`, it doesn't
   contain nulls, and the index is unique. The check for nulls is
   necessary because even a unique nullable index may contain more than
   one equal key with nulls.

Note, this patch slightly refactors the optimization, adding a few
comments and hopefully making it more understandable. In particular,
we remove the one-result-tuple optimization for exact EQ/REQ from
`vy_read_iterator_advance` and put it in `vy_read_iterator_evaluate_src`
instead. This way the whole optimization resides in one place.

Closes #10442

NO_DOC=bug fix

(cherry picked from commit 850673db5a69df2c7250d174ab15305624b2634a)

The test expects that any DDL operation aborts **all** concurrent
transactions, but since commit f5f061d051dc ("vinyl: do not abort
unrelated transactions on DDL") this isn't exactly true: transactions
that haven't read/written anything aren't aborted. In the test we expect
a transaction that haven't done anything to be aborted by DDL and it
**is** aborted most of them time but for a different reason: it reads
data that are later modified for `box.schema.user.create()` reads
`box.space._user:max()` to generate an id for the new user first. Since
it reads before writing anything, it has the "read-confirmed" isolation
level hence it's aborted by the transaction creating another user
because the latter updates `box.space._user:max()`. However, sometimes
both users are created and the test fails. This happens if the first
transaction manages to commit before the second one reads the `_user`
system space.

To fix the test and make the transaction creating the second user fail
due to DDL, let's add a read of the `_user` system space before putting
it to sleep. Actually, this even makes the test closer to the "original
test from #5998".

Closes #10444

NO_DOC=test fix
NO_CHANGELOG=test fix

(cherry picked from commit 62c051e22109369f9079b5adf4de30e0c53f6ca7)

Clang compiler is linking by default against libc++ rather than
libstdc++, which the code seems to be built against.

The patch adds "-stdlib=libstdc++" that is required by build on
OSS Fuzz and also removes dirty hacks in a build script [2][3]
and allows to switch to a static build in OSS Fuzz.

1. https://libcxx.llvm.org/docs/UsingLibcxx.html
2. https://github.com/google/oss-fuzz/commit/92d4e951c3a10b8b3d67ccd061ddaba71f01a328
3. https://github.com/google/oss-fuzz/commit/7a6315db98f574cc8beb4de7f9d2e71c6c589a34
4. https://github.com/google/oss-fuzz/pull/12322

NO_CHANGELOG=testing
NO_DOC=testing

(cherry picked from commit 79737d9d13d9e473687192be76fe055658b6e0b8)

This patch fixes a bug found by the ASAN instrumentation of LuaJIT
allocator [1]. The problem is using `memcpy` beyond the size of the
buffer being copied.

Failing tests:
  - ./test/sql-luatest/gh_10243_varbinary_bound_variable_test.lua

[1]: Issue #10231

Closes #10398

NO_DOC=bugfix
NO_CHANGELOG=bugfix
NO_TEST=rely on existing test (run failing tests with tarantool
build described in [1]).

(cherry picked from commit 1923078b4f65649f03fdd6f789f29ba635b836b7)

NO_DOC=tools
NO_TEST=tools
NO_CHANGELOG=tools

Also, remove unreleased/ entries.

NO_DOC=changelog
NO_TEST=changelog
NO_CHANGELOG=changelog

This commit introduces engine stubs that enable a new method
of fetching snapshots for anonymous replicas. Instead of using
the traditional read-view join approach, this update allows
file snapshot fetching. Note that file snapshot fetching
is only available in Tarantool EE.

Checkpoint fetching is done via IPROTO_IS_CHECKPOINT_JOIN,
IPROTO_CHECKPOINT_VCLOCK and IPROTO_CHECKPOINT_LSN fields.

If IPROTO_CHECKPOINT_JOIN is set to true, join will be done from
files: .snap for memtx, .run for vinyl, if false - from read view.

Checkpoint join allows to continue from the place, where client
stopped in case of snapshot fetching error. This allows to avoid
rebootstrap of an anonymous client. This can be done by specifying
CHECKPOINT_VCLOCK, which says from which file server should continue
join, client gets vclock at the beginning of the join. Specifying
CHECKPOINT_LSN allows to continue from some position in checkpoint.
Server sends all data >= CHECKPOINT_LSN.

If CHECKPOINT_VCLOCK is not specified, fetching is done from the latest
available checkpoint. If CHECKPOINT_LSN is not specified - start from
the beginning of the snap. So, specifying only IS_CHECKPOINT_JOIN
triggers fetching the latest checkpoint from files.

Needed for tarantool/tarantool-ee#741

NO_DOC=ee
NO_TEST=ee
NO_CHANGELOG=ee

(cherry picked from commit 2fca5c13)

This commit makes engine to send vclock without ignoring 0th component
during join, which is needed for checkpoint FETCH SNAPSHOT.

Currently engine join functions are invoked only from
relay_initial_join, which is done during JOIN or FETCH SNAPSHOT.
They respond with vclock of the read view we're going to send.

In the following commit checkpoint FETCH SNAPSHOT will be introduced,
which responds with vclock of the checkpoint, we're going to send.
Such vclock may include 0th component and it's crucial to send it to
a client, as in case of connection failure, client will send us the
same vclock and we'll have to use its signature to figure out, which
checkpoint client wants.

So, we have to send and receive 0th component of the vclock during
FETCH_SNAPSHOT. This commit also introduces decoding vclocks without
ignoring 0th component, as they'll be used in the following commit too.

Needed for tarantool/tarantool-ee#741

NO_DOC=internal
NO_TEST=ee
NO_CHANGELOG=internal

(cherry picked from commit 56058393)

This commit renames xrow_encode_vlock to xrow_encode_vclock_ignore0
since the next commit will introduce encoding vclock without ignoring
0th component, which is needed during sending the response to fetch
snapshot request.

This commit also removes internal field inside the replication_request
structure, as the following commit will use 'vclock' for
encoding/decoding vclock without ignoring component.

Needed for tarantool/tarantool-ee#741

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

(cherry picked from commit 313bd730)

From now on during initial join memtx engine prepares vclock, raft and
limbo states, it also sends them during memtx_engine_join.

It's done in order to simplify the code of initial join, as in the
consequent commit checkpoint initial join will be introduced and we want
relay code to handle it the same as read-view join without confusing
conditions.

Needed for tarantool/tarantool-ee#741

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

(cherry picked from commit 72cc2b3e)

Before this commit raft and limbo states were written at the end of the
checkpoint, which makes it very costly to access them.

Checkpoint join needs to access limbo and raft state in order to send
them during JOIN_META stage. We cannot use the latest states, like it's
done for read-view snapshot fetching: states may be far ahead of the
data, written to the checkpoint, which we're going to send.

This commit moves raft and limbo states after data from the system
spaces but before user data. We cannot put them right at the beginning
of the snapshot, because then we'll have to patch recovery process,
which currently strongly relies on the fact, that system spaces are
at the beginning of the snapshot (this was done in order to apply force
recovery only for user data). If we patch recovery process, then old
versions, where it's unpatched, won't be able to recover from the
snapshots done by the newer version, compatibility of snapshots will be
broken.

The current change is not breaking, old Tarantool versions can restore
from the snapshot made by the newer one.

Needed for tarantool/tarantool-ee#741

NO_DOC=internal
NO_CHANGELOG=internal

(cherry picked from commit 3da31b83)

Starting with commit f1c2127d ("replication: add META stage to JOIN")
replication master appends a special section, called IPROTO_JOIN_META to
the initial snapshot sent to the replica. This section contains the
latest raft term and synchronous transaction queue owner and term.

The section is only sent to nodes, which have a non-zero version_id.
For some reason, version_id encoding for FETCH_SNAPSHOT (analog of JOIN
for anonymous replicas) wasn't added in that commit, so anonymous
replicas do not receive synchronous queue state.

This leads to them raising ER_SPLIT_BRAIN errors later after join, when
the first synchronous row arrives.

In order to fix this, start encoding version_id in FETCH_SNAPSHOT
requests.

Closes #9401

NO_DOC=cherry-pick

(cherry picked from commit 58c006c9)

box_info_lsn
box_is_ro
box_ro_reason
box_wait_ro

Closes #10378

NO_CHANGELOG=minor change

@TarantoolBot document
Title: document 4 more symbols in public C API

box_info_lsn
box_is_ro
box_ro_reason
box_wait_ro

Their meaning is identical to lua methods:

box.info.lsn
box.info.ro
box.info.ro_reason
box.wait_ro

(cherry picked from commit f1c76b976077e8078c42990546d9dbbdea315425)

* FFI: Treat cdata finalizer table as a GC root.
* FFI: Turn FFI finalizer table into a proper GC root.

Part of #10199

NO_DOC=LuaJIT submodule bump
NO_TEST=LuaJIT submodule bump

`datetime` module has a function `:totable()` that converts
the information from a datetime object into the table format.
The field `timestamp` can be accessed in a datetime object:

```
tarantool> dt = datetime.new({timestamp=100})
---
...

tarantool> dt.timestamp
---
- 100
...
```

However, the timestamp is missed in a table produced by
`:totable()` method. The patch fixes that.

Fixes #10374

@TarantoolBot document
Title: Support of timestamp field in :totable()

The field `timestamp` is available in a table produced by
`:totable`:

```
tarantool> datetime.new({timestamp=100}):totable()
---
- timestamp: 100
  tz:
  sec: 40
  min: 1
  yday: 1
  day: 1
  nsec: 0
  isdst: false
  wday: 5
  tzoffset: 0
  month: 1
  year: 1970
  hour: 0
...

tarantool>
```

(cherry picked from commit f66020b670296642547cbfe0a34ab6c11d69989d)

There is no requirement for using fractional numbers
in datetime constructors. This issue will be addressed in [1].
The patch removes testcase with fractional numbers in datetime
constructor.

1. https://github.com/tarantool/tarantool/issues/10391

See also #10391
Needed for #10374

NO_CHANGELOG=testing
NO_DOC=testing

(cherry picked from commit 989c63c75f5f8c7b38dc67f8e5c32ce8bef1da08)

The commit 3c403661 ("datetime, lua: date parsing functions")
introduced a function `datetime.parse()`. The function accepts
a datetime format: 'iso8601', 'rfc3339' and a custom datetime
format. However, 'iso8601' as well as 'rfc3339' is a set of
datetime formats (see [1] and [2]), and `datetime.parse()`
supports a subset of these formats. There is a documentation
issue [3], where we should describe what formats are supported
and what formats are not.

The patch adds a tests for most of supported formats.
The tests were created using a page about date and time formats
in ISO 8601 and RFC 3339 standards [2] and these tests are not
exhaustive. More thorough testing can be done with property-based
testing and randomization. It will be a subject of the following
commits.

1. https://datatracker.ietf.org/doc/html/rfc3339
2. https://ijmacd.github.io/rfc3339-iso8601/
3. https://github.com/tarantool/doc/issues/3155



Follows up #6731
Needed for tarantool/doc#3155

Co-authored-by: Sergey Kaplun <skaplun@tarantool.org>

NO_CHANGELOG=testing
NO_DOC=tarantool/doc#3155

(cherry picked from commit dcb196366621db4ab39ef97029ee126d836b77fb)

`datetime` module has a function `:totable()` that converts
the information from a datetime object into the table format.
The commit 43e10ed3 ("build, lua: built-in module datetime")
added `tzoffset` field to the datetime object and to table
produced by `:totable()`. The commit 9ee45289
("datetime: datetime.TZ array") added fields `tz` and `tzindex` to
the datetime object, but not to the table produced by
`:totable()`. The patch fixes that. Note, `tzindex` is not added,
because it is an internal field.

```
tarantool> datetime.parse('2004-12-01T00:00 Europe/Moscow'):totable()
---
- tz: Europe/Moscow
  sec: 0
  min: 0
  yday: 336
  day: 1
  nsec: 0
  isdst: false
  wday: 4
  tzoffset: 180
  month: 12
  year: 2004
  hour: 0
...
```

Fixes #10331
Follows up #6751

@TarantoolBot document
Title: Support of tz field in :totable()

In addition to the `tzoffset` in a table produced by `:totable`
we added `tz` field.

```
tarantool> datetime.parse('2004-12-01T00:00 Europe/Moscow'):totable()
---
- tz: Europe/Moscow
  sec: 0
  min: 0
  yday: 336
  day: 1
  nsec: 0
  isdst: false
  wday: 4
  tzoffset: 180
  month: 12
  year: 2004
  hour: 0
...
```

(cherry picked from commit 90552e55e0921405c43ea086ae418a72c9f000e4)

This patch fixes a bug found by the ASAN instrumentation of LuaJIT
allocator [1]. The problem is using a Lua serializer object that has
been cleaned up by GC.

The crash occurs when executing `tarantool> \set output lua`.

Failing tests:
 - ./test/app-luatest/gh_7031_configure_eos_in_lua_console_test.lua
 - ./test/app-tap/console.test.lua
 - ./test/box/push.test.lua
 - ./app-tap/console_lua.test.lua
 - ./app-luatest/varbinary_test.lua

The `serializer_lua` static member is removed from
`src/box/lua/serialize_lua.c` along with `serializer_lua_init`,
because it is not used by any function other than `serializer_lua_init`,
which is not needed now too.

[1]: Issue #10231

Closes #10177 (this issue is a duplicate of #7404)

NO_TEST=rely on existing tests
NO_CHANGELOG=codehealth
NO_DOC=codehealth

(cherry picked from commit 830f27bbaf94dadcacc66a7f543e0e5feec3988d)

To ensure better encapsulation, maintainability, and portability of
the code, it is necessary to replace direct access to the fields of
global structures with calls using the Lua C API.

Closes #10284

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

(cherry picked from commit f7bb3fc7a1222107753dc420675ecda043b0a5d2)

Since commit 8f4be322 ("txm: disallow yields after DDL operation in
TX"), any DDL operation aborts **all** active transactions, even those
that wouldn't be affected by it anyway, see `memtx_engine_prepare()`,
`memtx_tx_abort_all_for_ddl()`. Actually, there's no need to do that in
Vinyl because it properly handles concurrent DDL operations, see commit
d3e12369 ("vinyl: abort affected transactions when space is removed
from cache"). Let's skip Vinyl transactions from consideration by
marking the Vinyl engine with a special flag.

Closes #10375

NO_DOC=bug fix

(cherry picked from commit f5f061d051dc6268949bfcb141d211142282578d)

CMake 3.29.0 produces a warning on configuration stage:

NO_WRAP
 | CMake Warning (dev) in cmake/ProtobufMutator.cmake:
 |  A logical block opening on the line
 |    /home/sergeyb/sources/MRG/tarantool/cmake/ProtobufMutator.cmake:38 (if)
 |   closes on the line
 |    /home/sergeyb/sources/MRG/tarantool/cmake/ProtobufMutator.cmake:40 (endif)
 |  with mis-matching arguments.
NO_WRAP

The patch fixes the warning.

NO_CHANGELOG=build
NO_DOC=build
NO_TEST=build

(cherry picked from commit 1e9e70f4c55c74c94922cb0beef44ae2b82255a9)

Protocol Buffers library has a name `protobufd` when CMAKE_BUILD_TYPE is
equal to "Debug". In other cases the name is `protobuf`. The patch fixes
the library name.

Follows up commit b11072a6 ("cmake: add dependencies for LuaJIT and
SQL fuzzers").

Follows up #4823

NO_CHANGELOG=build
NO_DOC=build
NO_TEST=build

(cherry picked from commit c0d8cb54)

An attempt to build Tarantool statically by Ninja (with the
enabled CMake option BUILD_STATIC_WITH_BUNDLED_LIBS), error below
is produced:

 | ninja: error: build.ninja:1405: bad $-escape
 | (literal $ must be written as $$)

The commit fixes that error. Follows up commit c92a1699
("cmake: support build using Ninja").

NO_CHANGELOG=build
NO_DOC=build
NO_TEST=build

(cherry picked from commit d48c40e69495b8cc3475ea0b184768054d87b780)

CMake 3.24+ produces a warning about the missed option
DOWNLOAD_EXTRACT_TIMESTAMP in `ExternalProject_Add()` [1]:

 | The DOWNLOAD_EXTRACT_TIMESTAMP option was not given and policy
 | CMP0135 is not set.

Documentation about policy CMP0135 [2] said:

CMake 3.23 and below set the timestamps of the extracted contents
to the same as the timestamps in the archive. When the URL
changes, the new archive is downloaded and extracted, but the
timestamps of the extracted contents might not be newer than the
previous contents. Anything that depends on the extracted contents
might not be rebuilt, even though the contents may change.

CMake 3.24 and above prefer to set the timestamps of all
extracted contents to the time of the extraction. This ensures
that anything that depends on the extracted contents will be
rebuilt whenever the URL changes.

1. https://cmake.org/cmake/help/latest/module/ExternalProject.html#url
2. https://cmake.org/cmake/help/latest/policy/CMP0135.html

NO_CHANGELOG=build
NO_DOC=build
NO_TEST=build

(cherry picked from commit 4c0e08d85cd39916818bc4b915ac3acdacce8fe4)

datetime object can be created using `datetime.new()` and
`datetime.parse()`. `datetime.new()` sets values of unspecified
fields to appropriate values of fields in a Unix time
(00:00:00 UTC on 1 January 1971, Thursday), see commit
43e10ed3 ("build, lua: built-in module datetime"):

```
tarantool> datetime.new()
---
- 1970-01-01T00:00:00Z
...

tarantool> datetime.new():totable()
---
- sec: 0
  min: 0
  yday: 1
  day: 1
  nsec: 0
  isdst: false
  wday: 5
  tzoffset: 0
  month: 1
  year: 1970
  hour: 0
```

The function `datetime.parse` converts an input string with the
date and time information into a datetime object. When fields are
not specified in a input string their values becomes undefined:

```
tarantool> dt.parse('01:01:01', {format ='%H:%M:%S'}):totable()
---
- sec: -59
  min: -58
  yday: 366
  day: 31
  nsec: 0
  isdst: false
  wday: 1
  tzoffset: 0
  month: 12
  year: 0
  hour: -22
...

tarantool>
```

The commit fixes aforementioned behaviour of `datetime.parse()`
by setting values for fields not specified by user to values of
appropriate fields of Unix time.

NO_DOC=bugfix

Fixes #8588

(cherry picked from commit 9ac56a1222e916951010091fbdbfc497f4a4971d)

The function `datetime_gmtoff()` has been introduced in commit
aec6fbac ("datetime: implement date.isdst") for a function
`datetime_increment_by()`. In commit 6ca07285
("datetime: fix interval arithmetic for DST") the function
`datetime_increment_by()` has been updated and call of the
function `datetime_gmtoff()` was removed. `datetime_gmtoff` is
dead code now. The patch removes `datetime_gmtoff()`.

NO_CHANGELOG=codehealth
NO_DOC=codehealth
NO_TEST=codehealth

(cherry picked from commit 625afe418151de1015483ed24acaa57e1f85efb3)

The patch fixes a number of typos in datetime source code.

NO_CHANGELOG=codehealth
NO_DOC=codehealth
NO_TEST=codehealth

(cherry picked from commit c9c5b9f139ccbf5372d1568827fbb50bec7239bb)

Since tarantool/luajit@a16313f large exponent double strings are not
considered convertible to number. It broke encoding lua objects to YAML
because single quotes weren't considered necessary for decoding.

This commit adds wrapping of every string containing infinite double values
into a single quotes.

Closes #10164

NO_DOC=bug fix

(cherry picked from commit 7c3f42590240525d2e543305b6c289ddb30054a2)

This patch adds encapsulation for IPv6 addresses in brackets when
calling uri.format (as per RFC 2732).

Closes #9556

NO_DOC=bugfix

(cherry picked from commit a49ec23b85edb684744eb525427465dfa4f660e1)

Before this patch `uri.parse(<uri-string-with-ipv6-addres>)`
did not work correctly. In particular, it did not parse an
IPv6 address if it contained `A-F`. It is a regression caused
by commit 1376aad9 ("Refactor src/uri.rl to support RFC3986
and add Lua bindings").

This patch fixes a bug where characters `A-F` are not
supported in IPv6.

Part of #9556

NO_DOC=bugfix

(cherry picked from commit 2eefc56a3e08a60f0b71e33621be851794131546)

The motivation is to exclude the file from diffs on GitHub's web
interface [1] and to exclude it from checkpatch checks [2].

[1]: https://docs.github.com/en/repositories/working-with-files/managing-files/customizing-how-changed-files-appear-on-github
[2]: https://github.com/tarantool/checkpatch/pull/75

NO_DOC=the patch is about the development process, it doesn't change
       anything in the shipped product
NO_CHANGELOG=see NO_DOC
NO_TEST=see NO_DOC

(cherry picked from commit fbea9862b087e161a48dc0ac181a1da06cc0ce9d)

When checkpoint fails, we abort it in all engines even if it wasn't
started successfully. If it fails right from the start so that checkpoint
in memtx wasn't started, assertion in `memtx_engine_abort_checkpoint`
fails - memtx doesn't expect that checkpoint will be aborted if it
failed to start. Let's do the same thing as vinyl does - no-op if
there is no checkpoint in progress.

Closes #10265

NO_CHANGELOG=reproducible only with error injection
NO_DOC=bugfix

(cherry picked from commit 6b484622259c01a2468b1f248dd6f1bcdc227021)

We've had a number of issues when Tarantool was permanently broken
(unable to recover after restart) because of a bad vylog record.
The `force_recovery` mode didn't help so the user would have no other
choice but to rebootstrap. A funny thing is those bugs were usually
caused by a race between the garbage collector and dump/compaction when
a vylog record was written for a dropped index. The worst thing that
could happen if we ignored such a bad record is an unused run file not
deleted from disk. Apparently, this is better than a permanent recovery
failure so let's support the `force_recovery` mode in vylog.

The tricky part here is handling checkpoint after restart. The problem
is that to create a vylog checkpoint, we load the previous vylog file
so we have to ignore errors if it was loaded in the `force_recovery`
mode.

Closes #10292

NO_DOC=bug fix

(cherry picked from commit c68e8a8e029d849d68c6018ed00b5a79cc769222)

The vinyl metadata log processor allocates its internal objects either
from malloc or region, neither of which should fail for small
allocations. Let's switch to xalloc to simplify the code. A good thing
about this change is that now we can ignore all errors raised by
vy_log_record_decode() and vy_recovery_process_record() if the
force_recovery flag is set (see the next commit).

Needed for #10292

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

(cherry picked from commit b6f015e98b7c30861dc33032ba7eca47de0cc198)

Temporary allocations from a region don't fail so let's use xalloc to
simplify the code.

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

(cherry picked from commit 28d51f8076b390c50d5b18f8e767c1eb540e5dcc)

Let's log the new value when an error injection is set in orer to ease
debugging in tests.

NO_DOC=logging
NO_TEST=logging
NO_CHANGELOG=logging

(cherry picked from commit 019bacbe)