Changelog: https://curl.se/changes.html#7_84_0

New release contains fixes for 4 security problems [1].

Patch adds a new option defined in curl build infrastructure with it's
default value used in 7.84.0. NOTE: PSL is a Public Suffix List (PSL),
it is a list of suffixes used for cookies.

1. https://curl.se/docs/releases.html

NO_DOC=libcurl submodule bump
NO_TEST=libcurl submodule bump

This patch fixes a bug where the ANY field type was replaced by the
SCALAR field type in the ephemeral space used in ORDER BY.

Closes #7345

NO_DOC=bugfix

After this patch, the result type of arithmetic between two unsigned
values will be INTEGER.

Closes #7295

NO_DOC=bugfix

This patch fixes the following error:

    [ 35%] Performing configure step for 'bundled-libcurl-project'
    *** Do not use buildconf. Instead, just use: autoreconf -fi
    ./buildconf: 8: exec: autoreconf: not found

To resolve this issue we need to install the `autoconf` and `automake`
packages before building.

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

This makes the test more real-life, and allows not to bother in the
child process with the memory allocated prior to fork.

Closes #7370

NO_DOC=test fix
NO_CHANGELOG=test fix

coio_fill_addrinfo allocates ai_local->ai_addr, which should be freed
in case of error.

Part of #7370

NO_DOC=bugfix
NO_TEST=memory leak
NO_CHANGELOG=minor bug

Since "x64/LJ_GC64: Fix fallback case of asm_fuseloadk64()."
(42853793ec3e6e36bc0f7dff9d483d64ba0d8d28) is backported into
tarantool/luajit trunk, box/bitset.test.lua and box/function1.test.lua
tests are no more fragile.

Follows up tarantool/tarantool-qa#234
Follows up tarantool/tarantool-qa#235

NO_DOC=test changes
NO_CHANGELOG=test changes
NO_TEST=test changes

This patch fixes `box-py/args.test.py` test and allows it to work
against tarantool installed from a package.

Closes tarantool/tarantool-qa#246

NO_DOC=testing stuff
NO_TEST=testing stuff
NO_CHANGELOG=testing stuff

This patch fixes `app-tap/tarantoolctl.test.lua` test and allows it to
work against tarantool installed from a package.

Part of tarantool/tarantool-qa#246

NO_DOC=testing stuff
NO_TEST=testing stuff
NO_CHANGELOG=testing stuff

This patch fixes `gh-1700-abort-recording-on-fiber-switch.test.lua` test
and allows it to work against tarantool installed from a package.

Part of tarantool/tarantool-qa#246

NO_DOC=testing stuff
NO_TEST=testing stuff
NO_CHANGELOG=testing stuff

This patch adds the new `make` test targets to run unit and functional
tests independending on each other. In some cases it can be useful.

New test targets:

* `test-unit` - run unit tests and exit after the first failure
* `test-unit-force` - run unit tests
* `test-func` - run functional tests and exit after the first failure
* `test-func-force` - run functional tests

Note, tests for 'small' lib are considered as unit tests as well.

Part of tarantool/tarantool-qa#246

NO_DOC=testing stuff
NO_TEST=testing stuff
NO_CHANGELOG=testing stuff

This patch fixes the following issue:

    Installing collected packages: greenlet, gevent
      Attempting uninstall: greenlet
        Found existing installation: greenlet 0.4.15
    ERROR: Cannot uninstall 'greenlet'. It is a distutils installed
        project and thus we cannot accurately determine which files
        belong to it which would lead to only a partial uninstall.
    make: *** [.travis.mk:86: deps_tests] Error 1
    Error: Process completed with exit code 2.

This happens because the `python3-gevent` package is installed via the
`apt` package manager in some test targets. Let's install test-run deps
via `pip` everywhere.

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

If readline 'show-mode-in-prompt' is on then test fails because it does
not handle prefix added to prompt in this mode. Let's use default
(compiled in) readline configuration instead of the one provided by
user or system config.

NO_DOC=test changes
NO_CHANGELOG=test changes
NO_TEST=test changes

Current implementation of tracking statements that delete a story has a
flaw, consider the following example:

tx1('box.space.s:replace{0, 0}') -- statement 1

tx2('box.space.s:replace{0, 1}') -- statement 2
tx2('box.space.s:delete{0}') -- statement 3
tx2('box.space.s:replace{0, 2}') -- statement 4

When statement 1 is prepared, both statements 2 and 4 will be linked to the
delete statement list of {0, 0}'s story, though, apparently, statement 4
does not delete {0, 0}.

Let us notice the following: statement 4 is "pure" in the sense that, in
the transaction's scope, it is guaranteed not to replace any tuple — we
can retrieve this information when we check where the insert statement
violates replacement rules, use it to determine "pure" insert statements,
and skip them later on when, during preparation of insert statements, we
handle other insert statements which assume they do not replace anything
(i.e., have no visible old tuple).

On the contrary, statements 1 and 2 are "dirty": they assume that they
replaced nothing (i.e., there was no visible tuple in the index) — when one
of them gets prepared — the other one needs to be either aborted or
relinked to replace the prepared tuple.

We also need to fix relinking of delete statements from the older story
(in terms of the history chain) to the new one during preparation of insert
statements: a statement needs to be relinked iff it comes from a different
transaction (to be precise, there must, actually, be no more than one
delete statement from the same transaction).

Additionally, add assertions to verify the invariant that the story's
add (delete) psn is equal to the psn of the add (delete) statement's
transaction psn.

Closes #7214
Closes #7217

NO_DOC=bugfix

Normally, if a server created by a test isn't stopped it should be
forcefully killed by luatest or test-run. For some reason, it doesn't
happen sometimes, which may lead to the next test failing to bind,
because all test servers that belong to the same luatest suite and have
the same alias share the same socket path (although they use different
directories). This looks like a test-run or luatest bug.

The vinyl-luatest/update_optimize test doesn't stop the test server
so because of this test-run/luatest bug, the next vinyl-luatest test
fails occasionally:

NO_WRAP
[001] vinyl-luatest/update_optimize_test.lua                          [ pass ]
[001] vinyl-luatest/gh_6568_replica_initial_join_rem>                 [ fail ]
[001] Test failed! Output from reject file /tmp/t/rejects/vinyl-luatest/gh_6568_replica_initial_join_removal_of_compacted_run_files.reject:
[001] TAP version 13
[001] 1..1
[001] # Started on Tue Jul  5 13:30:37 2022
[001] # Starting group: gh-6568-replica-initial-join-removal-of-compacted-run-files
[001] master | 2022-07-05 13:30:37.530 [189564] main/103/default.lua F> can't initialize storage: unlink, called on fd 25, aka unix/:(socket), peer of unix/:(socket): Address already in use
[001] ok     1  gh-6568-replica-initial-join-removal-of-compacted-run-files.test_replication_compaction_cleanup
[001] not ok 1  gh-6568-replica-initial-join-removal-of-compacted-run-files.test_replication_compaction_cleanup
[001] #   Failure in after_all hook: /home/vlad/.rocks/share/tarantool/luatest/process.lua:100: kill failed: 256
[001] #   stack traceback:
[001] #         .../src/tarantool/tarantool/test/luatest_helpers/server.lua:206: in function 'stop'
[001] #         ...src/tarantool/tarantool/test/luatest_helpers/cluster.lua:44: in function 'drop'
[001] #         ...ica_initial_join_removal_of_compacted_run_files_test.lua:34: in function <...ica_initial_join_removal_of_compacted_run_files_test.lua:33>
[001] #         ...
[001] #         [C]: in function 'xpcall'
[001] # Ran 1 tests in 1.682 seconds, 0 succeeded, 1 errored
NO_WRAP

Let's fix this by stopping the test server started by the
vinyl-luatest/update_optimize test.

NO_DOC=test
NO_CHANGELOG=test

The idea behind the new test is the same as the one used by
vinyl/select_consistency.test.lua: create a space with a few
compound secondary indexes that share the first part, then run
SELECT requests under heavy write load and check that results
match. However, in comparison to its predecessor, the new test
has a few improvements:

 1. It generates DML requests in multi-statement transactions.
 2. It checks non-unique indexes.
 3. It checks multikey indexes.
 4. It triggers L0 dumps not by box.snapshot, but by exceeding
    the box.cfg.vinyl_memory limit.
 5. It starts 20 write and 5 read fibers.
 6. It reruns the test after restart to check that recovery works fine.
 7. It checks that there's no phantom statements stored in
    the space indexes after the test.
 8. It runs the test with deferred DELETEs enabled and disabled.
    (see box.cfg.vinyl_defer_deletes).
 9. It is written in luatest.

The test takes about 20 seconds to finish so it's marked as long run.

Closes #4251

NO_DOC=test
NO_CHANGELOG=test

Currently it throws an error when encounter binary data, print
<binary> tag instead.

Closes #7040

NO_DOC=bugfix

Problem with W214 in src/box/lua/net_box.lua was fixed in previous
commit. Can bump luacheck version now.

NO_DOC=testing
NO_TEST=testing
NO_CHANGELOG=testing

Since 0.26.0 luacheck emits a warning on the `_box` variable.
From luacheck v.0.26.0 release notes:

"Function arguments that start with a single underscore
get an "unused hint". Leaving them unused doesn't result
in a warning. Using them, on the other hand, is a
new warning (№ 214)."

Renamed `_box` to `__box`, which isn't considered unused.

Closes #7304.

NO_DOC=testing
NO_TEST=testing
NO_CHANGELOG=testing

We must not throw exceptions from C code.

Currently, there's the only C function that uses diag_raise() - it's
space_cache_find_xc. We move it under ifdef(__cplusplus).

Follow-up #4735

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

current_session() is called from C code so it must not throw, but it may
if it fails to allocate a session. Practically, this is hardly possible,
because we don't limit the runtime arena, which is used for allocation
of session objects. Still, this looks potentially dangerous.

Gracefully handling an allocation failure in all places where
current_session() may be called would be complicated. Since it's more of
a theoretical issue, let's panic on a session allocation error, like we
do if we fail to allocate other mission critical system objects.

Closes #4735

NO_DOC=code health
NO_TEST=code health
NO_CHANGELOG=code health

The functions allocate and free a session so they should be called
new/delete, not create/destroy accroding to our naming convention.
While we are at it, also delete obsoleve comments to these functions:
they don't invoke session triggers.

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

C++ features are not used in this file.

Note, we need to move ifdef(__cplusplus) in user.h to make guest_user
and admin_user variables accessible from C code.

Also, we need to move initialization of session_vtab_registry to
session_init(), because most C compilers don't allow to initialize
a global variable with a value of another global variable.

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Our txn_limbo_is_replica_outdated check works correctly only when there
is a stream of PROMOTE requests. Only the author of the latest PROMOTE
is writable and may issue transactions. No matter synchronous or
asynchronous.

So txn_limbo_is_replica_outdated assumes that everyone but the node with
the greatest PROMOTE/DEMOTE term is outdated.

This isn't true for DEMOTE requests. There is only one server which
issues the DEMOTE request, but once it's written, it's fine to accept
asynchronous transactions from everyone.

Now the check is too strict. Every time there is an asynchronous
transaction from someone, who isn't the author of the latest PROMOTE or
DEMOTE, replication is broken with ER_SPLIT_BRAIN.

Let's relax it: when limbo owner is 0, it's fine to accept asynchronous
transactions from everyone, no matter the term of their latest PROMOTE
and DEMOTE.

This means that now after a DEMOTE we will miss one case of true
split-brain: when old leader continues writing data in an obsolete term,
and the new leader first issues PROMOTE and then DEMOTE.

This is a tradeoff for making async master-master work after DEMOTE.

The completely correct fix would be to write the term the transaction
was written in with each transaction and replace
txn_limbo_is_replica_outdated with txn_limbo_is_request_outdated, so
that we decide whether to filter the request or not judging by the term
it was applied in, not by the term we seen in some past PROMOTE from the
node. This fix seems too costy though, given that we only miss one case
of split-brain at the moment when the user enables master-master
replication (by writing a DEMOTE). And in master-master there is no such
thing as a split-brain.

Follow-up #5295
Closes #7286

NO_DOC=internal chcange

Currently there's only one place where applier_synchro_filter_tx
accesses limbo state under a latch: this place is
txn_limbo_is_replica_outdated. Soon there will be more accesses to limbo
parameters and all of them should be guarded as well.

Let's simplify things a bit and guard the whole synchro_filter_tx with
the limbo latch.

While we are at it remove txn_limbo_is_replica_outdated as not needed
anymore.

Part-of #7286

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Starting with commit deca9749
("replication: unify replication filtering with and without elections")
The filter works always, even when elections are turned off.

Reflect that in the comments for applier_synchro_filter_tx and
txn_limbo_is_replica_outdated.

Follow-up #6133

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

The 'small' lib test suite was not run for out-of-source builds since
the wrong symlink was created for test binaries and test-run couldn't
find them. Now it is fixed.

When test-run loads tests, first, it searches the suite.ini file and if
it exists test-run consider the dir as a test suite. So there was sense
to create a permanent link for 'small' lib tests.

Closes #4485

NO_DOC=testing stuff
NO_TEST=testing stuff
NO_CHANGELOG=testing stuff

Disable tests while building packages in the reusable_build.yml workflow
to speed up the build process in integration testing for tarantool and
modules/connectors.

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

Sometimes we need to disable testing while building deb/rpm packages to
speed up the build process. Now it is possible via `MAKE_CHECK` env var.
By default, testing is on, but if one defines `MAKE_CHECK=false`, tests
will be off.

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

This patch adds the tzdata package as a dependency for DEB/RPM tarantool
package since some tarantool datetime functionality needs this.

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

Now the test-run dependencies (pyyaml, gevent) have the corresponding
deb packages installable via the 'apt' package manager and finally it's
time to enable running tests in the package build process.

Closes #1341

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

Vinyl doesn't support the hot standby mode. There's a ticket to
implement it, see #2013. The behavior is undefined if running an
instance in the hot standby mode in case the master has Vinyl spaces.
It may result in a crash or even data corruption.

Let's raise an explicit error in this case.

Closes #6565

NO_DOC=bug fix

Since commit d2537d9d ("relay: cleanup error handling")
recover_remaining_wals() doesn't log the error it throws -
now callers of this function should catch and log the error.
hot_standby_f() doesn't catch the error so the diagnostic
message is lost if we fail to apply a row in the hot standby
mode. Fix this.

NO_DOC=bug fix
NO_TEST=checked in next commit
NO_CHANGELOG=minor bug in logging

If a nested tuple field is indexed, it can be accessed by [*] aka
multikey or any token:

  s = box.schema.create_space('test')
  s:create_index('pk')
  s:create_index('sk', {parts = {{2, 'unsigned', path = '[1][1]'}}})
  t = s:replace{1, {{1}}}
  t['[2][1][*]'] -- returns 1!

If a nested field isn't indexed (remove creation of the secondary index
in the example above), then access by [*] returns nil.

Call graph:

  lbox_tuple_field_by_path:
    tuple_field_raw_by_full_path
      tuple_field_raw_by_path
        tuple_format_field_by_path
          json_tree_lookup_entry
            json_tree_lookup

And json_tree_lookup matches the first node if the key is [*].
We shouldn't match anything to [*].

Closes #5226

NO_DOC=bug fix

__gcov_flush was removed in gcc11.

Since gcc11 __gcov_dump calls __gcov_lock at the start and
__gcov_unlock before returning. Same is true for __gcov_reset.
Because of that using __gcov_reset right after __gcov_dump since gcc11
is the same as using __gcov_flush before gcc11.

Closes #7302

NO_CHANGELOG=internal
NO_DOC=internal
NO_TEST=internal

Covered most of box_promote and box_demote with tests:
1. Promote/demote unconfigured box
2. Promoting current leader with elections on and off
3. Demoting follower with elections on and off
4. Promoting current leader, but not limbo owner with elections on
5. Demoting current leader with elections on and off
6. Simultaneous promote/demote
7. Promoting voter
8. Interfering promote/demote while writing new term to wal
9. Interfering promote/demote while waiting for synchro queue
   to be emptied
10. Interfering promote while waiting for limbo to be acked
    (similar to replication/gh-5430-qsync-promote-crash.test.lua)

Closes #6033

NO_DOC=testing stuff
NO_CHANGELOG=testing stuff

The test failed with the following output:

 TAP version 13
 1..3
 # Started on Tue Jun 28 13:36:03 2022
 # Starting group: pre-vote
 not ok 1	pre-vote.test_no_direct_connection
 #   .../election_pre_vote_test.lua:46: expected: a value evaluating to
					true, actual: false
 #   stack traceback:
 #   .../election_pre_vote_test.lua:65: in function 'retrying'
 #   .../election_pre_vote_test.lua:64: in function
				    'pre-vote.test_no_direct_connection'
 #   ...
 #   [C]: in function 'xpcall'
 ok     2	pre-vote.test_no_quorum
 ok     3	pre-vote.test_promote_no_quorum
 # Ran 3 tests in 6.994 seconds, 2 succeeded, 1 failed

This is the moment when one of the followers disconnects from
the leader and expects its `box.info.election.leader_idle` to grow.

It wasn't taken into account that this disconnect might lead to leader
resign due to fencing, and then a new leader would emerge and
`leader_idle` would still be small.

IOW, the leader starts with fencing turned off, and only resumes
fencing, once it has connected to a quorum of nodes (one replica in this
test). If the replica that we just connected happens to be the one we
disconnect in the test, the leader might fence, if it hasn't yet
connected to the other replica, because it immediately loses a quorum of
healthy connections right after gaining it for the first time.

Fix this by waiting until everyone follows everyone before each test
case.

The test, of course, could be fixed by turning fencing off, but this
might hide any possible future problems with fencing.

Follow-up #6654
Follow-up #6661

NO_CHANGELOG=test fix
NO_DOC=test fix

After scanning disk, the Vinyl read iterator checks if it should restore
the iterator over the active memory tree, because new tuples could have
been inserted into it while we yielded reading disk. We assume that
existing tuples can't be deleted from the memory tree, but that's not
always true - a tuple may actually be deleted by rollback after a failed
WAL write. Let's reevaluate all scanned sources and reposition the read
iterator to the next statement if this happens.

Initially, the issue was fixed by commit 83462a5c ("vinyl: restart
read iterator in case L0 is changed"), but it introduced a performance
degradation and was reverted (see #5700).

NO_DOC=bug fix
NO_TEST=already there
NO_CHANGELOG=already there

The Vinyl read iterator, which is used for serving range select
requests, works as follows:

 1. Scan in-memory sources. If found an exact match or a chain in
    the cache, return it.
 2. If not found, scan disk sources. This operation may yield.
 3. If any new data was inserted into the active memory tree, go to
    step 1, effectively restarting the iteration from the same key.

Apparently, such an algorithm doesn't guarantee any progress of a read
operation at all - when we yield reading disk on step 2 after a restart,
even newer data may be inserted into the active memory tree, forcing us
to restart again. In other words, in presence of an intensive write
workload, read ops rate may drop down to literally 0.

It hasn't always been like so. Before commit 83462a5c ("vinyl:
restart read iterator in case L0 is changed"), we only restored the
memory tree iterator after a yield, without restarting the whole
procedure. This makes sense, because only memory tree may change after
a yield so there's no point in rescanning other sources, including
disk. By restarting iteration after a yield, the above-mentioned commit
fixed bug #3395: initially we assumed that statements may never be
deleted from a memory tree while actually they can be deleted by
rollback after a failed WAL write.

Let's revert this commit to fix the performance degradation. We will
re-fix bug #3395 in the next commit.

Closes #5700

NO_DOC=bug fix
NO_TEST=should be checked by performance tests

* Avoid conflict between 64 bit lightuserdata and ITERN key.
* Reorganize lightuserdata interning code.
* test: fix path storage for non-concatable objects
* ARM64: Fix assembly of HREFK.
* FFI/ARM64: Fix pass-by-value struct calling conventions.
* test: set DYLD_LIBRARY_PATH environment variable
* x64/LJ_GC64: Fix fallback case of asm_fuseloadk64().
* FFI: Handle zero-fill of struct-of-NYI.
* Fix interaction between profiler hooks and finalizers.
* Flush and close output file after profiling run.
* Fix debug.debug() for non-string errors.
* Fix write barrier for lua_setupvalue() and debug.setupvalue().
* Fix FOLD rule for strength reduction of widening.
* Fix bytecode dump unpatching.
* Fix tonumber("-0") in dual-number mode.
* Fix tonumber("-0").
* Give expected results for negative non-base-10 numbers in tonumber().
* Add missing LJ_MAX_JSLOTS check.
* Add stricter check for print() vs. tostring() shortcut.

Closes #6548
Fixes #4614
Fixes #4630
Fixes #5885
Fixes tarantool/tarantool-qa#234
Fixes tarantool/tarantool-qa#235
Follows up #2712

NO_DOC=LuaJIT submodule bump
NO_TEST=LuaJIT submodule bump