Msgpuck is a submodule for tarantool and is needed while packaging.
Moved function of mp_encode_str0() from mp_error.cc into a common
place is msgpuck.h.

Needed for packaging workflow in tarantool/tarantool:
tarantool/tarantool#6260

NO_DOC=no visible changes
NO_CHANGELOG=no visible changes

The luatest_helpers.lua module confuses developers because luatest
itself has the helpers.lua module as well. It is unclear which one
should be used and when because it is hard to remember what stuff
provides each of them.

Actually, luatest_helpers.lua has functions that can be safely moved to
more proper places (luatest_helpers/server.lua & instances/default.lua).

The `get_vclock` function logically belongs to server stuff, so it was
moved to server.lua and slightly changed (`eval` call was replaced by
`exec`).

The `wait_vclock` function logically belongs to server stuff as well, so
it was moved to server.lua without changes.

The `instance_uri` function logically belongs to server stuff as well,
so it was moved to server.lua and slightly changed (it was renamed into
`build_instance_uri` and redundant `instance_id` param was dropped).

The `default_cfg`, `env_cfg`, `box_cfg` functions logically belongs to
instance stuff, so they were moved to default.lua without changes.

Thus, the luatest_helpers.lua module was dropped and tests using this
module were updated.

Closes tarantool/luatest#192

NO_DOC=testing stuff
NO_CHANGELOG=testing stuff

The luatest_helpers/asserts.lua module confuses developers because
luatest itself has the asserts.lua module as well. It is unclear which
one should be used and when because it is hard to remember what stuff
provides each of them.

Actually, luatest_helpers/asserts.lua has functions that are not really
about asserting and can be safely moved to more proper places like the
luatest_helpers/server.lua and luatest_helpers/cluster.lua modules.

The `assert_server_follow_upstream` function logically belongs to server
stuff, so it was moved to server.lua and slightly changed (renamed into
`assert_follows_upstream` and `eval` calls were replaced by `exec`).

The `wait_fullmesh` function logically belongs to cluster stuff, so it
was moved to cluster.lua and slightly changed (now it accepts one param
that can include wait timeout and delay, otherwise, defaults applied).

Thus, the luatest_helpers/asserts.lua module was dropped and tests using
this module were updated.

Part of tarantool/luatest#192

NO_DOC=testing stuff
NO_CHANGELOG=testing stuff

We have recently dropped 'checksum' machinery in test-run
(tarantool/test-run#321) and now there is no need to keep
checksums in suite.ini config files. So removing them.

Closes tarantool/tarantool-qa#152

* test: adapt tests checking loading bytecode files
* test: adapt disabled tests from PUC-Rio
* test: adapt tests checking traceback in tail call
* test: adapt test checking global environment

Closes #5687
Closes #5691
Closes #5703
Closes #5713
Part of #5870

NO_DOC=LuaJIT submodule bump
NO_CHANGELOG=no visible changes

Bump test-run to new version with the following improvements:

  - drop 'checksum' machinery (tarantool/test-run#316)
  - rerun fragile tests only with status == 'fail'

NO_DOC=testing stuff
NO_CHANGELOG=testing stuff

The function of the question (`tarantoolsqlIdxKeyCompare()`) compares a
tuple with a key. It extracts necessary fields from the tuple and runs
the compare function, which is basically `memcmp()` of two buffers (for
`varbinary` values -- numeric comparisons are different).

In the Debug build the function also re-verifies the result using a
separate comparison code. And here we compare two `memcmp()` return
values.

We shouldn't do it directly, because `memcmp()` only guarantees sign of
the result. Please, consider the linked issue for details.

NO_DOC=only debug build is affected
NO_CHANGELOG=only debug build is affected
NO_TEST=hard to create a stable reproducer

Fixes #6849

The broadcast doesn't need to go into the network, but
it will be needed locally for a built-in event. So
called "box.election".

Part of #6260

NO_DOC=no visible changes
NO_CHANGELOG=no visible changes

In case a leader steps down - e.g. switching to a 'voter' - it will send
appropriate message. This message being received by a follower
will not trigger a broadcast, if follower is a 'voter'.

NO_DOC=no visible changes
NO_CHANGELOG=no visible changes

 1. There's a non-ASCII character in a comment (’). Replase it with a
    plain comma ('). Fixes commit
    19be79e6 ("tools: gen-release-notes:
    adapt for enterprise").
 2. In Python 2 subprocess.Popen() doesn't support the 'with' clause.
    Do not use the 'with' clause. Fixes commit
    5715f194 ("tools: gen-release-notes:
    factor out popen helper")

As a result CI fails. I wonder why it passed on the PR...

Follow-up #6855

NO_DOC=tools
NO_CHANGELOG=tools

We're going to reuse the existing tools/gen-release-notes script for
generation of Enterprise Edition (EE) changelogs. To do that, we need to

 1. Add 'enterprise' to the head of the sections list. The new section
    will be used by EE changelog entries. We want them to go before
    open-source changelog entries.
 2. Gather changelog entries from the 'changelog/unreleased' directory
    of the superproject directory. EE uses the open-source repository as
    its submodule.

NO_DOC=tools
NO_CHANGELOG=tools

So that we can gather changelog entries from more than one directory.

NO_DOC=tools
NO_CHANGELOG=tools

We will use it for running other commands.

NO_DOC=tools
NO_CHANGELOG=tools

Trivial refactoring that replaces source_dir and entries_dir_relative
changelog_entries_sorted() arguments with entries_dir. Needed to make
the following work easier.

NO_DOC=tools
NO_CHANGELOG=tools

- Improve reporting from failed luatest tests (tarantool/test-run#319)
- Update luatest to 0.5.7 (tarantool/test-run#322)

This patch introduces a way to create a FINALIZER for user-defined
aggregate functions.

Closes #2579

@TarantoolBot document
Title: User-defined aggregate functions

User-defined aggregate functions are now available. To create a
user-defined aggregate function, the 'aggregate' option should be set to
'group'.  The function must have at least one argument. The last
argument is always the result of the function in the previous step, or
NULL if it is the first step. The last argument will be added
automatically.

Example:
```
box.schema.func.create("F1", {language = "Lua",
    body = [[
        function(x, state)
            if state == nil then
                state = {sum = 0, count = 0}
            end
            state.sum = state.sum + x
            state.count = state.count + 1
            return state
        end
    ]],
    param_list = {"integer", "map"},
    returns = "map",
    aggregate = "group",
    exports = {"SQL"},
})

box.execute('CREATE TABLE t (i INT PRIMARY KEY);')
box.execute('INSERT INTO t VALUES(1), (2), (3), (4), (5);')
box.execute('SELECT f1(i) FROM t;')
```

The result will be:
```
tarantool> box.execute([[select f1(i) from t;]])
---
- metadata:
  - name: COLUMN_1
    type: map
  rows:
  - [{'sum': 15, 'count': 5}]
...
```

To create a finalizer for a function, another function should be
created. This function should follow the following rules:
1) its name should be '<function name>_finalize';
2) it must take exactly one argument;
3) it must be a non-aggregate function.

If an aggregate function has a finalizer, the result of the aggregate
function will be determined by the finalizer.

Example:
```
box.schema.func.create("F1_finalize", {
    language = "Lua",
    body = [[
        function(state)
            if state == nil then
                return 0
            end
            return state.sum / state.count
        end
    ]],
    param_list = {"map"},
    returns = "number",
    exports = {'LUA', 'SQL'},
})
box.execute('SELECT f1(i) FROM t;')
```

The result will be:
```
tarantool> box.execute([[select f1(i) from t;]])
---
- metadata:
  - name: COLUMN_1
    type: number
  rows:
  - [3]
...
```

This patch introduces user-defined aggregate functions in SQL.

Part of #2579

This patch fixes the conditions under which COUNT() is optimized. At
some point, the conditions were broken, but since there was no other
aggregate function requiring zero arguments, this problem did not change
the behavior.

Needed for #2579

* gdb: add mmudata value to lj-gc output
* gdb: unwind Lua stack top part

Before this patch each connection reload server schema if it
changed. However in some cases it's actually redundant e.g. user
performs only call and eval operations. This patch introduces
"fetch_schema" option (true by default to keep backward
compatibility) that allows to ignore schema changes on the server.
As result remote spaces are unavailable and on_schema_reload
trigger doesn't work.

@TarantoolBot document
Title: Document fetch_schema netbox option

Currently user could specify "fetch_schema" flag in netbox
connection to control schema fetching from remote instance.
In case if this option is false remote spaces will be unavailable
and on_schema_reload trigger won't work.

```
c = netbox.connect(uri, {fetch_schema = false})
c.space -- always will be nil
c:on_schema_reload(function() end) -- never will be triggered
```

Closes #4789

When timer has 0 timeout and 0 repeat, during timers preparation
libev makes them inactive right away but puts into the array of
pending events.

Therefore to check if a timer is really active (will be executed
in the future) need to look both at ev_is_active() and
ev_is_pending().

It could happen only during split-vote, because all the other
places use election_timeout + random shift, while split vote uses
just random shift - it can be 0.

The patch makes raft do that for all timer checks. Also to make
the testing reliable and not depend on random the shift factor now
is configurable. For the test it is set to 0.

Closes #6847

NO_DOC=Bugfix
NO_CHANGELOG=Bug was not released

Although it should be fine to use a plain iostream concurrently from
different threads, this wouldn't work for more complex streams, e.g.
SSL. Let's add an assertion to catch this.

NO_DOC=debugging
NO_CHANGELOG=debugging

iostream must not be used concurrently by different threads.
It works for plain streams, but not for SSL streams.

NO_DOC=enterprise bug fix
NO_CHANGELOG=enterprise bug fix

The next commit will need to access it from applier_signal_ack, which is
defined upper. We could move applier_thread upper in the C file, but I
don't see any point to hide this struct: it brings no new dependencies.
Let's keep all applier structs together in the header file - it makes
the code easier to follow.

NO_DOC=refactoring
NO_CHANGELOG=refactoring

Currently, it works either way, but once we move the writer fiber to the
applier thread, it'll be crucial that no ACK can be signalled after the
applier is detached.

NO_DOC=refactoring
NO_CHANGELOG=refactoring

We check if it's time to switch the applier to the FOLLOW state from
applier writer, because we might need to call on_state triggers, which
may yield. The fact is FOLLOW triggers never yield so this isn't
necessary. Let's move applier_check_sync calls out of applier writer.

This is a step towards moving the applier fiber from the main to the
applier thread, which is required to properly handle SSL connections.

NO_DOC=refactoring
NO_CHANGELOG=refactoring

Applier writer fiber is started after JOIN so the only possible states
are READY, SYNC, and error states. It's fine to try to write on error -
the operation will be silently ignored (due to EPIPE check below).

This is a step towards moving the applier fiber from the main to the
applier thread, which is required to properly handle SSL connections.

NO_DOC=refactoring
NO_CHANGELOG=refactoring

Just like we do for other applier fibers. It's useful for debugging.
Note, we used proper names before commit
dacbf708 ("Introduce applier thread").

NO_DOC=minor change in logging
NO_CHANGELOG=minor change in logging

To avoid confusion with applier->thread.reader, which actually does
reading.

NO_DOC=refactoring
NO_CHANGELOG=refactoring

From time to time we can see failures of one of `box-luatest` tests.
It's `gh_6794_recover_nonmatching_xlogs.test_ignore_with_force_recovery`
test:

    [021] box-luatest/gh_6794_recover_nonmatching_xlogs_>
    [021] not ok 2 box-luatest.gh_6794_recover_nonmatching_xlogs.test_ignore_with_force_recovery #
    [021] Rejected result file: /tmp/tnt/rejects/box-luatest/gh_6794_recover_nonmatching_xlogs.reject
    [021] [ fail ]

More detailed failure log:

    not ok 2	box-luatest.gh_6794_recover_nonmatching_xlogs.test_ignore_with_force_recovery
    #   .../tarantool/test/luatest_helpers/server.lua:99: Waiting for "readiness" on server master-UCg6PXluSqYt (PID 85920) timed out
    #   stack traceback:
    #   	.../tarantool/test/luatest_helpers/server.lua:99: in function 'wait_for_readiness'
    #   	.../tarantool/test/luatest_helpers/server.lua:146: in function 'start'
    #   	...t/box-luatest/gh_6794_recover_nonmatching_xlogs_test.lua:32: in function 'box-luatest.gh_6794_recover_nonmatching_xlogs.test_ignore_with_force_recovery'
    #   	...
    #   	[C]: in function 'xpcall'

If we take a look at the log of a tarantool server that is used for
testing, we will see the failure reason:

    2022-02-04 21:06:12.063 [85920] main/103/default.lua evio.c:240 E> SocketError: unlink, called on fd 27, aka unix/:(socket), peer of unix/:(socket): Address already in use
    2022-02-04 21:06:12.063 [85920] main/103/default.lua F> can't initialize storage: unlink, called on fd 27, aka unix/:(socket), peer of unix/:(socket): Address already in use

Both tests from the gh_6794_recover_nonmatching_xlogs_test.lua file used
the same alias (master) for the tarantool server, hence in both tests
the tarantool server used a socket with the same name (master.iproto).
So we had a race here: if the socket from the previous test was released
in time, our test succeeded, otherwise, failed.

To fix the issue it was decided to create tarantool servers with
different aliases. This gives us sockets with different names, hence no
socket intersection.

Closes tarantool/tarantool-qa#151

NO_CHANGELOG=test stuff
NO_DOC=testing stuff

All tests from the update_optimize.test.lua file were fully migrated
to the luatest framework. The old test files were deleted.

Closes #6556

NO_CHANGELOG=testing stuff
NO_DOC=testing stuff

`get_proc_name` code is clogged with procedure cache management logic:
refactor this logic out.

Needed for #4002

NO_DOC=refactoring
NO_CHANGELOG=refactoring

Co-authored-by: Egor Elchinov <elchinov.es@gmail.com>
Co-authored-by: Georgiy Lebedev <g.lebedev@tarantool.org>

As soon as replica subscribes, relay starts scanning the first requested
WAL file to find the position from which to start replication. While the
scan is in progress, relay never sends heartbeats to the replica, so a
really long scan can make the replica timeout before relay finally finds
the point to replicate from.

Two similar problems were already addressed in
30ad4a55 ("relay: yield explicitly every N sent rows") and
17289440 ("recovery: make it yield when positioning in a WAL")

These patches do not fix the issue completely for relay: even though now
it yields while scanning a WAL, it doesn't send heartbeats anyway.

Fix the issue by not only yielding, but sending heartbeats as well while
WAL scan is in progress.

Follow-up #5979
Closes #6706

Sent rows were counted explicitly before the patch
17289440 ("recovery: make it yield when positioning in a WAL").
Now all the row count details are hidden inside the xstream.

Remove the redundant field.

Follow-up #5979

The _session_settings space is a virtual space which only contains
information about the current session, hence, it should be accessible
by everyone without granting any additional privileges.

New users are granted public role by default: grant read,write access
on _session_settings space to public role.

Closes #6310

@TarantoolBot document
Title: public role rw access on _session_settings space

Public role (which is granted to new users by default) now has read,
write access on _session_settings_space.

According to the readline library documentation, the callback passed to
rl_callback_handler_install is supposed to free the string passed to it:

> As with readline(), the handler function should free the line when it
> it finished with it.

( See https://tiswww.case.edu/php/chet/readline/readline.html#SEC41 )

Our callback leaks it.

Closes #6817

NO_DOC=bug fix

tarantool/luatest#214 adds copy directory semantics to
luatest_helpers.Server: reuse it in the scope of this test.

Follow-up #tarantool/luatest#214

NO_DOC=refactoring
NO_CHANGELOG=refactoring

Currently, one can only specify a work directory that will be directly
passed to box.cfg (via workdir option passed to luatest.Server, via the
TARANTOOL_WORKDIR environment variable, or via explicitly specifying
work_dir in box.cfg), which implies that the test server will be run in
the specified directory.

One might need to start a Tarantool instance from a certain snapshot or
set of xlogs — for instance, to test Tarantool over an older schema
version.

For diff-tests the test-run harness provided a 'workdir=' option when
calling inspector:cmd('create server ...'), which solved this issue by
copying the specified directory's contents into the test server's
actual working directory (see tarantool/test-run/blob/
d16cbdc2702e7d939d5d34e41730fd83d2c65879/lib/preprocessor.py#L274-L284).

To solve this issue using the luatest_helpers harness we introduce a
datadir option for luatest_helpers.Server with the same semantics.

Needed for #6571
Closes tarantool/luatest#214

NO_DOC=luatest helpers
NO_CHANGELOG=luatest helpers

The MVCC transaction manager story garbage collection introduced a
subtle code dependency: TREE index iterators can get broken (see
definition of “broken” at struct bps_tree_iterator), because the
elements they are referencing can change during story garbage
collection.

We coined the notion “MVCC TRANSACTION MANAGER STORY GARBAGE
COLLECTION BOUND” to refer to this issue explicitly: iterators and the
tree elements they reference must not be used after this point in code.

Closes #6344

NO_DOC=bug fix