* gdb: add mmudata value to lj-gc output
* gdb: unwind Lua stack top part

Before this patch each connection reload server schema if it
changed. However in some cases it's actually redundant e.g. user
performs only call and eval operations. This patch introduces
"fetch_schema" option (true by default to keep backward
compatibility) that allows to ignore schema changes on the server.
As result remote spaces are unavailable and on_schema_reload
trigger doesn't work.

@TarantoolBot document
Title: Document fetch_schema netbox option

Currently user could specify "fetch_schema" flag in netbox
connection to control schema fetching from remote instance.
In case if this option is false remote spaces will be unavailable
and on_schema_reload trigger won't work.

```
c = netbox.connect(uri, {fetch_schema = false})
c.space -- always will be nil
c:on_schema_reload(function() end) -- never will be triggered
```

Closes #4789

When timer has 0 timeout and 0 repeat, during timers preparation
libev makes them inactive right away but puts into the array of
pending events.

Therefore to check if a timer is really active (will be executed
in the future) need to look both at ev_is_active() and
ev_is_pending().

It could happen only during split-vote, because all the other
places use election_timeout + random shift, while split vote uses
just random shift - it can be 0.

The patch makes raft do that for all timer checks. Also to make
the testing reliable and not depend on random the shift factor now
is configurable. For the test it is set to 0.

Closes #6847

NO_DOC=Bugfix
NO_CHANGELOG=Bug was not released

Although it should be fine to use a plain iostream concurrently from
different threads, this wouldn't work for more complex streams, e.g.
SSL. Let's add an assertion to catch this.

NO_DOC=debugging
NO_CHANGELOG=debugging

iostream must not be used concurrently by different threads.
It works for plain streams, but not for SSL streams.

NO_DOC=enterprise bug fix
NO_CHANGELOG=enterprise bug fix

The next commit will need to access it from applier_signal_ack, which is
defined upper. We could move applier_thread upper in the C file, but I
don't see any point to hide this struct: it brings no new dependencies.
Let's keep all applier structs together in the header file - it makes
the code easier to follow.

NO_DOC=refactoring
NO_CHANGELOG=refactoring

Currently, it works either way, but once we move the writer fiber to the
applier thread, it'll be crucial that no ACK can be signalled after the
applier is detached.

NO_DOC=refactoring
NO_CHANGELOG=refactoring

We check if it's time to switch the applier to the FOLLOW state from
applier writer, because we might need to call on_state triggers, which
may yield. The fact is FOLLOW triggers never yield so this isn't
necessary. Let's move applier_check_sync calls out of applier writer.

This is a step towards moving the applier fiber from the main to the
applier thread, which is required to properly handle SSL connections.

NO_DOC=refactoring
NO_CHANGELOG=refactoring

Applier writer fiber is started after JOIN so the only possible states
are READY, SYNC, and error states. It's fine to try to write on error -
the operation will be silently ignored (due to EPIPE check below).

This is a step towards moving the applier fiber from the main to the
applier thread, which is required to properly handle SSL connections.

NO_DOC=refactoring
NO_CHANGELOG=refactoring

Just like we do for other applier fibers. It's useful for debugging.
Note, we used proper names before commit
dacbf708 ("Introduce applier thread").

NO_DOC=minor change in logging
NO_CHANGELOG=minor change in logging

To avoid confusion with applier->thread.reader, which actually does
reading.

NO_DOC=refactoring
NO_CHANGELOG=refactoring

From time to time we can see failures of one of `box-luatest` tests.
It's `gh_6794_recover_nonmatching_xlogs.test_ignore_with_force_recovery`
test:

    [021] box-luatest/gh_6794_recover_nonmatching_xlogs_>
    [021] not ok 2 box-luatest.gh_6794_recover_nonmatching_xlogs.test_ignore_with_force_recovery #
    [021] Rejected result file: /tmp/tnt/rejects/box-luatest/gh_6794_recover_nonmatching_xlogs.reject
    [021] [ fail ]

More detailed failure log:

    not ok 2	box-luatest.gh_6794_recover_nonmatching_xlogs.test_ignore_with_force_recovery
    #   .../tarantool/test/luatest_helpers/server.lua:99: Waiting for "readiness" on server master-UCg6PXluSqYt (PID 85920) timed out
    #   stack traceback:
    #   	.../tarantool/test/luatest_helpers/server.lua:99: in function 'wait_for_readiness'
    #   	.../tarantool/test/luatest_helpers/server.lua:146: in function 'start'
    #   	...t/box-luatest/gh_6794_recover_nonmatching_xlogs_test.lua:32: in function 'box-luatest.gh_6794_recover_nonmatching_xlogs.test_ignore_with_force_recovery'
    #   	...
    #   	[C]: in function 'xpcall'

If we take a look at the log of a tarantool server that is used for
testing, we will see the failure reason:

    2022-02-04 21:06:12.063 [85920] main/103/default.lua evio.c:240 E> SocketError: unlink, called on fd 27, aka unix/:(socket), peer of unix/:(socket): Address already in use
    2022-02-04 21:06:12.063 [85920] main/103/default.lua F> can't initialize storage: unlink, called on fd 27, aka unix/:(socket), peer of unix/:(socket): Address already in use

Both tests from the gh_6794_recover_nonmatching_xlogs_test.lua file used
the same alias (master) for the tarantool server, hence in both tests
the tarantool server used a socket with the same name (master.iproto).
So we had a race here: if the socket from the previous test was released
in time, our test succeeded, otherwise, failed.

To fix the issue it was decided to create tarantool servers with
different aliases. This gives us sockets with different names, hence no
socket intersection.

Closes tarantool/tarantool-qa#151

NO_CHANGELOG=test stuff
NO_DOC=testing stuff

All tests from the update_optimize.test.lua file were fully migrated
to the luatest framework. The old test files were deleted.

Closes #6556

NO_CHANGELOG=testing stuff
NO_DOC=testing stuff

`get_proc_name` code is clogged with procedure cache management logic:
refactor this logic out.

Needed for #4002

NO_DOC=refactoring
NO_CHANGELOG=refactoring

Co-authored-by: Egor Elchinov <elchinov.es@gmail.com>
Co-authored-by: Georgiy Lebedev <g.lebedev@tarantool.org>

As soon as replica subscribes, relay starts scanning the first requested
WAL file to find the position from which to start replication. While the
scan is in progress, relay never sends heartbeats to the replica, so a
really long scan can make the replica timeout before relay finally finds
the point to replicate from.

Two similar problems were already addressed in
30ad4a55 ("relay: yield explicitly every N sent rows") and
17289440 ("recovery: make it yield when positioning in a WAL")

These patches do not fix the issue completely for relay: even though now
it yields while scanning a WAL, it doesn't send heartbeats anyway.

Fix the issue by not only yielding, but sending heartbeats as well while
WAL scan is in progress.

Follow-up #5979
Closes #6706

Sent rows were counted explicitly before the patch
17289440 ("recovery: make it yield when positioning in a WAL").
Now all the row count details are hidden inside the xstream.

Remove the redundant field.

Follow-up #5979

The _session_settings space is a virtual space which only contains
information about the current session, hence, it should be accessible
by everyone without granting any additional privileges.

New users are granted public role by default: grant read,write access
on _session_settings space to public role.

Closes #6310

@TarantoolBot document
Title: public role rw access on _session_settings space

Public role (which is granted to new users by default) now has read,
write access on _session_settings_space.

According to the readline library documentation, the callback passed to
rl_callback_handler_install is supposed to free the string passed to it:

> As with readline(), the handler function should free the line when it
> it finished with it.

( See https://tiswww.case.edu/php/chet/readline/readline.html#SEC41 )

Our callback leaks it.

Closes #6817

NO_DOC=bug fix

tarantool/luatest#214 adds copy directory semantics to
luatest_helpers.Server: reuse it in the scope of this test.

Follow-up #tarantool/luatest#214

NO_DOC=refactoring
NO_CHANGELOG=refactoring

Currently, one can only specify a work directory that will be directly
passed to box.cfg (via workdir option passed to luatest.Server, via the
TARANTOOL_WORKDIR environment variable, or via explicitly specifying
work_dir in box.cfg), which implies that the test server will be run in
the specified directory.

One might need to start a Tarantool instance from a certain snapshot or
set of xlogs — for instance, to test Tarantool over an older schema
version.

For diff-tests the test-run harness provided a 'workdir=' option when
calling inspector:cmd('create server ...'), which solved this issue by
copying the specified directory's contents into the test server's
actual working directory (see tarantool/test-run/blob/
d16cbdc2702e7d939d5d34e41730fd83d2c65879/lib/preprocessor.py#L274-L284).

To solve this issue using the luatest_helpers harness we introduce a
datadir option for luatest_helpers.Server with the same semantics.

Needed for #6571
Closes tarantool/luatest#214

NO_DOC=luatest helpers
NO_CHANGELOG=luatest helpers

The MVCC transaction manager story garbage collection introduced a
subtle code dependency: TREE index iterators can get broken (see
definition of “broken” at struct bps_tree_iterator), because the
elements they are referencing can change during story garbage
collection.

We coined the notion “MVCC TRANSACTION MANAGER STORY GARBAGE
COLLECTION BOUND” to refer to this issue explicitly: iterators and the
tree elements they reference must not be used after this point in code.

Closes #6344

NO_DOC=bug fix

The results of diff tests depend on specific numbers (tuple
size for example). Remove this dependence for some tests,
because after tuple compression implementation, tuple size
will be change.

Part of #2695

NO_CHANGELOG=test fix
NO_DOC=test fix

Due to the fact that we used `const char**` to save a pointer to the
original msgpack header, when restoring the pointer, we received an
incorrect value and as a consequence assertion failure later. Correctly
saving and restoring the pointer fixed this problem.

NO_DOC=bug fix
NO_CHANGELOG=bug fix

There were two problems with resource release in performance test:
- because of manually zeroing of `box_tuple_last`, tuple_format
  structure was not deleted. `box_tuple_last` should be zeroed in
  `tuple_free` function.
- invalid loop for resource release in one of the test cases.
This patch fix both problems.

NO_CHANGELOG=test fix
NO_DOC=test fix

A memtx tree iterator remembers the last tuple returned to the user and
its hint so that it can restore iteration if the index is changed. To
prevent the tuple from being freed, it references it. The problem is
it's not enough for a functional index, because the latter allocates key
parts separately from tuples (it stores pointers to them in memtx tree
hints). As a result, if a tuple is deleted from the tree, its key parts
will be immediately freed, even if the tuple itself is referenced. Since
key parts are necessary to restore an iterator, this results in a use
after free bug.

To fix the issue, let's store a copy of the current tuple key part in
the iterator along with the tuple. If a key part is small, the copy is
stored in a preallocated fixed-size buffer, otherwise it's allocated
with malloc.

Closes #6786

So as not to set tree_iterator::current directly. Currently, we only
need to ref/unref tuple when we update current, but in the future we
will also alloc/free func index key part, a pointer to which is stored
in a hint.

Needed for #6786

These functions are used only by memtx functional indexes and they
depend on neither tuple_format nor tuple. No need to keep them in
tuple_format_vtab.

Needed for #6786

* memprof: report JIT-side allocations as internal

Closes #5679

This patch drops IPROTO_SHUTDOWN and IPROTO_FEATURE_GRACEFUL_SHUTDOWN
(the feature hasn't been officially released yet so it's okay). Instead,
the server now generates IPROTO_EVENT{key='box.shutdown', value=true}
before running on_shutdown triggers. The on_shutdown trigger installed
by IPROTO has been greatly simplified - now it just stops listening for
new connections. A new on_shutdown trigger is installed by the session
infrastructure code - the trigger waits for all sessions that subscribed
to 'box.shutdown' event to close. Net.box uses conn:watch to subscribe
internally to the new event. As before, it runs user-defined on_shutdown
triggers and then waits for active requests to finish before closing the
connection.

Notes about the tests:
 1. We don't need graceful_shutdown_errinj test, because
    a. The feature can be disabled in net.box without errinj by adding
       internal _disable_graceful_shutdown connection option.
    b. Since there's no new packet type, we don't need to check that
       IPROTO_ID/AUTH handle it correctly in net.box.
 2. The graceful_shutdown.test_shutdown_aborted_on_reconnect_2 changed:
    since triggers use the watcher infrstructure, the same trigger can't
    run in parallel with itself.

The main benefit of reusing IPROTO_EVENT instead of introducing a new
request type is that connections will have to do less work to support
the feature (to some extent, it'd be supported even if the connector
doesn't do any work and just lets the user subscribe to events).

Suggested-by @unera
Follow-up a33f3cc7
("net.box: support graceful shutdown protocol")
Follow-up 6f29f9d7
("iproto: introduce graceful shutdown protocol")

Follow-up #5924

NO_CHANGELOG=already added
NO_DOC=already added, will be updated manually

Currently, to resubscribe registered watchers net.box installs an
on_connect trigger that writes IPROTO_WATCH requests to the send buffer.
The requests will be sent by the net.box state machine once all
on_connect triggers return. The problem is this trigger may run before a
user-defined trigger that closes the connection, in which case close()
will hang forever, because since #6338, the close method blocks until
the send buffer is emptied, while it can't be emptied in case close is
called by the state machine itself (see #5358).

Let's fix this issue by making close() omit blocking if called from
the state machine fiber.

No user is affected, because IPROTO_WATCH hasn't been officially
released yet so neither changelog nor doc is required.

Closes #6824

NO_CHANGELOG=bug not released
NO_DOC=NO_CHANGELOG

This commit fixes the following assertion failure that occurrs on an
attempt to register a watcher for a connection to a server that doesn't
support watchers (e.g. 2.8):

  src/box/lua/net_box.c:2283: netbox_transport_dispatch_response:
    Assertion `transport->inprogress_request_count > 0' failed.

The problem is that an IPROTO_WATCH request isn't accounted in
inprogress_request_count, because the server isn't supposed to reply
to it. However, if the server doesn't support the request type, it
will reply to it with an error, breaking the assumption made by the
client.

We fix the assertion itself by explicitly excluding requests with sync=0
from accounting on the client side (IPROTO_WATCH has sync=0), because
such requests aren't supposed to have a matching reply. Also, to avoid
errors on the server side, we make the client code not send IPROTO_WATCH
and IPROTO_WATCH if the server doesn't set the 'watchers' feature bit in
reply to the IPROTO_ID request from the client.

No user is affected, because IPROTO_WATCH hasn't been officially
released yet so neither changelog nor doc is required.

Closes #6819

NO_CHANGELOG=bug not released
NO_DOC=NO_CHANGELOG

IPROTO triggers input processing after sending output, provided the
output channel isn't clogged, which totally makes sense. There's one
nuance here: currently, input is triggered only if something was written
to the output channel. Before the IPROTO_WATCH request type was added by
commit 4be5de4c ("iproto: add watchers
support"), every request had a response so it worked just fine. The new
request type is special - the server doesn't reply to it. As a result,
if input was blocked upon reaching the box.cfg.net_msg_max limit, it
might never get resumed, because input processing might never be
triggered. Fix this issue by triggering input processing even if there
was no output to flush.

No user is affected, because IPROTO_WATCH hasn't been officially
released yet so neither changelog nor doc is required.

Closes #6818

NO_CHANGELOG=bug not released
NO_DOC=NO_CHANGELOG

This change splits the debug_coverage.yml workflow into two independent
workflows: debug.yml and coverage.yml.

The first one will be run on push to a branch and creating/updating
external pool requests. In other words, it will be run always.

The second one will be run on push to a branch only if the branch has
the `full-ci` suffix (some-branch-full-ci) or the `full-ci` label is set
on a pool request, internal or external - it doesn't matter.

Why do we need it?

1. The coverage testing is heavy testing due to enabling long tests
   and takes at least 2 times more time than debug build testing. So
   testing that is done at the first stage (lint, release, debug) will
   pass faster.
2. The coverage report as a pool request comment from coveralls.io can
   be obtained only if a pool request event happened. It is a limitation
   of the `coverallsapp/github-action` action [1]. So there is no sense
   to run the coverage workflow on push events. But the coverage report
   on push still can be seen on coveralls.io if the branch has the
   `full-ci` suffix.
3. The logic of the debug_coverage.yml workflow is tricky enough and no
   one understands how it works. So we would like to have an absolutely
   transparent and simple workflow.

[1] https://github.com/coverallsapp/github-action/tree/8cbef1dea373ebce56de0a14c68d6267baa10b44#coveralls-github-action

Closes tarantool/tarantool-qa#147

NO_DOC=ci
NO_CHANGELOG=ci

Fix static build for macOS 11.5 or higher.
On macOS SDK ver. 11.5 some `*.dylib` files was replaced with `*.tbd`.
So we replace `libunwind.dylib` on `libunwind.tbd`.
Because of macOS 10.15 support being dropped
conditional is not needed.

Closes #6052

Because of the typo, the test fails on my localhost like this:

  There is no test with name 'test_ignore_with_force_recovery'
    but hook 'after_test' is defined for it

(I use luatest installed with luarocks).

Follow-up #6794
Follow-up 8c8b7739
  ("recovery: fix recovering from 1.6 xlogs/snaps")

Tarantool 1.6 didn't rotate xlogs on snap creation.
This means, that Tarantool 1.6 data dir may contain files like
0...0.xlog and 0...01.snap, where 0...0.xlog covers changes both before
and after the snap creation.

New Tarantool versions fail to recover from such files: they recover the
data correctly, but fail to update the vclock accordingly.

Let's allow recovering from such files and update the vclock
accordingly, but only when force_recovery is set.

Patch 634f59c7 ("recovery: panic in case of recovery and replicaset
vclock mismatch") added the panic, which helped to discover the issue.
Before that patch Tarantool would silently recover with an outdated
vclock.

Also, while we're at it, refactor the check leading to a panic: it never
happens after hot_standby, so move it into "else" branch of hot_standby.

Closes #6794

Privileges passed to box.schema.user.grant are resolved in quite a
naïve manner: we do not check that all of them have been resolved.

Also, the privilege_resolve function itself is poorly designed (instead
of generic string processing, we have a bunch of if's): extract parsing
of the privileges string into privilege_parse and process it
through a generic FSA.

Closes #6199

Well, the diff is not nice, so I'll summarize the changes for humans.

General changes:

* Moved patch submission and so on to the end of the document.
* Reformatted to usual 80 columns, inlined and actualized links.
* Reformatted lists: start from a capital letter, end with a period.
* Minor wording tweaks to make it sound better.

Badges:

* Travis CI and GitLab CI -> GitHub Actions.
* Removed Slack and Gitter.
* Google Groups -> GitHub Discussions.
* Added Stack Overflow.

Define Tarantool as in-memory computing platform as on the website.

Added license information.

Appserver:

* Rewrote the sentence regarding compatibility with Lua 5.1 and
  highlighted key LuaJIT features instead. There are doubts within the
  team about declaring 100% compatibility with Lua 5.1, let's go to the
  safe ground so.
* Splitted the list in a more granular way.
* Added links to mysql and pg connectors.
* Added queue, vshard, cartridge with links.
* Added a link to the modules page.

Database:

* Replaced MsgPack with MessagePack. It is called so on the official
  website.
* Replaced 'optional persistence' with 'complete WAL-based persistence'
  in the memtx description to make it less confusing. There are users,
  which believe that memtx does not offer persistency.
* Moved SQL down.
* Added JSON path indexes.
* Added synchronous quorum-based replication.
* Added RAFT-based automatic leader election.
* Added a link to the connectors list.

Platforms:

* Dropped OpenBSD from the list. It is not verified in CI, so I feel it
  like 'we have no commitments here'.
* Replaced x86 with more explicit x86_64.
* Moved Mac OS before FreeBSD (we have more users here).
* Added aarch64/M1.

Tarantool is ideal for...

* Kept intact, just reformatted to 80 columns.

Download:

* Added 'or using Docker' in addition to 'as a binary package to your
  OS'.

Added a link to the 'Awesome Tarantool' list.

Report bugs / send feedback or patch:

* Replaced Google Groups with GitHub Discussions.
* Added Stack Overflow.
* Highlighted that we accept pull requests and linked 'How to get
  involved' guide.

Fixes #6579

* Actually implement maxirconst trace limit.
* Fix string.char() recording with no arguments.

Closes #6371
Part of #6548

This commit adds a new script, tools/check-commits. The script takes git
revisions in the same format as git-rev-list (actually, it feeds all its
arguments to git-rev-list) and runs some checks on each of the commits.

For example, to check the head commit, run:

  ./tools/check-commits -1 HEAD

To check the last five commits, run:

  ./tools/check-commits HEAD~5..HEAD

Currently, there are only two checks, but in future we may add more
checks, e.g. check diffs for trailing spaces:
 - The commit message contains a documentation request. Can be
   suppressed with NO_DOC=<reason> in the commit message.
 - A new changelog entry is added to changelog/unreleased by the commit.
   Can be suppressed with NO_CHANGELOG=<reason> in the commit message.

This commit also adds a new workflow triggered on pull request,
lint/commits. The workflow runs the tools/check-commits script on all
the commits added by the pull request. The workflow doesn't run on push,
because it's problematic to figure out what commits are new on a branch.
Besides, we don't want to run it on push to release branches, because
it's a pure dev workflow.

Example output:

Checking commit a33f3cc7 PASS
Checking commit 6f29f9d7 FAIL
SHA:     6f29f9d7
SUBJECT: iproto: introduce graceful shutdown protocol
  ERROR:
    Changelog not found in changelog/unreleased. If this commit
    doesn't require changelog, please add NO_CHANGELOG=<reason>
    to the commit message.
Checking commit fbc25aae FAIL
SHA:     fbc25aae
SUBJECT: Update small submodule
  ERROR:
    Missing documentation request ('@TarantoolBot document' not
    found in the commit message). If this commit doesn't need to
    be documented, please add NO_DOC=<reason> to the commit
    message.
  ERROR:
    Changelog not found in changelog/unreleased. If this commit
    doesn't require changelog, please add NO_CHANGELOG=<reason>
    to the commit message.

NO_DOC=ci
NO_CHANGELOG=ci