We're going to add a whole bunch of them. Putting them all in
a sub-directory will help keeping the file tree organized.

Note, we have to update .gitignore so that the patches/ sub-directory
is ignored only at the top level (it's used by quilt).

NO_DOC=build
NO_TEST=build
NO_CHANGELOG=build

This is useful for example for the analysis of performance complaints
from users, when they claim that one version of Tarantool is slower
than another, in fact comparing debug and release builds.

NO_DOC=minor change
NO_TEST=minor change

This update pulls the following commits

 * Constify mp_char2escape
   https://github.com/tarantool/msgpuck/commit/28a7421cf7fa538a0180c79bd9c12ee0dd8c12eb

   This is code cleanup.

 * Don't escape forward slash in mp_snprint
   https://github.com/tarantool/msgpuck/commit/e05a538d076509063240a00f2e703ede7f803a87

   This commit disables escaping of the forward slash character in
   mp_snprint, because the function produces JSON-like string and
   according to the JSON spec, we don't need to escape it.

Follow-up #8117

NO_DOC=submodule update
NO_TEST=submodule update
NO_CHANGELOG=minor change

This commit adds the json_escape_forward_slash variable and a tweak for
it that is now used by the compat module. The new variable configures
whether '/' is escaped when encoded in JSON.

Note, the old implementation was quite messy so we had to rework it:

 - Drop luaL_serializer::encode_escape_forward_slash. This was an
   undocumented serializer option implemented only by the JSON
   serializer and used only by the compat module. Now, we use the
   json_escape_forward_slash global tweak for all JSON serializers
   instead, because making this tweak configurable per serializer
   doesn't make much sense.

 - Don't use mp_char2escape for escaping JSON characters. Historically,
   we used this table defined in libmsgpuck in the json_escape utility
   function. It equals the escape table used by the JSON encoder so it
   looks more reasonable to use the JSON encoder escape table in
   json_escape. This commit moves the JSON encoder escape table to
   util.c and adds an inline utility function json_escape_char which is
   now used by the JSON encoder and json_escape.

 - Drop an extra JSON escape table with '/' escaped. We had two escape
   tables in JSON, one with escaped '/' and another with unescaped '/'.
   Duplicating tables like this is error-prone and ugly. Let's use one
   table with '/' unescaped and check the json_escape_forward_slash flag
   in json_escape_char instead. The cost of this check is negligible
   performance-wise.

This commit also drops the lua/compat.c source file, because it isn't
used anymore.

While we are at it, remove any mentions of MsgPack from the changelog
entry for the json_escape_forward_slash compat option, because it isn't
relevant anymore.

Closes #8117

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

This commits adds the yaml_pretty_multiline variable and a tweak for it
that is now used by the compat module. The new variable configures
whether all multiline strings are encoded in the block scalar style or
only those that contain a '\n\n' substring.

Part of #8117

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

This commit adds a tweak for the fiber_channel_close_mode variable
and makes the compat module use the new tweak instead of switching
the variable directly.

Note that we don't drop fiber_channel_set_close_mode function, because
it's still used in unit tests.

Part of #8117

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

This commit adds the new 'internal.tweaks' Lua module for toggling
internal tweaks. The module is implemented as a Lua table with the
__index and __newindex metamethods overridden, which makes it possible
to get/set a tweak value with a simple assignment, for example:

  tarantool> tweaks = require('internal.tweaks')
  ---
  ...

  tarantool> tweaks.fiber_channel_close_mode = 'graceful'
  ---
  ...

  tarantool> tweaks.fiber_channel_close_mode
  ---
  - graceful
  ...

(Note, currently there are no tweaks available; tweaks are added in the
following commits.)

Getting a value of an unknown tweak returns nil. Setting a value of an
unknown tweak raises 'No such option' error. Setting an invalid value
for an existing tweak raises 'Invalid value' error. The module also
implements __serialize and __autocomplete metamethods that return all
tweaks and their values in a table, for example:

  tarantool> tweaks
  ---
  - fiber_channel_close_mode: forceful
    yaml_pretty_multiline: false
    json_escape_forward_slash_default: true
  ...

Closes #7883
Needed for #8117

NO_DOC=internal
NO_CHANGELOG=internal

This commit adds an internal C API for registering tweaks. A tweak is
an object that provides a convenient setter/getter API for a global C
variable.

To register a tweak, use a TWEAK_XXX macro at the global level in
a C source file:

  static int my_var;
  TWEAK_INT(my_var);

The name of a tweak equals the name of the underlying variable
("my_var" in the example above).

To set/get a tweak value, use the tweak_set and tweak_get functions:

  struct tweak_value val;
  tweak_get("my_var", &val);
  val.ival = 42;
  tweak_set("my_var", &val);

The tweak_value struct is a variant that can contain one of three types:
int, bool, and string, one per each available tweak types:

  TWEAK_BOOL(bool_var);
  TWEAK_INT(int_var);
  TWEAK_ENUM(enum_name, enum_var);

The TWEAK_ENUM macro is special, as it also requires the name of the
enum type to be passed. When a enum tweak value is exported/imported,
it is converted to a string using the STR2ENUM macro.

It's also possible to iterate over all registered tweaks using the
tweak_foreach() function.

The tweak registry is a simple hash table mapping tweak names to
tweak objects, which in turn point to underlying variables. Tweaks
are registered at startup using the constructor function attribute.

Part of #7883
Needed for #8117

NO_DOC=internal
NO_CHANGELOG=internal

The test started hanging after commit d560fb3f ("Revert "replication:
set default replication_sync_timeout to 0"") because it still expected a
zero replication_sync_timeout. This wasn't caught by PR's full-ci for
some reason.

Follow-up #8223

NO_DOC=test fix
NO_CHANGELOG=test fix

Increase changelog verbosity, add examples of how to achieve old
behavior.

Follow-up #5272

NO_DOC=added in original commit
NO_TEST=changelog

Instead of setting the `replication_sync_timeout` default to new value
(0) unconditionally, add a compat option which will control the default
value. The compat option is named "box_cfg_replication_sync_timeout"
and is "old" by default, meaning default `replication_sync_timeout` is
300.

The user has to set it to "new" before the initial box.cfg call for the
default value to change to 0.

Follow-up #5272

NO_DOC=amended an existing doc issue, see https://github.com/tarantool/doc/issues/3295

This reverts commit 67cb4e4e.

The commit introduced new behaviour unconditionally and by default,
which's considered a breaking change. It was decided to add a compat
option for this change. So revert the commit.

NO_DOC=amended an existing doc issue manually. See https://github.com/tarantool/doc/issues/3295

Bitset index size calculation uses the cardinality of the 'flag' bitset,
but when the bitset index is empty, i.e., uninitialized, the
'flag' bitset is not allocated, hence we should simply return 0.

Closes #5809

NO_DOC=bugfix

After adding the autorequiring luatest [1,2], there is no need to use
the following approach now:

```
local t = require('luatest')
local g = t.group()

server:exec(function()
    local t = require('luatest') -- duplicate
    t.assert(...)
end)
```

Modern approach looks like:

```
local t = require('luatest')
local g = t.group()

-- `t` already available in the remote server
server:exec(function() t.assert(...) end)

-- also it works with any variable
local my_custom_t = require('luatest')

server:exec(function()
    my_custom_t.assert(...) -- already available
end)
```

[1] tarantool/luatest#277
[2] tarantool/luatest#289

Part of tarantool/luatest#233

NO_DOC=test fix
NO_TEST=test fix
NO_CHANGELOG=test fix

We didn't take into consideration the fact, that precision
value passed to control the width of nanoseconds part in
datetime_object:format could be more than maximum positive
value, integer may have. Currently it leads to segfault.

```
tarantool> require('datetime').new{}:format('%2147483648f')
```

We should check errno in order to find out, if overflow
occurs. The problem is the fact, that `width` variable must
have int type due to snprintf requirements ("%*d") and
strtol returns long. Errno won't be set if returned value
is in bounds [INT_MAX, LONG_MAX], but it will overflow
int resulting in inconsistent behavior.

So, let's save the result of strotl to the temp value.
If this value doesn't belong to the above-mentioned set,
or errno was set, we assign to `width` maximum value, it
may have: 9.

Closes tarantool/security#31

NO_DOC=bugfix

The function was replaced by luaT_newmodule(). See commit e3cf5a5d
("lua: add built-in module registration function") for the motivation
and details.

Now all luaL_register_module() usages are eliminated (including
Tarantool Enterprise source code) and we can safely drop it.

Part of #7774
Follows up PR #8173

NO_DOC=no user visible changes: the function is internal
NO_TEST=see NO_DOC
NO_CHANGELOG=see NO_DOC

To improve the stability of the tests, let's use unix sockets for iproto
connection instead of ports.

NO_DOC=testing stuff
NO_TEST=testing stuff
NO_CHANGELOG=testing stuff

During creation of a function definition, an error condition for spaces is
diagnosed for wrong options: change it to its own error code.

While we are here, cover missing function definition error conditions.

Closes #7972

NO_DOC=bugfix

Bump test-run to new version with the following improvements:

- Bump luatest to 0.5.7-25-g9e117e6 [1]

[1] tarantool/test-run@7c77fcb

NO_DOC=testing stuff
NO_TEST=testing stuff
NO_CHANGELOG=testing stuff

Fixing a bug with nodes in 'manual' election mode bumping the term
excessively revealed a hang in election_pre_vote test. Turns out the
test passed thanks to the previous buggy behaviour.

The following behaviour is expected: when a node is configured in manual
election mode, calling box.ctl.promote() on it should make it bump term
once, try to gather votes and fail on timeout.

Once the extra term bump on timeout was removed in commit 5765fdc4
("raft: fix 'manual' nodes bumping the term excessively"),
box.ctl.promote() without a quorum started hanging.

Let's return the correct behaviour: 'manual' nodes should transition
back to follower if an election timeout passes after the promotion
without any term outcome.

Enable the test_promote_no_quorum testcase of election_pre_vote test
back, since it's fixed now.

Follow-up #8168
Closes #8217

NO_DOC=bugfix
NO_CHANGELOG=changes not released behaviour

The `tarantool_version` symbol identifies the binary as Tarantool (see also
https://github.com/tarantool/tarantool/blob/f991f7c0be73558f0710f0af871d07e8bd506efe/tools/tarabrt.sh#L179-L180

).
It is not exported and thus can be optimized away by LTO — add it to the
exports list.

Closes #8129

Acked-by: Aleksandr Lyapunov <alyapunov@tarantool.org>

NO_CHANGELOG=<internal change>
NO_DOC=<internal change>
NO_TEST=<no convenient way to test devtools>

There used to be ';' as a default local End Of Stream in Lua, which
could be confusing for a human. Now the default local EOS is '' and it
can be configured per session with `\set output lua,local_eos=`.
Works on both local and remote consoles.

Closes #7031

@TarantoolBot document
Title: configure local End Of Stream symbol in Lua console

The old behavior was:
```
tarantool> \set output lua
true;
tarantool> 123
123;
```

Now default EOS is '', you can set it like this:
```
tarantool> \set output lua
true
tarantool> 123
123
tarantool> \set output lua,local_eos=#
true#
tarantool> 123
123#
tarantool> \set output lua,local_eos=
true
tarantool> 123
123
```

If you don't provide a symbol, it would be configured to ''.

Refactor parse_output function to handle several opts at once in
`\set output` command.

NO_DOC=refactoring
NO_CHANGELOG=refactoring
NO_TEST=refactoring

Interactive console test helper could be used for remote connection too,
for that purpose prompt needs to be changed accordingly to connection
type.

This patch introduces getter and setter for the prompt. Prompt is
configured per session, so it is advised to create a new session
for each test case (e.g. with before_each()).

NO_CHANGELOG=test helper change
NO_DOC=test helper change

box.schema.upgrade is called automatically by box.cfg if the instance
is rw and box.cfg.replication is unset. This may be unexpected and
dangerous (the upgrade operation is irreversible). Let's drop auto
schema upgrade.

A look into the history of auto schema upgrade:

The feature was first introduced by commit 29a197cf ("box: auto
upgrade to 1.7.5") to automatically create the _truncate system space so
that space.truncate continues to work after update. Later on, we figured
out that auto-upgrade shouldn't be called if the instance is read-only,
see commit 6cfefcd4 ("Do not auto upgrade schema if read only"), or
configures replication, see commit 582a85d4 ("box: disable schema
auto upgrade for replication"). This turned this initially neat and
simple feature into a monstrous creature. We should have zapped it long
time ago.

Closes #8207

NO_DOC=auto schema upgrade wasn't documented

Bump test-run to new version with the following improvements:

- Bump luatest to 0.5.7-24-g8de38b3 [1] [2]
- Add ignore_zero option to get_vclock [3]

[1] https://github.com/tarantool/test-run/pull/370
[2] https://github.com/tarantool/test-run/pull/368
[3] https://github.com/tarantool/test-run/pull/367

NO_DOC=testing stuff
NO_TEST=testing stuff
NO_CHANGELOG=testing stuff

The field will not be shown to a user in the interactive console.

NO_CHANGELOG=There are no releases with the field.
NO_DOC=The field is internal and so it is not present in the
       documentation (and shouldn't).

Reported-by: Mons Anderson <mons@cpan.org>

The memtx transaction manager MVCC invariant violation described in
c8eccfbb could actually lead to a bug described in #7394, since
`space:count` inherently relies on this invariant: add a test for this
case.

Closes #7394

NO_CHANGELOG=<bug was fixed in c8eccfbb>
NO_DOC=bugfix

A tuple update with the first operation creating a new field
somewhere deep in the tuple and the second operation trying to go
into that new field could crash. This happened because the route
branching function xrow_update_route_branch() missed this case.

It can be detected when see that the bar path is already fully
used (the next JSON token is END), and the new operation's path
is still not END.

Closes #8216

NO_DOC=bugfix

A tuple update with the first operation creating a new field
inside an array and the second operation trying to go into that
field could crash. This happened because the branching function
xrow_update_op_do_field_##op_type() didn't take into account newly
set scalar fields and an `unreachable()` was hit.

Part of #8216

NO_DOC=bugfix
NO_CHANGELOG=next commit

They are going to be needed above their declaration prior to this
patch. Also had to add a couple of obvious comments to calm
checkpatch down.

Needed for #8216

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

It is planned to allow to overload a built-in module by an external one.
The implementation assumes storing of the built-in modules in a table
that is different from `package.loaded`. It means that accessing a
module as `package.loaded[<module_name>]` may fail.

However, we still have a guarantee that a registered module is
accessible using `require()`. Use it for accessing built-in modules from
our code.

Notes on different modules:

* `console`

  Reorder the print/log callbacks to place the print callback first. If
  something is going very wrong (say, someone removes the internal.print
  module), things will be consistent: both callback will not be set.

* `session`

  `box` is accessible as `require('box')` and as `_G.box`. Use the
  latter.

src/lua/serializer.c and src/lua/digest.c also access `package.loaded`
directly. I left them out of scope here, it requires a bit more in-depth
look.

Part of #7774

NO_DOC=user visible behavior is unchanged, pure refactoring change
NO_TEST=see NO_DOC
NO_CHANGELOG=see NO_DOC

The module can't be pulled as `require('compat')`, because it is a
tarantool specific tuning mechanism, not a general compatibility module.
The API is exposed as `require('tarantool').compat`.

Before this patch the module was loaded as `package.loaded.compat`,
captured and then removed from `package.loaded`. However, it would be
easier to follow if we'll just assign an internal name for the module.

The module is renamed to `internal.compat`. The Lua/C part of the module
is renamed to `internal.compat.lib`.

Part of #7774

NO_DOC=user visible behavior is unchanged, pure refactoring change
NO_TEST=see NO_DOC
NO_CHANGELOG=see NO_DOC

There are modules that are implemented as two parts: a Lua/C module for
internal use and a public module written on Lua. There is a practice to
name both parts the same: just capture the internal part within the
public part and rewrite `package.loaded` then.

This name overlap is confusing at reading the sources and complicates
debugging. And it conflicts with a built-in module loading logic that
will be implemented for #7774.

Let's use `foo.lib` for the internal part and `foo` for the public one.
This approach is already used in some built-in modules.

src/box/lua/upgrade.lua requires src/box/lua/xlog.lua, so changed the
loading order.

Eliminated extra `internal` field in `uri.lib`, because the whole module
is internal.

Part of #7774

NO_DOC=user visible behavior is unchanged, pure refactoring change
NO_TEST=see NO_DOC
NO_CHANGELOG=see NO_DOC

The idea of the refactoring is described in a previous commit.

The change is straightforward:

* Use `luaL_findtable()` to find/create a nested table in `_G`.
* Use `luaL_setfuncs()` to assign functions to a table.
* Use `luaT_newmodule()` to register a module for `require()`.

`require('box.internal.<...>')` calls are replaced with
`box.internal.<...>`, because `box` submodules are not registered as
modules anymore (IOW, are not placed into `package.loaded`).

However `_G.box == require('box')`, it remains unchanged.

One may wonder why fourth argument of `luaL_findtable()` (size hint for
the hash part of the table) is always zero in the added calls. It is an
effect of the following points:

* Those calls are not on hot paths and, moreover, there is no sense to
  use `luaL_findtable()` with a string literal as the second argument on
  a hot path (no sense to parse the string literal in runtime if a
  performance is the goal).
* Using `lengthof()` macro would add more code to read for a human (the
  argument itself, a line carry to fit 80 symbols, an inclusion of
  trivia/util.h) at no gain.
* A hardcoded array length is a developer's stress factor: it looks like
  it requires an update at a change of the array. So zero is better.

Part of #7774

NO_DOC=user visible behavior is unchanged, pure refactoring change
NO_TEST=see NO_DOC
NO_CHANGELOG=see NO_DOC

See the previous commit regarding the idea behind this refactoring.

The src/box/lua part is updated in a next commit, only `src/lua` is
updated here. It is split to ease reading.

The patch is mostly straightforward, I'll highlight several interesting
places.

* `compat`

  `_G.internal.compat` was created by `luaL_register()`. Removing of
  this call drops `_G.internal`: it is not created anymore.

* `fiber.channel` and `fiber.cond`

  The initialization code add fields into fiber module. So we call
  `require('fiber')` here.

  The alternative is to access `package.loaded.fiber` directly, however
  I plan to eliminate such accesses in further patches. (Built-in
  modules registration process will use its own table instead of
  `package.loaded`.)

* `http.client`

  Here we also eliminate a direct access to `package.loaded`, which is
  important for implementing #7774.

  Dropping of the `package.loaded['http.client']` field in unnecessary,
  because it is anyway replaced by the resulting module table in
  `tarantool_lua_init()`.

* `utf8`

  This module is also set to `_G.utf8` for compatibility with Lua 5.3.
  It remains unchanged.

Part of #7774

NO_DOC=user visible behavior is unchanged, pure refactoring change
NO_TEST=see NO_DOC
NO_CHANGELOG=see NO_DOC

This commit starts a series, which refactors built-in module
registration mechanisms. The series aims several goals:

* Offer a standard tarantool specific way to register a built-in module.
  It allows to change the registration process for all built-in modules
  at once.
* Make the API of such functions simple and clean.
* Eliminate known problems of the existing approach to register built-in
  modules.

This particular commit offers the new `luaT_newmodule()` function. It
replaces `luaL_register()` and `luaL_register_module()` to some extent.

Here I also start a practice to use `luaL_setfuncs()` instead of
`luaL_register(L, NULL, funcs)` for readability purposes. The function
is part of Lua 5.2 ([documentation][1]), but it is implemented in
LuaJIT.

Let's look on examples below.

## How it works now

First of all, consider the [documentation][2] about the
`luaL_register()` function. Then look on the examples.

```c
luaL_register(L, "foo.bar", funcs);
```

Creates a table `package.loaded['foo.bar']` and `_G.foo.bar` (if neither
of them exists), fill it with the functions and pushes it to the stack.

What is not ok:

* Pollutes `_G`.
* `package.loaded` is written flat, but `_G` is written deeply.
* No control over (un)intended module rewritting.

It also worth to note that we can't change this function, because it is
part of the Lua API. So we need another function for our registration
process in order to implement #7774.

Another usage of the function makes it even more confusing:

```c
luaL_register(L, NULL, funcs);
```

Does it create a table, fill it with the functions and push it to the
stack? No.

Unlike other usages, it fills a table on the top of the stack with the
functions and leaves it on the stack. And it actually has no relation to
module registration.

There is tarantool specific function `luaL_register_module()`, which
looks as an attempt to solve some of the problems.

```c
luaL_register_module(L, "foo.bar", funcs);
```

It doesn't touch `_G` and, surprisingly, changes how a module is written
to `package.loaded`.

The call creates a table (or finds it) in `package.loaded.foo.bar`,
fills it with the functions and pushes to the stack. *This* function
writes `package.loaded` deeply.

It leaves us with `luaL_register()` for assigning `_G.foo.bar` (with
pollution of `package.loaded`). It leaves us with `luaL_register()` for
setting functions into a table on the stack (with confusing name in this
context and confusing contract). And we has no a function to just assign
`package.loaded[modname]` without the deep tables creation and with
appropriate checks.

## How it will work

```c
luaT_newmodule(L, "foo.bar", funcs);
```

Create a table, set it to `package.loaded['foo.bar']` and push into the
stack. Assert that the table doesn't exist.

```c
luaL_setfuncs(L, funcs, 0);
```

Add functions into a table on top of the stack.

```c
luaL_findtable(L, LUA_GLOBALSINDEX, "foo.bar", 0);
luaL_setfuncs(L, funcs, 0);
```

Find/create a table `_G.foo.bar` and add functions to it.

Next commits will adopt built-in modules to follow this practice.

[1]: https://www.lua.org/manual/5.2/manual.html#luaL_setfuncs
[2]: https://www.lua.org/manual/5.1/manual.html#luaL_register

Part of #7774

NO_DOC=user visible behavior is unchanged, pure refactoring change
NO_TEST=see NO_DOC
NO_CHANGELOG=see NO_DOC

Generate a test build key when the ALPINE_BUILD_KEY secret is empty.
It is needed for fork PRs where secrets are unavailable.

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

Since applied transactions are committed asynchronously, they are
inherently 'read committed', hence their isolation level should be adjusted
accordingly.

Closes #8121

NO_DOC=bugfix

XLOG_READ_AHEAD defines the number of bytes added to the `count' argument
of the `pread' syscall, currently it equals to 16 KB. However xlog and snap
files are written by 128 KB chunks of data, which turn into ~80 KB chunks
after compression (in average, depending on the data), so the 16 KB read-
ahead doesn't make any sense.

According to performance experiments, 8 MB readahead gives the best results
for large files. However, 8 MB read buffers would increase memory
consumption of the replication relays, which usually read small portions of
data and does not need such a big buffers. For this reason, dynamically-
sized read buffer is implemented by this patch. The minimal readahead is
now 128 KB, and the maximal is 8 MB. As a result, the recovery time of a
900 MB snapshot decreased from 25 sec to 13 sec (when reading from HDD).
Performance of a recovery from SSD is not affected.

Closes #8108

NO_DOC=performance improvement
NO_TEST=performance improvement