Currently replicaset state machine tracking the number of connected,
loading and synced appliers may perform unnecessary decrementing of
their count. On debug version this may lead to assertion failure.
Here's the way it may happen:
  1. Any kind of exception occurs in applier thread and leads to
     invoking its destructor (applier_thread_data_destroy), which
     is set with scoped guard;
  2. Cbus call is made in order to remove the corresponding applier
     from the thread. According to the fact that cbus_call is
     synchronous, we yield, waiting for the result from the applier
     thread.
  3. During yielding user calls reconfiguration, which invokes
     replicaset_update. Old appliers are pruned: for every replica
     trigger on changing state machine counter is deleted after which
     we stop fiber and wait its join.
  4. If the first replica in replicaset_foreach is not the errored
     one and the errored fiber wakes up during yielding with
     fiber_join, then zero decrementing happens.

Let's clear the above mentioned triggers for all replicas at the
first place and only after that stop and join their applier fibers.

Closes #7590

NO_DOC=bugfix

Before the change there was an unexpected behavior when using
channel:close(), as it closed the channel entirely and discarded all
unread events.

This commit introduces graceful channel close option in tarantool.compat
(gh-7000) that allows to select new or old behavior.

With the new behavior `close()` marks channel as closed for writing.
Only when all events are extracted, the channel is closed entirely. If
there are no events in the channel, it is closed as usual.

Document that describes new API can be found on notion (private):
https://www.notion.so/fiber-channel-graceful-close-53b2788ed1f144598c4c0e1229c2eb69

Requires #7060
Requires #8007
Closes #7746
See also #7000

@TarantoolBot document
Title: new compat option fiber_channel_close_mode

New behavior is gracefully closing fiber channel by marking it
read-only, instead of destroying. Full API description can be found
on notion (private):
https://www.notion.so/tarantool/Fiber-channel-graceful-close-53b2788ed1f144598c4c0e1229c2eb69

A jiggle in tarantool's initialization code can lead to trace numbers
above 9. We can either:

* accept trace numbers above 9 in the test
* or drop traces from the initialization code in the test and assume
  that the new trace will have number 1.

Igor Munkin suggested to stick with the second approach to avoid
dependency on the initialization code.

NO_DOC=test fixup, no user visible changes
NO_CHANGELOG=see NO_DOC

Co-authored-by: Igor Munkin <imun@tarantool.org>

Currently all non-main threads have all the signals blocked, however
according to `man pthread_sigmask':

> If SIGBUS, SIGFPE, SIGILL, or SIGSEGV are generated while they
> are blocked, the result is undefined, unless the signal was
> generated by kill(2), sigqueue(3), or raise(3).

On macOS they are actually blocked, causing the faulting instruction
to loop indefinitely. While on Linux they are not blocked, however the
signal handler registered by sigaction is not executed. Don't block them.

Closes #8023
Closes #8083

NO_DOC=bugfix

There are 4 fatal signals that cannot be blocked by sigmask if they
are caused by the CPU exception (rather than kill, sigqueue or raise):
SIGILL, SIGBUS, SIGFPE, SIGSEGV. Currently the crash module handles
only SIGSEGV and SIGFPE. This patch adds handlers for SIGBUS and SIGILL.
SIGBUS is usually raised by macOS on access to the unmapped memory, and
SIGILL is possible, for example, while running AVX version of memcpy on
a CPU without AVX support.

Faulting address siginfo->si_addr is valid for all these signals, so
print it unconditionally.

Part of #8023
Part of #8083

NO_DOC=See next commit
NO_CHANGELOG=See next commit

Currently, core constraints are dropped on commit. That is why
it is impossible to drop constraint and drop objects it references to
at the same transaction. Let's drop constraints in two steps - detach
them when DDL occurs, then reattach on rollback or delete on commit.

Closes #7339

NO_DOC=bugfix

NO_TEST=no visible changes
NO_CHANGELOG=stubs
NO_DOC=stubs

The patch adds xalloc helpers for ibuf.

NO_TEST=trivial
NO_CHANGELOG=internal
NO_DOC=internal

This patch removes code that was used to implement the SQL check
constraint as they are now replaced by BOX constraint. Also, the syntax
for enabling/disabling check constraints has been removed as BOX
constraints do not support this feature.

Follow-up #6986

NO_DOC=Already introduced.
NO_CHANGELOG=Already introduced.

This patch removes code that was used to implement the SQL foreign key
as they are now replaced by BOX foreign keys.

Follow-up #6986

NO_DOC=Refactoring.
NO_TEST=Refactoring
NO_CHANGELOG=Refactoring.

- Add box.cfg.password_history_length configuration option.
  It will specify the max number of entries to keep in the
  auth_history field of the _user system space.
- Add new error code ER_OLD_PASSWORD, which will be raised on
  an attempt to reuse an old password.
- Set auth_history if box.internal.prepare_auth_history is defined.
  The function takes a user id.
- Add auth_history argument to box.internal.check_password so that
  it can check if the new password matches an old one.
- Add box_lua_security_init, because auth checks will be done in C.

Needed for https://github.com/tarantool/tarantool-ee/issues/298

NO_DOC=ee
NO_TEST=ee
NO_CHANGELOG=ee

See the doc bot request for the description of the new fields.

Note that we only store the value of the 'last_modified' field
in struct user_def, because 'auth_history' will be used only in
Lua code.

Needed for https://github.com/tarantool/tarantool-ee/issues/298
Needed for https://github.com/tarantool/tarantool-ee/issues/299

NO_CHANGELOG=no user-visible effects in CE; will be added to EE

@TarantoolBot document
Title: Document auth_history and last_modified _user space fields

Field name: auth_history. Field no: 6. Type: array.
Description: The field stores an array of previous authentication data:
when a user password is changed, the last value of the 'auth' field is
appended to 'auth_history'.  The length of the history is configured by
the `box.cfg.password_history_length` option, which is available only
in Tarantool EE, where it's used to prevent users from reusing old
passwords. In Tarantool CE, the array is always empty.

Field name: last_modified. Field no: 7. Type: unsigned.
Description: The field stores the timestamp (seconds since Unix epoch)
of the last user password update. It's never used in Tarantool CE.
In Tarantool EE, it's used to disable users that haven't changed the
password for more than `box.cfg.password_lifetime_days`.

`box.schema.upgrade()` sets the new field values to an empty array
and 0 for users that haven't updated them yet.

We disable system space triggers to generate a bootstrap snapshot,
but this doesn't turn off space format checks. As a result, if
a space format is updated during the upgrade sequence, it may turn
out that data inserted earlier doesn't match the new format.
We encountered this issue only once, when we added new fields to
the _func system space, see commit 200a492a ("box: introduce Lua
persistent functions"). Back then, we simply added the new fields
to the earlier update function. However, modifying the update history
looks ugly. We're planning to add new fields to the _user system
space so we should find a way to avoid that.

The fix is simple: we clear all system space formats before disabling
system space triggers and reset them back after the bootstrap is done.
To achieve that, we have to eliminate usage of any functions that access
tuple fields by name.

Needed for https://github.com/tarantool/tarantool-ee/issues/298
Needed for https://github.com/tarantool/tarantool-ee/issues/299

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Use O_CLOEXEC flag instead.

If Tarantool is forked before executing box.cfg{}, e.g. using io.popen(),
the child process could start with stdin linked to /dev/urandom.
This happens because wal_writer_singleton and vy_log_writer are not yet
initialized, i.e. `fd' fields are 0, then atfork child handler wal_atfork()
is called. It checks that xlog is opened (the check succeeded as 0 != -1)
and closes its fd 0, in fact closing stdin (twice). Next, Tarantool opens
the file /dev/urandom during initialization, and it receives the lowest
unused file descriptor, which is 0. Then luaL_loadfile() loads stdin as a
Lua chunk, in effect reading random numbers.

This happens on glibc 2.28 and older, as newer versions do not invoke
atfork handlers during io.popen():
https://sourceware.org/bugzilla/show_bug.cgi?id=17490

Closes #7886

NO_DOC=bugfix
NO_TEST=Tested by test/app-luatest/gh_5747_crash_multiple_args_test.lua

This patch fixes an issue where collation could change the computed
type of a built-in function argument when choosing a function
implementation.

Closes #7992

NO_DOC=bugfix

Workflow submodule_update now brings latest commits in stable branches
further to tarantool/sdk, where integration tests can run with other
enterprise products and modules.

In tarantool/tarantool-ee a pull request is still opened, so the
maintainers can merge it at any time.

In tarantool/sdk the test branches are named in the following pattern:
`TarantoolBot/test-ce-master`
`TarantoolBot/test-ce-2.10`
and so on. This pattern can be easily distinguished by workflows,
which run integration tests and alert developers on failures.
It is also consistent with branches with the same purpose of integration
testing, that are created by the sdk_test workflow:
`TarantoolBot/test-ce-{feature_branch_name}`

NO_DOC=CI
NO_TEST=CI
NO_CHANGELOG=CI

Resolves tarantool/sdk#355

The submodule_update workflow only works with git repositories and
doesn't require computational power. It's mostly limited by the network
throughput. So it makes sense to run this workflow on lightweight
runners, labeled with `flavor-1-2` (that is, 1 CPU and 2Gb RAM).

New runners have granular labels: `self-hosted`, OS, arch and flavor.

NO_DOC=CI
NO_TEST=CI
NO_CHANGELOG=CI

The test sends SIGTERM to the process while it is writing a
snapshot. Using `g.server.process:kill('TERM')`.

On a next line the test was doing `g.server:stop()` which also
sends SIGTERM. Apparently, the second SIGTERM sometimes killed the
server immediately.

It usually happened inside fiber_free(). Sometimes right after the
file tt_exit_file.txt.inprogress was created but before it was
renamed to tt_exit_file.txt.

The reason is in signal_free() being called quite early. After
that any signal becomes fatal and is not caught by the handlers.

The patch also increases on-shutdown timeout not to depend on it
for certain. Although it shouldn't be a problem right now.

Follow-up #7743

NO_DOC=test
NO_CHANGELOG=test

This patch refactors the SQL memory allocation system. There are three
main changes:
1) now, when allocating memory, no additional 8 bytes are allocated to
remember the size of the allocated memory, so instead of
sql_malloc()/sqlRealloc()/sql_free(), the malloc()/realloc()/free()
functions are used;
2) the malloc()/realloc() functions were used through the
xmalloc()/xrealloc() macros, so checks for memory allocation errors were
removed;
3) there is no need for an explicit "sql *db" argument for most of the
functions, so it has been omitted.

Part of #1544

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

This commit adds:
 - Call to box.internal.check_password in box.schema.user.passwd and
   box.schema.user.create to check the provided password. The function
   will be implemented in EE.
 - A set of new box configuration options that specify how a password is
   checked. They are available only if box.internal.check_password is
   defined, i.e. in EE.
 - A new error code that will be raised if the password is weak.

Needed for https://github.com/tarantool/tarantool-ee/issues/297

NO_DOC=ee
NO_TEST=ee
NO_CHANGELOG=ee

Currently, all the EE configuration options are visible in CE, but an
attempt to set them either fails or ignored. Let's be consistent and
simply hide those options from CE, like we do with the feedback daemon.
The only problem here is that some tests print the whole config to the
result file: box/cfg, box/admin, app-tap/init_script. We fixed box/cfg
and box/admin tests by adding EE keys to the cfg_filter if tests are
run by a EE binary. This should make the output the same between CE and
EE. As for app-tap/init_script, since it doesn't use cfg_filter, we
removed printing of the whole config to the result file and instead
print just box.cfg.memtx_memory - it should be enough to check that
box.cfg is accessible from an init script.

Note, we use explicit 'if' statement in ifdef_XXX functions rather than
'a and b or c', because the latter doesn't work as expected if 'b' is
boolean false.

While we are at it, make cfg_getb helper return a boolean value instead
of integer, and false instead of -1 in case of nil. (I've no idea why
it was made to return -1 in case of nil in the first place.)

NO_DOC=undocumented
NO_CHANGELOG=undocumented

JIT has been disabled for these 3 tests on arm64 to avoid weird fails, but
they are still flaky in CI. Perhaps the JIT traces from other tests affect
them. Let's try to do jit.flush() together with jit.off().

Also disable JIT for pagination_netbox_test.lua, which suffers from the
same problem as pagination_test.lua.

Part of #6599
Part of #7739
Part of #8011

NO_DOC=testing stuff
NO_CHANGELOG=testing stuff

This patch fixes possible memleak.

NO_DOC=Possible bug
NO_TEST=No proper way to test
NO_CHANGELOG=Possible bug

If the authentication method isn't specified with the user credentials,
the client uses the one received in reply to IPROTO_ID. If it's unknown
to the client, it falls back on AUTH_METHOD_DEFAULT.

A test and a changelog will be added to the Enterprise Edition, which
supports more than one authentication method.

Part of #7989

NO_DOC=ee
NO_TEST=ee
NO_CHANGELOG=ee

If the authentication method isn't specified with the user credentials,
the client uses the one received in reply to IPROTO_ID. If it's unknown
to the client, it falls back on AUTH_METHOD_DEFAULT.

A test and a changelog will be added to the Enterprise Edition, which
supports more than one authentication method.

Part of #7989

NO_DOC=ee
NO_TEST=ee
NO_CHANGELOG=ee

Part of #7989

@TarantoolBot document
Title: Document IPROTO_AUTH_TYPE - new key of IPROTO_ID

A new key was added to the `IPROTO_ID` response:
name `IPROTO_AUTH_TYPE`, code 0x5b, value type `MP_STR`.

Its value equals the name of the authentication method that is
currently used on the server for generating user authentication
data from a password `box.schema.passwd()`. It's configured with
`box.cfg.auth_type`.

In Community Edition, the only authentication method currently
available is 'chap-sha1'. Other authentication methods may be used
in Enterprise Edition.

A client can use the received value as the authentication method
that should be used by default, in case it isn't specified explicitly
along with the user credentials.

We need it to retrieve the default authentication method that should be
used by the applier if not set explicitly in URI. In future, we may also
use it for other things, e.g. fetching the master's name.

The IPROTO_ID request is sent only if the master's version is >= 2.10.0.
An error received in reply to IPROTO_ID is logged, but doesn't abort
replication, because IPROTO_ID is optional.

Needed for #7989

NO_DOC=internal
NO_CHANGELOG=internal

The ./test/luatest_helpers/interactive_tarantool.lua module is not
a luatest helper. So moving it to the ./test/ dir and removing empty
./test/luatest_helpers/.

NO_DOC=testing stuff
NO_TEST=testing stuff
NO_CHANGELOG=testing stuff

Some internal modules have been recently copied to luatest repo [1] and
now they can be safely removed, and the corresponding functionality from
luatest can be used instead.

Affected modules:

- test/luatest_helpers/cluster.lua

[1] tarantool/luatest#271

Closes tarantool/luatest#237
Closes tarantool/luatest#269

NO_DOC=testing stuff
NO_TEST=testing stuff
NO_CHANGELOG=testing stuff

Bump test-run to new version with the following improvements:

- Bump luatest to 0.5.7-20-g48c406a [1]

[1] tarantool/test-run#366

Part of tarantool/luatest#237

NO_DOC=testing stuff
NO_TEST=testing stuff
NO_CHANGELOG=testing stuff

Before the patch, fiber->storage.lua.stack is used for
`panic` calls. However, some fibers don't have any Lua
state saved in their storage (for example, space
triggers).

After the patch, the Lua state pointed by `cur_L` is
used to make those calls, as it is always present.

Closes #6647
NO_DOC=bugfix

If an IO stream is encrypted, it should set flag IOSTREAM_IS_ENCRYPTED.
If an authentication method requires the channel to be encrypted, it
should set flag AUTH_METHOD_REQUIRES_ENCRYPTION. An attempt to use an
authentication method that requires encryption over an unencrypted IO
stream will raise an error ER_UNSUPPORTED("Authentication method '%s'
does not support unencrypted connection"). This check is performed by
both net.box and applier.

Needed for https://github.com/tarantool/tarantool-ee/issues/322

NO_DOC=ee
NO_TEST=ee
NO_CHANGELOG=ee

The only reason why libunwind.h is included into backtrace.h (and then
transitively into fiber.h) is use of unw_word_t type. Let's replace it
with uintptr_t and use unw_word_t only for interacting with the unwind
library.

This commit partially reverts commit 6d088b56 ("build: fix
libunwind.h search in test files"), because we don't need to
include libunwind.h into tests anymore.

Fixes #8025

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=changelogs/unreleased/fix-libunwind-h-search.md

Before commit 24323448 ("log: add log.new() function that creates a
new logger"), log_write_flightrec() was called from log_vsay() regardless
of the log level. After, the log level is checked in Lua, so log_vsay()
may not be called. This patch restores the original behaviour by moving
log_write_flightrec() calls to say_default() and say().

Part of https://github.com/tarantool/tarantool-ee/issues/320

NO_DOC=bugfix
NO_CHANGELOG=unreleased
NO_TEST=will be enabled in EE, because there are no flightrec in CE

If env vars like TT_LOG, TT_LOG_LEVEL etc that specify log options are set
then log.cfg() should take them into account if called before box.cfg().

Closes #6011
Closes #7461

@TarantoolBot document
Title: log: take into account env vars on log.cfg call
Since: 2.11

If env vars like TT_LOG, TT_LOG_LEVEL etc that specify log options are set
then log.cfg() take them into account if called before box.cfg().

The 'copy' ptr is checked for NULL after dereferencing. Found by PVS.

https://jira.vk.team/browse/TNT-698

It can hardly result in a null-ptr dereference, because vy_stmt_dup()
uses malloc(), which doesn't normally fail (instead, the OOM killer just
kills the process if there's a severe shortage of memory, see #3534).
Still, PVS rightfully complains about it so let's fix it.

Fixes commit 902d212b ("vinyl: zap vy_write_iterator->format")

NO_DOC=bug fix
NO_TEST=unreproducible
NO_CHANGELOG=unreproducible

This commit adds function stubs security_init and security_free that
are called after auth_init and before auth_free, respectively. We'll
define them in the EE repository along with the enabling macros:
ENABLE_SECURITY and SECURITY_SOURCES. We'll start with adding extra
authentication methods in security_init. Later on, we'll implement more
security features, like password strength enforcement or authentication
delay.

Needed for https://github.com/tarantool/tarantool-ee/issues/295

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

After checking that the received authentication method and type are
compatible with auth_request_check(), authenticate() tries to
authenticate it using the user's authentication method with
authenticate_request(). The problem is the user may use a different
authenticate method from the one received in the request while
authenticate_request() expects the request to be valid. As a result,
it may crash in this case. Fix this by ensuring that the user's
authentication method matches the one received in the request.

Follow-up commit b5754d3f ("box: make auth subsystem pluggable")
Follow-up #7986

NO_DOC=bug fix
NO_CHANGELOG=unreleased
NO_TEST=will be added to EE, because CE supports just one auth method

The authentication method can be specified via the 'auth_type' uri
parameter, e.g.

  box.cfg({replication = 'user:password@host:port?auth_type=chap-sha1'})

The only authentication method supported by Community Edition (CE)
is 'chap-sha1' so we don't document or announce this feature in CE.
More methods and tests will be added to Enterprise Edition (EE).

Part of #7988

NO_DOC=ee
NO_CHANGELOG=ee

This commit adds a new option for net.box.connect - auth_type.
The option takes an authentication method name (string) that will
be used to authenticate the client on connect. Like user name and
password, the new option may also be passed via uri parameters, e.g.

  net.connect('host:port', {
      user = 'user',
      password = 'password',
      auth_type = 'chap-sha1',
  })
  net.connect('user:password@host:port?auth_type=chap-sha1')

The only authentication method supported by Community Edition (CE)
is 'chap-sha1' so we don't document or announce this feature in CE.
More methods and tests will be added to Enterprise Edition (EE).

Part of #7988

NO_DOC=ee
NO_CHANGELOG=ee