In some cases unsafe extension decoding was done without bound and type
checks: add necessary checks.

Closes tarantool/security#73

NO_DOC=bugfix

(cherry picked from commit 1de6a071)

This patch addresses coverity complain 1535241.

Follow-up #8047

NO_TEST=nit
NO_CHANGELOG=nit
NO_DOC=nit

(cherry picked from commit 089cbfa9)

If the 'after' key is less than the search key in case of ge/gt or
greater than the search key in case of le/lt, the iterator either
crashes (vinyl) or returns invalid result (memtx). This happens because
the engine implementation doesn't expect an invalid 'after' key.
Let's fix this by raising an error at the top level in case the 'after'
key doesn't meet the search criteria.

Closes #8403
Closes #8404

NO_DOC=bug fix
NO_CHANGELOG=unreleased

(cherry picked from commit c561202d)

Currently, if the position isn't compatible with the index, we raise
an error like "Invalid key part count ...". From this error it's
difficult to figure out whether it's for the given iterator position of
for the search key. Let's always raise ER_ITERATOR_POSITION in this
case. Later on we'll use stacked diag to add extra error info.

Needed for #8403
Needed for #8404

NO_DOC=bug fix
NO_CHANGELOG=unreleased

(cherry picked from commit 81d43c17)

We need to compare a tuple position with a search key in select() and
pairs() to make sure the tuple position meets the search criteria. The
problem is that we strip the MessagePack header from the position while
key_compare() takes keys with headers. Let's make key_compare take keys
without headers like the rest of comparator functions. Since in Vinyl we
often need to compare keys with headers, we also add vy_key_compare()
helper function.

Needed for #8403
Needed for #8404

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

(cherry picked from commit 41b8a012)

To improve customer experience it was decided to disable JIT engine on
Tarantool startup for macOS builds. Either way, JIT will be aboard as a
result of the changes and more adventurous users will be able to enable
it via <jit.on> in their code.

Furthermore, for convenient maintenance of JIT default behaviour CMake
configuration option "LUAJIT_JIT_STATUS" is introduced.

Closes #8252

NO_DOC=no behaviour changes

(cherry picked from commit ae0db476)

If the tuple pointed to by 'after' isn't in the space, eq pagination
would skip it because of the broken 'equals' flag update in the memtx
iterator.

Closes #8373

NO_DOC=bug fix
NO_CHANGELOG=unreleased

(cherry picked from commit bf275ed5)

Refactor logging of dangerous `select` call out of `check_select_safety`
and add it to `box.internal` so that it can be reused for read views.

Change all occurrences of 'dangerous' to 'long' to be consistent with the
log message.

Needed for tarantool/tarantool-ee#211

NO_CHANGELOG=<refactoring>
NO_DOC=<refactoring>
NO_TEST=<refactoring>

(cherry picked from commit 4221a983)

The `log_level' configuration parameter can be set as a number or a string.
When it is a string, cfg_geti() returns 0. Use log_default->level instead,
which is initialized earlier during box_init_say().

Closes #8287

NO_DOC=bugfix
NO_CHANGELOG=minor bug

(cherry picked from commit 41ead021)

This patch fixes an assertion or segmentation error if a FOREIGN KEY or
CHECK constraint is declared before the first column.

Closes #8392

NO_DOC=bugfix of the bug added in the current release
NO_CHANGELOG=bugfix of the bug added in the current release

We use coio_connect() to connect the replica to a remote peer. It
implies no timeout, and does a non-blocking connect() to the peer and
then waits for the socket to become writable indefinitely.

When the remote peer changes its IP address, connect() might try
connecting to the old address for as long as ~ 2 minutes (given the
default tcp_syn_retries value of 6).

This blocks replica from trying to reconnect to the updated address and
is pretty inconvenient.

Let's use coio_connect_timeout() instead and use
replication_disconnect_timeout() as a timeout, like everywhere else in
master-replica communication.

Closes #7294

NO_DOC=bugfix

(cherry picked from commit 0486a489)

Force recovery needs to follow the following logic: any unsuccessful system
space request must make recovery fail (including failure to decode an xrow,
when we are not sure we have finished processing system space request). If
the request is a non-insert one (e.g., raft or synchro) or addresses a user
space, it means we have finished processing system space requests, and from
this moment force recovery can be enabled — change the behaviour
accordingly.

We assume the request order in the snapshot is the following:
1. system space requests;
2. user space requests;
3. non-insert requests (e.g., raft or synchro).

Refactor the force recovery logic: add a enumeration to track snapshot
recovery state and add a new diagnostic for the case when the snapshot
contains has no system spaces.

Closes #7974

NO_DOC=bugfix

(cherry picked from commit b1095c1c)

In some cases we don't have the whole space struct, but we want to
determine whether the provided space identifier corresponds to a system
space — add a `space_id_is_system` helper and refactor `space_is_system` to
reuse it.

Needed for #7974

NO_CHANGELOG=refactoring
NO_DOC=refactoring
NO_TEST=refactoring

(cherry picked from commit 928e5733)

Patch fixes a bug when body in response couldn't be decoded:

NO_WRAP
```
tarantool> httpc = require('http.client').new()
tarantool> response = httpc:get('https://jsonplaceholder.typicode.com/todos/1'

)
tarantool> response:decode()

---
- error: 'builtin/http.client.lua:301: attempt to index field ''decoders'' (a nil
    value)'
...
```
NO_WRAP

Now response object contains table with decoders defined by user in his
http client instance. We hide this table on response serialization by
adding underscore because decoders there is not a part of API.

Reported-by: Alexander Turenko <alexander.turenko@tarantool.org>

Fixes #8363

NO_DOC=bugfix

(cherry picked from commit 83168b25)

Error labes got mixed up. Fix the order and add a test.

Fixes commit 3f026339 ("vinyl: implement iterator pagination").

Closes #8372

NO_DOC=bug fix
NO_CHANGELOG=fix for unreleased feature

(cherry picked from commit 074e4eb8)

When calculating `rows_index` in `netbox_decode_execute` etc we use
implementation defined behavior of compiler. `i < mapsize` and this is
fine according to standard as unsigned arithmetic is well defined on
overflows. But then we cast the result to int and this is implementation
defined as the result can not be represented by int.

Closes https://github.com/tarantool/security/issues/98

NO_DOC=minor
NO_TEST=minor
NO_CHANGELOG=minor

(cherry picked from commit baba38e6)

The new compat option 'fiber_slice_default' is added to control
the default value of the max fiber slice. The old default is no limit
(both warning and error slice equals TIMEOUT_INFINITY). The new default
is {warn = 0.5, err = 1.0}.

Follow-up #6085

NO_DOC=tarantool/doc#3057
NO_CHANGELOG=unreleased

This commit adds the new field 'max_slice' to fiber.info() that reports
the max slice applied to the given fiber. The value is a table with two
fields: 'err' for error slice and 'warn' for warning slice. Values
greater than or equal to TIMEOUT_INFINITY aren't reported.

Follow-up #6085

NO_DOC=tarantool/doc#3057
NO_CHANGELOG=unreleased

Error messages raised when an invalid slice is specified are confusing:

  tarantool> fiber = require('fiber')
  ---
  ...

  tarantool> fiber.set_max_slice('foo')
  ---
  - error: slice must be a table or a number
  ...

  tarantool> fiber.set_max_slice({})
  ---
  - error: 'bad argument #3 to ''?'' (number expected, got nil)'
  ...

Let's change the error message to "slice must be a number or a table
{warn = <number>, err = <number>}".

Follow-up #6085

NO_DOC=undocumented
NO_CHANGELOG=unreleased

It doesn't belong to Tarantool sources. The tools directory looks like
the right place for it.

NO_DOC=code cleanup
NO_TEST=code cleanup
NO_CHANGELOG=code cleanup

This patch allows to downgrade tuple foreign keys to SQL foreign keys
and some tuple check constraints to SQL check constraints. The only
tuple check constraints that can be downgraded are those using functions
with the SQL_EXPR language.

Closes #7718

NO_DOC=already introduced
NO_CHANGELOG=already introduced

Added `-d` option for activation of debugger shell:
- it calls debugger shell in `luadebug.lua` instead of
  a standard interactive shell from `console.lua`;
- that option complements original way for starting a
  debugging shell via `require 'luadebug'()`, but is a
  little bit easier.

NB! At the moment when we enter debugging mode instead of
a standard Tarantool console, we change banner to:
```
Tarantool debugger 2.11.0-entrypoint-852-g9e6ed28ae
type 'help' for interactive help
```

Part of #7456

@TarantoolBot document
Title: Command-line option `-d` for console debugger.

Please see third_party/lua/README-luadebug.md for a full description
of different ways to activate debugging shell.

2.10.5 version is going to be released at the same time as 2.11.0 thus
let's add it to the box.schema.downgrade_versions().

NO_DOC=internal
NO_CHANGELOG=internal
NO_TEST=internal

Currently it's possible to set the log level for a particular Lua module,
or to change the default log level, however there is no way to change it
only for Tarantool system messages. This patch introduces a fake module
name "tarantool" for this purpose.

Closes #8320

NO_CHANGELOG=Unreleased feature
NO_DOC=Will update tarantool/doc#3264

Add a streaming data input/output object for http.client. The
input/output object can be created using the same methods and the
same options as a normal request, but with a new option
{chunked = true}.

Closes #7845

@TarantoolBot document
Title: Stream input/output interface for http.client

An uncompleted io object has only headers and cookies fields. A
completed io object has the same fields as a normal request, but
without the `body` field.

The io object interface looks like the socket object interface
and should have the same description:

```
io_object:read(chunk[, timeout])
io_object:read(delimiter[, timeout])
io_object:read({chunk = chunk, delimiter = delimiter}[, timeout])
io_object:write(data[, timeout])
```

The difference is in the method `finish`. Unlike socket:close()
it has an optional parameter `timeout`:

```
io_object:finish([timeout])
```

Be careful, the call may yield a fiber. The idea is to wait
until a HTTP connection is finished by the server-side or
force finish the connection from client-time after a timeout
value.

The default timeout value is 10 seconds for all methods.

Usage example:

```lua
local io = httpc:post(url, nil, {chunked = true})

local write_chan = fiber.channel()
fiber.new(function()
    fiber.name("write to " .. url)
    while true do
        local data = write_chan:get()
        if data == nil then
            break
        end
        io:write(data, 1)
    end
end)

local recvdata
while recvdata = io:read('\r\n', 1) do
    local decoded = json.decode(recvdata)
    if condition(decoded) then
        write_chan:put(data)
    end
    if condition(decoded) then
        io:finish(1)
    end
end
write_chan:close()
```

See also:
* https://www.tarantool.io/en/doc/latest/reference/reference_lua/socket/#lua-function.socket_object.read
* https://github.com/tarantool/tarantool/issues/7845#issuecomment-1298538412
* https://github.com/tarantool/tarantool/issues/7845#issuecomment-1298821779

We have a code that performs calculations to obtain the address from the
given index, the address is then passed to cdataV() to get cdata value from
the stack. But this doesn't work for pseudo-indexes (e.g. upvalue indexes).
This patch brings index2adr() from luajit/src/lj_api.c, which accepts all
kinds of indexes, so that the calculations are no longer needed. Also the
helper function luaL_tocpointer() is introduced.

Closes #8249

NO_DOC=bugfix

The built-in module registration process performs several assertions:

- luaT_newmodule() checks for attempts to register an already registered
  module.
- luaT_setmodule() does the same, but it allows the same value be
  'registered' several times.

Attempt to register different values as the same built-in module is
definitely an error. Once a module is registered, it may be required and
captured. If its value is changed afterwards, different parts of
tarantool's code will capture different values for the same built-in
module. It is very unlikely that such effect may be intended.

Anyway, tarantool's code doesn't do this. However, now, after
implementing the override modules feature, it is possible to influence
tarantool's initialization from a user provided code.

It means that we can't assume the situations described above as
impossible ones. A developer of an override module should receive an
error if the code of the module leads to such a situation.

`assert()` works only in Debug build, but most of user's code is tested
on RelWithDebInfo builds. So incorrect override module may be
implemented and distributed. It is undesirable.

Let's replace `assert()` with explicit `panic()`.

Adjusted testing helpers:

* `treegen` now allows to just write given script into given directory
  without using templates. Sometimes it is more convenient.
* `justrun` can return pure stdout (without JSONL decoding) and can
  return stderr if requested.

`nojson` option in `justrun` may look unneeded for given test cases, but
it provides better diagnostics if something going wrong and one of the
test cases fails (say, if tarantool runs successfully instead of going
to panic).

Follows up #7774

NO_DOC=The override modules feature is not released yet.
NO_CHANGELOG=see NO_DOC

Use case: an override module is designed to replace a built-in module
only when the module is required from an application, not from the
platform itself.

Now it is possible:

```lua
-- override/fio.lua
local loaders = require('internal.loaders')

if loaders.initializing then
    loaders.no_package_loaded.fio = true
    return loaders.builtin.fio
end

return {
    whoami = 'override.fio',
}
```

Follows up #7774

NO_DOC=Planned as an internal API. See the previous commit.
NO_CHANGELOG=see NO_DOC

The flag indicates, whether the loading of built-in modules is finished.
May be useful for override modules to determine, whether it is required
from inside tarantool or from application's code.

Follows up #7774

NO_DOC=It is not intended to be officially supported. I would prefer to
       document ability of loading a replacement module itself, but I
       wouldn't document how to write such a module.
NO_CHANGELOG=see NO_DOC

It eliminates a need to call `package.setsearchroot()` in the main
script to find modules in the application's directory.

Such layout is typical for cartridge based applications. It it also
used, when different versions of modules are in use in different
applications and installing the modules into the system is not an
option.

Regarding the implementation. There are two possible ways to add more
searching logic: add more entries into `package.path` and
`package.cpath` or add/wrap a loader. I'm going the second way, because
we already have a logic around .rocks and override modules implemented
as loaders. It is easier to follow.

However, I would reimplement all those loaders as paths generators for
`package.path` and `package.cpath` in a future. See also #3136.

Follows up #7774
Fixes #8182

@TarantoolBot document
Title: Default module search paths now include the main script directory

Let's assume that there is an application in the directory
/path/to/myapp with modules inside:

```
+ /path/to/myapp/
  +- .rocks/share/tarantool/
     +- foo.lua
  +- init.lua
  +- myapp/
     +- bar.lua
```

Now the following command automatically adds the path to the main script
directory to search paths for modules:

```sh
$ tarantool /path/to/app/init.lua
```

So `require('foo')` and `require('myapp.bar')` works without any
additional work.

To be used in loaders, see the next commit.

While I'm here, actualized `tarantool_lua_init()` description.

Follows up #7774
Part of #8182

NO_DOC=for internal use
NO_CHANGELOG=see NO_DOC
NO_TEST=see NO_DOC

A script for preparing tarantool for executing LuaJIT submodule test
suites has a `debug.getupvalue()` call. It is fragile and the previous
commit promises that it'll be rewritten. It is done here.

The override loader can be disabled using an internal API or by the
`TT_OVERRIDE_BUILTIN` environment variable. The latter is not used
anywhere at the moment of writing and added just in case.

Follows up #7774

NO_CHANGELOG=not planned to be a documented feature, just a tiny
             workaround for specific tests
NO_DOC=see NO_CHANGELOG

This commit finishes implementation of the logic for overriding built-in
modules. It adds the loader, which looks into a file system for
`override.foo` module when `foo` is required. It works as for
`require()` in an application/module code as well as for `require()`
called from tarantool during its initialization.

Added replacements into the treegen testing helper.

Disabled the override loader in the LuaJIT submodule test suites. It is
done using `debug.getupvalue()`, which is a fragile way. Don't worry:
the next commit reimplements this logic.

Fixes #7774

@TarantoolBot document
Title: Built-in modules overriding

Now it is possible to override a built-in module. Everything is simple
here, in fact: if `override.foo` module exists on default search paths,
it'll replace built-in `foo` module.

We're going to provide an ability to override a built-in module. The
code of an override module will work in a bit unusual circumstances: a
part of tarantool's Lua runtime is initialized, but a part isn't.

However, nothing prevents us from providing the `arg` global variable at
this stage. It used to be accessible anywhere in a user defined code.
Let's keep this property by making it accessible from an override
module.

Part of #7774

NO_TEST=It will be tested as part of the override feature.
NO_CHANGELOG=It is not possible to execute any user provided code at
             this loading stage until modules overriding will be
             implemented. So this commit doesn't change any behavior
             that a user might observe before this commit.
NO_DOC=It spreads exiting runtime guarantee to a new stage, where a user
       defined code may appear. Nothing new is introduced.

It is necessary to implement overloading of a built-in module by an
external one. `require('foo')` will work this way: check for
`override.foo` module on the file system and then load built-in `foo`.

If `foo` is already assigned to `package.loaded`, the loaders will not
be called at all, so we can't check the file system.

This commit changes a built-in module registration process: the modules
are assigned into another table called `loaders.builtin`. A special
loader is added to `package.loaders` to look into this table.

Comments on the luajit-test-init.lua changes:

* The LuaJIT tests use its own tap module, so the script unloads the
  built-in tap module. However, now it is not enough to remove it from
  `package.loaded` (it'll be loaded again at next require()). It should
  also be removed from `loaders.builtin`.

  Maybe it would be better to move this external tap module to
  <...>/override/tap.lua in a future.
* `package.loaded` may miss `internal.print` and `internal.pairs` if
  they were not required. The right way to obtain the modules is to
  require() them.

Part of #7774

NO_TEST=no user visible changes
NO_CHANGELOG=see NO_TEST
NO_DOC=see NO_TEST

Further commits will change how built-in modules are registered, so a
direct assignment to `_LOADED` (`package.loaded`) is not suitable
anymore.

Precisely, it would work, but would forbid to implement an override
module.

Part of #7774

NO_TEST=no user visible changes
NO_CHANGELOG=see NO_TEST
NO_DOC=see NO_TEST

There are two flags in the LuaJIT useful for debugging and runtime
configuration purposes: `-j` and `-b`. However, if you want to
check the same Lua code from the Tarantool, you will need to make
some adjustments in the script itself, as those flags are not
present in the Tarantool's CLI.

This patch introduces those flags to the Tarantool, so debugging is
much more convenient now. Flags are working the same as they do in
the LuaJIT.

Closes #5541

NO_DOC=http://luajit.org/running.html#opt_b
NO_DOC=http://luajit.org/running.html#opt_j

Let's use xregion_alloc instead of region_alloc, because memory
allocations from fiber region shouldn't normally fail, see #3534.

Closes tarantool/security#97

NO_DOC=bug fix
NO_TEST=shouldn't normally happen
NO_CHANGELOG=see NO_TEST

This patch introduces new sql_seq_scan_default compat option. This
compat option allows to set default value for sql_seq_scan session
setting. Note that sql_seq_scan_default compat option only affects
sessions during initialization. This means that you should set
sql_seq_scan_default before running box.cfg{} or creating a new session.

Closes #8096

NO_DOC=Already exists

This patch removes unused ck_constraint.c module.

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring