Closes #7797

NO_DOC=security fix
NO_TEST=security fix

getenv() return values cannot be trusted, because an attacker might set
them. For instance, we shouldn't expect, that getenv() returns a value
of some sane size.

Another problem is that getenv() returns a pointer to one of
`char **environ` members, which might change upon next setenv().

Introduce a wrapper, getenv_safe(), which returns the value only when
it fits in a buffer of a specified size, and copies the value onto the
buffer. Use this wrapper everywhere in our code.

Below's a slightly decorated output of `grep -rwn getenv ./src --include
*.c --include *.h --include *.cc --include *.cpp --include *.hpp
--exclude *.lua.c` as of 2022-10-14.
`-` marks invalid occurences (comments, for example),
`*` marks the places that are already guarded before this patch,
`X` mars the places guarded in this patch, and
`^` marks places fixed in the next commit:

NO_WRAP
```
* ./src/lib/core/coio_file.c:509:	const char *tmpdir = getenv("TMPDIR");
X ./src/lib/core/errinj.c:75: const char *env_value = getenv(inj->name);
- ./src/proc_title.c:202: * that might try to hang onto a getenv() result.)
- ./src/proc_title.c:241:	* is mandatory to flush internal libc caches on getenv/setenv
X ./src/systemd.c:54: sd_unix_path = getenv("NOTIFY_SOCKET");
* ./src/box/module_cache.c:300: const char *tmpdir = getenv("TMPDIR");
X ./src/box/sql/os_unix.c:1441: azDirs[0] = getenv("SQL_TMPDIR");
X ./src/box/sql/os_unix.c:1446: azDirs[1] = getenv("TMPDIR");
* ./src/box/lua/console.c:394: const char *envvar = getenv("TT_CONSOLE_HIDE_SHOW_PROMPT");
^ ./src/box/lua/console.lua:771: local home_dir = os.getenv('HOME')
^ ./src/box/lua/load_cfg.lua:1007: local raw_value = os.getenv(env_var_name)
X ./src/lua/init.c:575: const char *path = getenv(envname);
X ./src/lua/init.c:592: const char *home = getenv("HOME");
* ./src/find_path.c:77: snprintf(buf, sizeof(buf) - 1, "%s", getenv("_"));
```
NO_WRAP

Part-of #7797

NO_DOC=security

This patch fixes the issue described in issue #5310 when the tuple
format has more fields than the space format. This solution is more
general than the solution in 89057a21.

Follow-up #5310
Closes #4666

NO_DOC=bugfix

Since our diagnostics use the `__FILE__` macro, they provide absolute
paths, which is kind of redundant and inconsistent: replace them with
relative ones.

As for debugging information, replacing absolute paths with relative ones
also requires an extra command to tell the debugger where to find the
source files, which is not convenient for developers: provide a new
`DEV_BUILD` option (turned off by default), which replaces absolute paths
with relative ones in debugging information if turned off.

Strip the prefix map flags from compiler flags exported to tarantool via
`src/trvia/config.h`.

Closes #7808

NO_DOC=<verbosity>
NO_TEST=<verbosity>

Fixes bug with determination of debug build (tarantool/test-run#352).

NO_CHANGELOG=<internal submodule version update>
NO_TEST=<submodule version update>
NO_DOC=<submodule version update>

Setting hardening compiler flags is used in three places: default build,
static build and enterprise build — refactor it into a separate module.

Follow-up e6abe1c9

NO_CHANGELOG=refactoring
NO_DOC=refactoring
NO_TEST=refactoring

e6abe1c9 passes compiler flags to dependencies via a `<project>_build` macro
parameter, which is, firstly, inconvenient, and, secondly, as a result, not
all dependencies got the required compiler flags passed: use global
variables instead and pass these flags to skipped dependencies.

Follow-up e6abe1c9

NO_CHANGELOG=refactoring
NO_DOC=refactoring
NO_TEST=refactoring

`add_compile_flags` macro relied on undefined behaviour: instead of
unsetting the `_lang` and `_flag` variables, it tried to unset their
values — fix this.

NO_CHANGELOG=build
NO_DOC=build
NO_TEST=build

Add missing .gitignore entry for autogenerated .lua.c translation units
located in <third_party/lua> directory.

Follows up #7593

NO_DOC=.gitignore
NO_TEST=.gitignore
NO_CHANGELOG=.gitignore

Signed-off-by: Igor Munkin <imun@tarantool.org>

If we raise different errors in case of entering an invalid password and
entering the login of a non-existent user during authorization, it will
open the door for an unauthorized person to enumerate users.
So let's unify raised errors in the cases described above.

Closes #tarantool/security#16

NO_DOC=security fix

Information about test code coverage is useful on code review.
This patch enables code coverage job in default ci.

NO_CHANGELOG=not a user-visible change
NO_DOC=not a user-visible change
NO_TEST=ci

This is a straightforward patch that makes the read view iterator struct
fixed-size so that it can be allocated on stack. This is a pre-requisite
for the raw C API for read views: the function that creates an iterator
should allocate no memory.

Closes #7813

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

At the moment we are not yet compatible with readline
support inside of Tarantool console. Diagnose that situation
at the moment debugger started and bail out.

NO_TEST=interactive
NO_DOC=Markdown updated

NO_TEST=see it elsewhere

Part of #7593

@TarantoolBot document
Title: Console debugger for Lua

Console debugger luadebug.lua
==============================

Module `luadebug.lua` is available as console debugger of Lua scripts.
It's activated via:

```
local debugger = require 'luadebug'
debugger()
```

Originally we have used 3rd-party code from slembcke/debugger.lua but
significantly refactored since then.

Currently available console shell commands are:
```
    c|cont|continue
    - continue execution
    d|down
    - move down the stack by one frame
    e|eval $expression
    - execute the statement
    f|finish|step_out
    - step forward until exiting the current function
    h|help|?
    - print this help message
    l|locals
    - print the function arguments, locals and upvalues
    n|next|step_over
    - step forward by one line (skipping over functions)
    p|print $expression
    - execute the expression and print the result
    q|quit
    - exit debugger
    s|st|step|step_into
    - step forward by one line (into functions)
    t|trace|bt
    - print the stack trace
    u|up
    - move up the stack by one frame
    w|where $linecount
    - print source code around the current line
```

Console debugger `luadebug.lua` allows to see sources of builtin
Tarantool module (e.g. `@builtin/datetime.lua`), and it uses new
function introduced for that purpose `tarantool.debug.getsources()`,
one could use this function in any external GUI debugger (i.e. vscode
or JetBrains) if need to show sources of builtin modules while they
have been debugged.

> Please see third_party/lua/README-luadebug.md for a fuller description
> of an original luadebug.lua implementation.

Created luatest test for interactive debugger luadebug.lua.
We use separate debug-target.lua for execution under
control of debugger session.

NO_DOC=test
NO_CHANGELOG=test

Number of 'call' hook invocations is always greater than
number of 'return' hook events. (i.e. ffi functions does use
`call` but there is no `return`. Same is for builtin functions
like `assert` or `error`).

We count number of stack frames available, and stop once
level is once again less or equal to the necessary one.
Math.huge is used to stop on each line (trace mode).

NO_CHANGELOG=internal
NO_DOC=internal
NO_TEST=internal

Do not use non-standard DBG_NOCOLOR, but use NO_COLOR [^1]
for disabling color output on terminals.

[^1]: https://no-color.org/

NO_DOC=later
NO_CHANGELOG=internal
NO_TEST=internal

Show debugger header only when repl is called.

NO_TEST=bugfix
NO_DOC=bugfix
NO_CHANGELOG=bugfix

Create command matcher and help map using convenient
builder function.

Commands described in a form:
```
    {'f.inish|step_out', 'step out', cmd_finish},
    {'p.rint $expression', 'execute the expression', cmd_print},
```
where we could introduce multiple aliases, commands with arguments,
commands descriptions and function handlers.

NO_DOC=internal
NO_CHANGELOG=internal
NO_TEST=internal

- refactor setmetatable usage to be more idiomatic;
- get rid of hardcode of `luadebug.lua` and use DEBUGGER
  instead. It will be necessary later for controlling
  of a name displayed by shell;
- refactor colored output facilities.

NO_TEST=refactoring
NO_DOC=refactoring
NO_CHANGELOG=refactoring

Get rid of all irrelevant for the Tarantool code:
- branches of code for non-LuaJIT interpreter;
- LuaJIT, but Windows-specific code;
- luadebug.lua (former debugger.lua) readline functions
  were incompatible with Tarantool console readline,
  so disable it entirely;
- Also fixed multiple luacheck warnings.

NO_DOC=refactoring
NO_CHANGELOG=refactoring
NO_TEST=refactoring

Extend Tarantool kernel internal API with the call
`tarantool.debug.getsources()` to allow to retrieve sources
of a Tarantool `builtin/*` modules to show them in the
debugger shell.

Created simple luatest script for checking consistency
of a values returned from `require 'tarantool'.debug.getsources()`
and an ctual script file content we expected to receive.

NO_DOC=see future commit
NO_CHANGELOG=see future commit

Enable automatical visualization of current context
after every next debugger step - so called "autolisting".

This supposedly resemble lldb behaviour, which is simplifying
debugging in CHUI.

NO_DOC=internal
NO_CHANGELOG=internal
NO_TEST=internal

Attempt to better format style of the imported
luadebug.lua (formerly known as debugger.lua)

NO_DOC=refactoring
NO_CHANGELOG=rafactoring
NO_TEST=refactoring

Use 3rd-party module https://github.com/slembcke/debugger.lua
as a basis for our debugger CLI shell, rename it to
luadebug.lua to avoid collision with vscode and JetBrains
debugger helper scripts.

NO_DOC=yet
NO_CHANGELOG=yet
NO_TEST=yet

The _vfunc system space is the sysview for the _func system space.
However, the _vfunc format is different from the _func format. This
patch makes the _vfunc format the same as the _func format.

Closes #7822

NO_DOC=bugfix

Allocation of URI parameters and their values dynamic arrays is done
inefficiently: they are reallocated each time a new parameter or parameter
value is added — grow them exponentially instead.

`struct uri_param` and `struct uri` are exposed in Lua via FFI
(see src/lua/uri.lua): add warnings about the necessity of reflecting
changes to them in `ffi.cdecl`.

Closes #7155

NO_DOC=optimization
NO_TEST=optimization

We used to ignore timezone difference (in `tzoffset`) for
datetime subtraction operation:

```
tarantool> datetime.new{tz='MSK'} - datetime.new{tz='UTC'}
---
- +0 seconds
...

tarantool> datetime.new{tz='MSK'}.timestamp -
           datetime.new{tz='UTC'}.timestamp
---
- -10800
...
```

Now we accumulate tzoffset difference in the minute component
of a resultant interval:

```
tarantool> datetime.new{tz='MSK'} - datetime.new{tz='UTC'}
---
- -180 minutes
...
```

Closes #7698

NO_DOC=bugfix

We did not take into consideration the fact that
as result of date/time arithmetic we could get
in a different timezone, if DST boundary has been
crossed during operation.

```
tarantool> datetime.new{year=2008, month=1, day=1,
			tz='Europe/Moscow'} +
	   datetime.interval.new{month=6}
---
- 2008-07-01T01:00:00 Europe/Moscow
...
```

Now we resolve tzoffset at the end of operation if
tzindex is not 0.

Fixes #7700

NO_DOC=bugfix

In PR #7791 the coverage stuff is slightly updated. The changes are
related to excluding some directories where the coverage will not be
calculated. The patch itself is straightforward enough, but I can see
the problems with uploading the results to coveralls.io:

NO_WRAP
    Using lcov file: ./coverage.info
    FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

    <--- Last few GCs --->

    [2683200:0x327c000]    27836 ms: Scavenge 2033.4 (2038.7) -> 2033.2 (2044.2) MB, 6.8 / 0.0 ms  (average mu = 0.258, current mu = 0.243) allocation failure
    [2683200:0x327c000]    27854 ms: Scavenge 2036.7 (2044.2) -> 2037.1 (2044.4) MB, 15.6 / 0.0 ms  (average mu = 0.258, current mu = 0.243) allocation failure
    [2683200:0x327c000]    27871 ms: Scavenge 2037.2 (2044.4) -> 2036.7 (2055.9) MB, 17.1 / 0.0 ms  (average mu = 0.258, current mu = 0.243) allocation failure

    <--- JS stacktrace --->

    ==== JS stack trace =========================================

        0: ExitFrame [pc: 0x140dc19]
    Security context: 0x2481918808d1 <JSObject>
        1: encode(aka encode) [0x23ec5e514a11] [/opt/actions-runner/_work/_actions/coverallsapp/github-action/v1.1.2/node_modules/qs/lib/utils.js:~118] [pc=0x193ecd3d5d82](this=0x35d34e4804b1 <undefined>,0x1e5895380119 <Very long string[60387428]>)
        2: stringify(aka stringify) [0x23ec5e5174c9] [/opt/actions-runner/_work/_actions/coverallsapp/github-action/v1.1.2/n...

     1: 0xa1a640 node::Abort() [/opt/actions-runner/externals/node12/bin/node]
     2: 0xa1aa4c node::OnFatalError(char const*, char const*) [/opt/actions-runner/externals/node12/bin/node]
     3: 0xb9a62e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/opt/actions-runner/externals/node12/bin/node]
     4: 0xb9a9a9 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/opt/actions-runner/externals/node12/bin/node]
     5: 0xd57c25  [/opt/actions-runner/externals/node12/bin/node]
     6: 0xd582b6 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [/opt/actions-runner/externals/node12/bin/node]
     7: 0xd64b75 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/opt/actions-runner/externals/node12/bin/node]
     8: 0xd65a25 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/opt/actions-runner/externals/node12/bin/node]
     9: 0xd670cf v8::internal::Heap::HandleGCRequest() [/opt/actions-runner/externals/node12/bin/node]
    10: 0xd15b35 v8::internal::StackGuard::HandleInterrupts() [/opt/actions-runner/externals/node12/bin/node]
    11: 0x1071176 v8::internal::Runtime_StackGuard(int, unsigned long*, v8::internal::Isolate*) [/opt/actions-runner/externals/node12/bin/node]
    12: 0x140dc19  [/opt/actions-runner/externals/node12/bin/node]
NO_WRAP

I have tried to use the latest version of the action (1.1.3), but I have
got the same issue.

It looks like it is a `coverallsapp/github-action` issue due to issue
in JavaScript due to inefficient amount of memory. The corresponding
bug [1] was filed against the repo.

So I suggest switching to the console utility `coveralls-lcov`. I did
some internal testing and it works fine.

[1] https://github.com/coverallsapp/github-action/issues/133

Fixes tarantool/tarantool-qa#278

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

Currently, in case of recovery from an old snapshot, Tarantool allows to
perform DDL operations on an instance with non-upgraded schema.
It leads to various unpredictable errors (because the DDL code assumes
that the schema is already upgraded). This patch forbids the following
operations unless the user has the most recent schema version:
- box.schema.space.create
- box.schema.space.drop
- box.schema.space.alter
- box.schema.index.create
- box.schema.index.drop
- box.schema.index.alter
- box.schema.sequence.create
- box.schema.sequence.drop
- box.schema.sequence.alter
- box.schema.func.create
- box.schema.func.drop

Closes #7149

NO_DOC=bugfix

By default a user might not have privileges to access the _schema space,
that will cause an error during schema_needs_upgrade(), which calls
get_version(). Fix this by using C variable dd_version_id, which is
updated in the _schema.version replace trigger.

There's a special case for upgrade() during bootstrap() - triggers are
disabled during bootstrap, that's why dd_version_id is not being updated.
Handle this by passing _initial_version=1.7.5 to the upgrade function.

Part of #7149

NO_DOC=internal
NO_CHANGELOG=internal

It's quite inconvenient to extend LuaJIT integration workflow now: one
needs to patch GitHub workflow file in Tarantool repository and then
setup integration testing in LuaJIT repository.

This patch introduces a new workflow parameter that can replace several
existing parameters (e.g. buildtype and GC64) and allow to easy extend
integration CI in LuaJIT (with rare little touches in Tarantool).

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

Reviewed-by: Sergey Kaplun <skaplun@tarantool.org>
Reviewed-by: Yaroslav Lobankov <y.lobankov@tarantool.org>
Signed-off-by: Igor Munkin <imun@tarantool.org>

This patch fixed the assertion when JOIN uses index of unsupported type.

Closes #5678

NO_DOC=bugfix

This commit adds support of transaction isolation levels introduced
earlier for memtx mvcc by commit ec750af6 ("txm: introduce
transaction isolation levels"). The isolation levels work exactly in
the same way as in memtx:

 - Unless a transaction explicitly specifies the 'read-committed'
   isolation level, it'll skip prepared statements, even if they are
   visible from its read view. The background for this was implemented
   in the previous patches, which added the is_prepared_ok flag to
   cache and mem iterators.

 - If a transaction skips a prepared statement, which would otherwise be
   visible from its read view, it's sent to the most recent read view
   preceding the prepared statement LSN. Note, older prepared statements
   are still visible from this read view and can actually be selected if
   committed later.

 - A transaction using the 'best-effort' isolation level (default) is
   switched to 'read-committed' when it executes the first write
   statement.

The implementation is tested by the existing memtx mvcc tests that were
made multi-engine in the scope of this commit. However, we add one more
test case - the one that checks that a 'best-effort' read view is
properly updated in case there is more than one prepared transaction.
Also, there are few tests that relied upon the old implementation and
assumed that select from Vinyl may return unconfirmed tuples. We update
those tests here as well.

Closes #5522

NO_DOC=already documented

To implement read-confirmed and best-effort isolation levels, we need
to skip unconfirmed (aka prepared) statements in the cache iterator. To
achieve that, we add a new flag is_prepared_ok. Unless the flag is set,
the iterator will skip prepared statements even if they are visible from
the iterator read view. Note, in contrast to the mem iterator, we don't
need to keep track of the min skipped statement LSN, because the cache
is just a view of the underlying levels so we'll find it out when we
descend to the mem level.

Needed for #5522

NO_DOC=internal
NO_CHANGELOG=internal

To implement read-confirmed and best-effort isolation levels, we need
to skip unconfirmed (aka prepared) statements in the mem iterator. To
achieve that, we add a new flag is_prepared_ok. Unless the flag is set,
the iterator will skip prepared statements even if they are visible from
the iterator read view. Upon skipping a statement, the iterator updates
min_skipped_plsn if the LSN of the skipped statement is less. We'll use
this LSN to update the transaction read view accordingly.

Needed for #5522

NO_DOC=internal
NO_CHANGELOG=internal

unit/vy_mem:
 - Remove the code creating unused lsregion.
 - Make test key_def and tuple_format global variables.
 - Replace assert() with fail().

unit/vy_cache:
 - Add missing test plan.

both:
 - Move history_node_pool to test/unit/vy_iterator_helpers.c.

Needed for #5522

NO_DOC=test
NO_TEST=test
NO_CHANGELOG=test

This commit replaces vy_tx_manager::last_prepared_tx with a list of all
prepared transactions sorted by PSN. We need this list to find the
newest prepared transaction that is older than a given PSN, which is
required to switch to the most recent read view in the 'best-effort'
isolation level when an unconfirmed (prepared) statement is skipped.

Needed for #5522

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

Unconfirmed statements have lsn >= MAX_LEN. Let's add a helper function
that would check this.

Needed for #5522

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring