fiber._internal.schedule_task() is an API for a singleton fiber
worker object. It serves for not urgent delayed execution of
functions. Main purpose - schedule execution of a function, which
is going to yield, from a context, where a yield is not allowed.
Such as an FFI object's GC callback.

It will be used by SWIM and by fio, whose destruction yields, but
they need to use GC finalizer, where a yield is not allowed.

Part of #4727

box.error.set(err) sets err to instance's diagnostics area. Argument err
is supposed to be instance of error object. This method is required
since we are going to avoid adding created via box.error.new() errors to
Tarantool's diagnostic area.

Needed for #1148
Part of #4778

We are going to introduce more tests related to error module, so let's
move all error-related tests from box/misc.test.lua to a separate test
file (box/error.test.lua).

Needed for #1148

Testing via plain C interface with a shell is not stable,
the shell might simply be misconfigured or not found and
we will simply stuck forever (the signal handling in libev
is tricky and requires at least idle cycles or similar to
pass event processing). Thus lets rather run a program we
know is presenting in the system (popen-child executable).

Fixes #4811

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

vy_build_insert_tuple() processes insertion into secondary indexes being
created. It contains yield points during which in-memory level of LSM
tree may change (for example rotate owing to triggered dump). So after
yield point it is required to fetch from LSM struct pointer to mem again
to operate on valid metadata. This patch updates pointer to mem after
mentioned yield point.

Closes #4810

box_check_config() didn't check memtx_memory and vinyl_memory
upper bound. As a result, it was possible to set memory size
higher than what the quota allows as maximum.

That worked only when box.cfg() was called first time, because
quota_init() does not check its value. Subsequent box.cfg() calls
use quota_set(), which aborts the program if a size is too big.
Only in debug mode. In release quota_set() also worked with any
sizes.

Closes #4705

Reviewed-by: Igor Munkin <imun@tarantool.org>
Reviewed-by: Nikita Pettik <korablev@tarantool.org>

We need to set a thread cancellation guard, because
another thread may cancel the current thread at a
really bad time (messages flush, mutex lock)

Fixes: #4127

Users keep complaining about too short fiber name. New limit is
255, should be enough for any sane name.

Closes #4394

Reviewed-by: Cyrill Gorcunov <gorcunov@gmail.com>
Reviewed-by: Nikita Pettik <korablev@tarantool.org>

@TarantoolBot document
Title: fiber.name length limit.

It was 32, now it is 255. Besides, it seems like `fiber.name`
`{truncate = true}` option is not documented.

By default, if a new name is too long, `fiber.name(new_name)`
fails with an exception. To make it always succeed there is an
option 'truncate': `fiber.name(new_name, {truncate = true})`. It
truncates the name to the max length if it is too long.

Box.cfg{listen = 0} automatically chooses a port. But it was
impossible to obtain a real port the instance is bound to.

An ability to see a real port may help to make test-run more
robust, because it won't depend on which ports are free, and
won't need to pre-choose them in advance.

Now box.info.listen shows a real address, or nil when listen is
turned off. Also a real address is logged instead of the dummy
0-port one.

Closes #4620

@TarantoolBot document
Title: box.info.listen - real address

New value in box.info - listen. It is a real address to which the
instance was bound. For example, if box.cfg.listen was set with
a zero port, box.info.listen will show a real port. The address
is stored as a string:

    - unix/:<path> for UNIX domain sockets;
    - <ip>:<port> for IPv4;
    - [ip]:<port> for IPv6.

If the instance does not listen anything, box.info.listen is nil.

This patch introduces "current" function for sequences.
It returns the last retrieved value of specified sequence or
throws an error if no value has been generated yet.

This patch partially reverts 3ff1f1e3
(box: remove sequence_get) here similar function "get" was removed
to avoid possible misleading with "currval" function of PosgreSQL
that returns the last obtained value of the sequence in the scope
of current session. In contrast "current" returns the last globally
retrieved value of the sequence.

Closes #4752

Reviewed-by: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Reviewed-by: Nikita Pettik <korablev@tarantool.org>

@TarantoolBot document
Title: sequence:current()

This patch introduces "current" function for sequences.
It returns the last retrieved value of specified sequence or
throws an error if no value has been generated yet ("next"
has not been called yet or right after "reset" is called).

Lua:

Example:

```lua
sq = box.schema.sequence.create('test')
---
...
sq:current()
---
- error: Sequence 'test' is not started
...
sq:next()
---
- 1
...
sq:current()
---
- 1
...
sq:set(42)
---
...
sq:current()
---
- 42
...
sq:reset()
---
...
sq:current()  -- error
---
- error: Sequence 'test' is not started
...
```

C API:

```C
int
box_sequence_current(uint32_t seq_id, int64_t *result);
```

Where:
  * seq_id - sequence identifier;
  * result - pointer to a variable where the current sequence
  value will be stored on success.

Returns 0 on success and -1 otherwise. In case of an error user
could get it via `box_error_last()`.

Absence of the body in the unprepare response forces users to perform
additional checks to avoid errors. Adding an empty body fixes this problem.

Closes #4769

Reviewed-by: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Reviewed-by: Nikita Pettik <korablev@tarantool.org>

It is needed for performing iproto tests in Lua.

Needed for #4769

Reviewed-by: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Reviewed-by: Nikita Pettik <korablev@tarantool.org>

Since this build flag has been removed as a result of reverting the
tarantool/luajit@d4e985a, its definition in the corresponding Tarantool
cmake file is irrelevant.

Furthermore, considering the breakage faced in #4770 the following tests
are introduced:
* the check whether space __pairs metamethod is set to space.pairs to
  create a Lua Fun iterator that handles __pairs manually underneath.
* the check whether pairs builtin behaviour doesn't change when __pairs
  is set e.g. on space object.

Follow-up #4560
Closes #4770

Reviewed-by: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Signed-off-by: Igor Munkin <imun@tarantool.org>

In #4684 it was found that box.tuple.* contained some private
functions: bless(), encode(), and is().

Bless() and encode() didn't make any sense for a user, so they
were hidden into box.internal.tuple.*.

But box.tuple.is() is actually a useful thing. It is harnessed in
the tests a lot, and is likely to be already used by customers,
because it is available in box.tuple.* for a long time. It is a
matter of time when someone will open a doc ticket saying that
box.tuple.is() is not documented. The patch makes it legally
public.

Follow-up #4684

@TarantoolBot document
Title: box.tuple.is()
```Lua
box.tuple.is(object)
```
A function to check whether a given object is a tuple cdata
object. Returns true or false. Never raises nor returns an error.

Basic test for popen engine

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

This code is called from C, so it shouldn't throw.

Closes #4753

Reviewed-by: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Reviewed-by: Nikita Pettik <korablev@tarantool.org>

Use 'filename' grep_log() option instead of custom log search.

Reviewed-by: Alexander Turenko <alexander.turenko@tarantool.org>

When tarantool tries to recover rtree from a snapshot and memtx_memory value
is lower than it has been when the snapshot was created, server suffers from
segmentation fault. This happens because there is no out of memory error
handling in rtree lib. In another words, we do not check the result of
malloc operation.
The execution flow in case of recovery uses different way and the secondary
keys are build in batches. That way has no checks and reservations.
The patch adds memtx_rtree_index_reserve implementation to make sure that any
memory allocation in rtree will fail. Although this gives us no additional
optimization as in case of memtx_tree, the memory reservation prevents
tarantool from segmentation fault. If there is not enough memory to be reserved
server will fail gently with the "Failed to allocate" error message.

Closes #4619

Despite what was stated in the documentation, netbox.connect was not always
equivalent to netbox.self. In particular, they converted tuple to different
types - table and cdata respectively.

The patch fixes the issue and covers all cases where netbox.self and connect
perform conversion of types - e.g., for box.error.

Closes #4513

In 89c73e64 ("fio: respect
$TMPDIR in fio.tempdir(), when it is set") was added a test
checking that fio.tempdir() returns a path to a folder, stored
by a path specified in $TMPDIR environment variable.

Check was done by calling Lua returned_path:find(tmpdir_path).
If tmpdir path contained 'special' characters such as '.', it
didn't match, because string.find() takes a regular expression,
not just a string.

string.startswith() works fine.

Follow-up #4794

The patch fixes a bug for the commit b0f588f6 where statistics on
box.execute was collected twice. This happened because
sql_prepare_and_execute called sql_execute under the hood, so there's
no need to do rmean_collect in both of them.

Follow-up #4756

TMPDIR is an environment variable used to tell what a directory
should be used to create temporary files. It is described in the
POSIX standard, and should be used by programs creating temporary
files.

Closes #4794

@TarantoolBot document
Title: fio.tempdir() $TMPDIR

fio.tempdir() stores created temporary directory into /tmp by
default. This can be changed by setting TMPDIR environment
variable. Before starting Tarantool, or at runtime by
os.setenv().

It was possible to leak user password through setting 'replication'
configuration option in first box.cfg invocation. This happened due
to unconditional logging in load_cfg function. The patch introduces
conditional logging.

Closes #4493

Extend <ALTER TABLE> statement to drop table constraints by their
names.

Closes #4120

@TarantoolBot document
Title: Drop table constraints in SQL
Now, it is possible to drop table constraints (PRIMARY KEY,
UNIQUE, FOREIGN KEY, CHECK) using
<ALTER TABLE table_name DROP CONSTRAINT constraint_name> statement
by their names.

For example:

tarantool> box.execute([[CREATE TABLE test (
                             a INTEGER PRIMARY KEY,
                             b INTEGER,
                             CONSTRAINT cnstr CHECK (a >= 0)
                        );]])
---
- row_count: 1
...

tarantool> box.execute('ALTER TABLE test DROP CONSTRAINT cnstr;')
---
- row_count: 1
...

The same for all the other constraints.

Clarify the error message for better user handling. Add the name
of space where the constraint under dropping wasn't founded.

Part of #4120

When checking wheter rejoin is needed, replica loops through all the
instances in box.cfg.replication, which makes it believe that there is a
master holding files, needed by it, since it accounts itself just like
all other instances.
So make replica skip itself when finding an instance which holds files
needed by it, and determining whether rebootstrap is needed.

We already have a working test for the issue, it missed the issue due to
replica.lua replication settings. Fix replica.lua to optionally include
itself in box.cfg.replication, so that the corresponding test works
correctly.

Closes #4759

We have a mechanism for restoring rows originating from an instance that
suffered a sudden power loss: remote masters resend the isntance's rows
received before a certain point in time, defined by remote master vclock
at the moment of subscribe.
However, this is useful only on initial replication configuraiton, when
an instance has just recovered, so that it can receive what it has
relayed but haven't synced to disk.
In other cases, when an instance is operating normally and master-master
replication is configured, the mechanism described above may lead to
instance re-applying instance's own rows, coming from a master it has just
subscribed to.
To fix the problem do not relay rows coming from a remote instance, if
the instance has already recovered.

Closes #4739

Found another problem with the test:

 | /builds/DtQXhC5e/0/tarantool/tarantool/test/unit/popen.c:63:6:
 | error: variable 'rc' is used uninitialized whenever 'if' condition is true [-Werror,-Wsometimes-uninitialized]
 |        if (handle == NULL)
 |            ^~~~~~~~~~~~~~

Decided to revert it and fix in background.

This reverts commit 40a51647.

Found another problem with the test:

 | /builds/DtQXhC5e/0/tarantool/tarantool/test/unit/popen.c:63:6:
 | error: variable 'rc' is used uninitialized whenever 'if' condition is true [-Werror,-Wsometimes-uninitialized]
 |        if (handle == NULL)
 |            ^~~~~~~~~~~~~~

Decided to revent the test and so revent its disabling.

This reverts commit bceaf05c.

This test is buggy, need to rewrite. Thus to not block other developers
which refer the master branch just disable it.

The test was added in 40a51647 ('test:
unit/popen').

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
Reviewed-by: Alexander Turenko <alexander.turenko@tarantool.org>

Basic tests for popen engine

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

The commit e8009f41 ('box: user.grant
error should be versatile') did not do proper clean-up: it grants
non-default privileges for user 'guest' and does not revoke them at the
end. That caused occasional failures of other tests, all with the same
error saying user 'guest' already had access on universe.

This case should be handled by test-run in a future, see [1].

[1]: https://github.com/tarantool/test-run/issues/156

Follows up #714

After #4736 regression fix (in fact it just reverts the new logic in
small) it is possible again that a fiber's region may hold a memory for
a while, but release it eventually. When the used memory exceeds 128 KiB
threshold, fiber_gc() puts 'garbage' slabs back to slab_cache and
subtracts them from region_used() metric. But until this point those
slabs are accounted in region_used() and so in fiber.info() metrics.

This commit fixes flakiness of test cases of the following kind:

 | fiber.info()[fiber.self().id()].memory.used -- should be zero
 | <...workload...>
 | fiber.info()[fiber.self().id()].memory.used -- should be zero

The problem is that the first `<...>.memory.used` value may be non-zero.
It depends of previous tests that were executed on this tarantool
instance.

The obvious way to solve it would be print differences between
`<...>.memory.used` values before and after a workload instead of
absolute values. This however does not work, because a first slab in a
region can be almost used at the point where a test case starts and a
next slab will be acquired from a slab_cache. This means that the
previous slab will become a 'garbage' and will not be collected until
128 KiB threshold will exceed: the latter `<...>.memory.used` check will
return a bigger value than the former one. However, if the threshold
will be reached during the workload, the latter check may show lesser
value than the former one. In short, the test case would be unstable
after this change.

It is resolved by restarting of a tarantool instance before such test
cases to ensure that there are no 'garbage' slabs in a current fiber's
region.

Note: This works only if a test case reserves only one slab at the
moment: otherwise some memory may be hold after the case (and so a
memory check after a workload will fail). However it seems that our
cases are small enough to don't trigger this situation.

Call of region_free() would be enough, but we have no Lua API for it.

Fixes #4750.

Calling prepare and execute did not update corresponding request statistics
in the box.stat table. This happened because methods that collect stats were
never called where they should have been.

Closes #4756

Error message on granted privileges was not flexible and
did not distinguish between universal or any other privileges
leaving either 'nil' or simply '' at the end.

Closes #714

The bug was in an attempt to update a record in _space_sequence
in-place, to add field path and number. This was not properly
supported by the system space's trigger, and was banned in the
previous patch of this series.

But delete + tuple update + insert work fine. The patch uses them.

To test it the old disabled and heavily outdated
xlog/upgrade.test.lua was replaced with a smaller analogue, which
is supposed to be created separately for each upgrade bug.
According to the new policy of creating test files.

The patch tries to make it easy to add new upgrade tests and
snapshots. A new test should consist of fill.lua script to
populate spaces, snapshot, needed xlogs, and a .test.lua file.
Fill script and binaries should be in the same folder as test file
name, which is located in version folder. Like this:

 xlog/
 |
 + <test_name>.test.lua
 |
 +- upgrade/
    |
    +- <version>/
    |   |
    |   +-<test_name>/
    |     |
    |     +- fill.lua
    |     +- *.snap
    |     +- *.xlog

Version is supposed to say explicitly what a version files in
there have.

Closes #4771

Anyway this does not work for generated sequences. A proper
support of update would complicate the code and won't give
anything useful.

Part of #4771

In case if we unable to revert guard page back to
read|write we should never use such slab again.

Initially I thought of just put panic here and
exit but it is too destructive. I think better
print an error and continue. If node admin ignore
this message then one moment at future there won't
be slab left for use and creating new fibers get
prohibited.

In future (hopefully near one) we plan to drop
guard pages to prevent VMA fracturing and use
stack marks instead.

Reviewed-by: Alexander Turenko <alexander.turenko@tarantool.org>
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

Both madvise and mprotect calls can fail due to various
reasons, mostly because of lack of free memory in the
system.

We log such cases via say_x helpers but this is not enough.
In particular tarantool/memcached relies on diag error to be
set to detect an error condition:

 | expire_fiber = fiber_new(name, memcached_expire_loop);
 | const box_error_t *err = box_error_last();
 | if (err) {
 |	say_error("Can't start the expire fiber");
 |	say_error("%s", box_error_message(err));
 |	return -1;
 | }

Thus lets use diag_set() helper here and instead of macros
use inline functions for better readability.

Fixes #4722

Reported-by: Alexander Turenko <alexander.turenko@tarantool.org>
Reviewed-by: Alexander Turenko <alexander.turenko@tarantool.org>
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>