Closes #5456

Reviewed-by: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Reviewed-by: Igor Munkin <imun@tarantool.org>

Co-authored-by: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Co-authored-by: Igor Munkin <imun@tarantool.org>

Closes #5455

Reviewed-by: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Reviewed-by: Igor Munkin <imun@tarantool.org>

Co-authored-by: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Co-authored-by: Igor Munkin <imun@tarantool.org>

In a commit "httpc: temporary disable redirecting test case"
(4cc6978a) a set of testcases
has been disabled while the bug #4180 is open.
Right now bug is closed so it is a time to enable testcases back.

Follows up: #4180

TL;DR httpd.py warns that port is busy when restart server. With socket
option SO_REUSEADDR it allows to restart httpd.py without problem.

Socket option SO_REUSEADDR tells the kernel that even if this port is
busy, go ahead and reuse it anyway. If it is busy, but with another
state, you will still get an address already in use error. It is useful
if your server has been shut down, and then restarted right away while
sockets are still active on its port.

Set correct encoding on opening file.

Follows up: #5538

Pass http body as byte string, define string literals correctly
and make items() iterable.

Follows up: #5538

Closes #5469

Reviewed-by: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Reviewed-by: Igor Munkin <imun@tarantool.org>

Co-authored-by: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Co-authored-by: Igor Munkin <imun@tarantool.org>

Closes #5468

Reviewed-by: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Reviewed-by: Igor Munkin <imun@tarantool.org>

Co-authored-by: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Co-authored-by: Igor Munkin <imun@tarantool.org>

Not all invalid characters were ignored by base64 decoder
causing data corruption and reads beyond decode table
(faults under ASAN).

Added corresponding check into base64 unit test.

Fixes: #5627

Since a space holds pointers to tuples it must increment
reference counters of its tuples and must decrement counters
of tuples that are deleted from the space.

Memtx TX manager also holds references to processing tuples
in the same way.

Before this patch there was a logic: while a tuple is dirty
it belongs to TX manager and does not belong to the space. Only
when a tuple is cleared and it is still in space it will be
referenced by space.

That logic leads to crashes in some DDL requests since they
works with indexes directly. For example deleting an index
causes dereferencing of all its tuples - even dirty.

This patch changes the logic. Now all tuples that are physically
in the primary index of the space a referenced. Once removed from
primary index - the tuple is dereferenced. TX manager references
tuples like before - every holding tuple is referenced.

Part of #5628

There was a typo in collection of read set - a dirty
tuple was added instead of clean.

Closes #5559

The problem happened when a tuple story was delete by two
statements, one committed and one not committed.

Part of #5628

Previously, in small allocator, memory pools
were allocated at the request, which in the case
of a small slab_alloc_factor led to use
pools with incorrect sizes. This patch changed
small allocator behavior, now we allocate pools
on the stage of allocator creation. Also we use
special function to find appropriate pool, which
is faster, then previous version with rbtree.
This change fixes #5216.

Also moved a check, that the slab_alloc_factor is in
the range (1.0, 2.0] from small allocator to memtx_engine.
If factor is not in range change it to 1.0001 or 2.0 respectively

Closes #5216

Coverity pointed that quorum value is int64_t
and could be trimmed in error message. Fix it
using a proper formatting.

CID 1500388
Part-of #5446

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

When tarantool launched with -e flag and in
script after there is an error, program hangs.
This happens because shed fiber launches separate
fiber for init user script and starts auxiliary
event loop. It's supposed that fiber will stop
this loop, but in case of error in script, fiber
tries to stop a loop when the last one isn't
started yet.

Added a flag, which will watch is loop started and
when fiber tries to call `ev_break()` we can be sure
that loop is running already.

Fixes #4983

To run Tarantool fuzzers on OSS Fuzz infrastructure it is needed to pass
library $LIB_FUZZING_ENGINE to linker and use external CFLAGS and
CXXFLAGS. Full description how to integrate with OSS Fuzz is in [1] and
[2].

Patch to OSS Fuzz repository [2] is ready to merge.

We need to pass options with "-fsanitize=fuzzer" two times
(in cmake/profile.cmake and test/fuzz/CMakeLists.txt) because:

- cmake/profile.cmake is for project source files,
  -fsanitize=fuzzer-no-link option allows to instrument project source
  files for fuzzing, but LibFuzzer will not replace main() in these
  files.

- test/fuzz/CMakeLists.txt uses -fsanitize=fuzzer and not
  -fsanitize=fuzzer-no-link because we want to add automatically
  generated main() for each fuzzer.

1. https://google.github.io/oss-fuzz/getting-started/new-project-guide/
2. https://google.github.io/oss-fuzz/advanced-topics/ideal-integration/
3. https://github.com/google/oss-fuzz/pull/4723

Closes #1809

Fuzzing tools uses evolutionary algorithms. Supplying seed corpus
consisting of good sample inputs is one of the best ways to improve
fuzz target’s coverage. Patch adds a corpuses that can be used with
existed fuzzers. The name of each file in the corpus is the sha1
checksum of its contents.

Corpus with http headers was added from [1] and [2].

1. https://google.github.io/oss-fuzz/getting-started/new-project-guide/
2. https://en.wikipedia.org/wiki/List_of_HTTP_header_fields
3. https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers

The libFuzzer allow to minimize corpus with help of `-merge` flag:
when 1 is passed, any corpus inputs from the 2nd, 3rd etc. corpus
directories that trigger new code coverage will be merged into the
first corpus directory, when 0 is passed an existed corpus will be
minimized.

All provided corpuses in a patch were minimized.

Part of #1809

There is a number of bugs related to parsing and encoding/decoding data.
Examples:

- csv: #2692, #4497, #2692
- uri: #585

One of the effective method to find such issues is a fuzzing testing.
Patch introduces a CMake flag to enable building fuzzers (ENABLE_FUZZER)
and add fuzzers based on LibFuzzer [1] to csv, http_parser and uri
modules. Note that fuzzers must return 0 exit code only, other exit
codes are not supported [2].

NOTE: LibFuzzer requires Clang compiler.

1. https://llvm.org/docs/LibFuzzer.html
2. http://llvm.org/docs/LibFuzzer.html#id22

How-To Use:

$ mkdir build && cd build
$ cmake -DENABLE_FUZZER=ON \
	-DENABLE_ASAN=ON \
	-DCMAKE_BUILD_TYPE=Debug \
	-DCMAKE_C_COMPILER="/usr/bin/clang" \
	-DCMAKE_CXX_COMPILER="/usr/bin/clang++" ..
$ make -j
$ ./test/fuzz/csv_fuzzer -workers=4 ../test/static/corpus/csv

Part of #1809

Closes #5454

Reviewed-by: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Reviewed-by: Igor Munkin <imun@tarantool.org>

Co-authored-by: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Co-authored-by: Igor Munkin <imun@tarantool.org>

Closes #5453

Reviewed-by: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Reviewed-by: Igor Munkin <imun@tarantool.org>

Co-authored-by: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Co-authored-by: Igor Munkin <imun@tarantool.org>

Follow-up #5435

The test involves writing synchronous transactions on one node and
making other nodes confirm these transactions after its death.
In order for the test to work properly we need to make sure the old
node replicates all its transactions to peers before killing it.
Otherwise once the node is resurrected it'll have newer data, not
present on other nodes, which leads to their vclocks being incompatible
and noone becoming the new leader and hanging the test.

Follow-up #5435

It is possible that a new leader (elected either via raft or manually or
via some user-written election algorithm) loses the data that the old
leader has successfully committed and confirmed.

Imagine such a situation: there are N nodes in a replicaset, the old
leader, denoted A, tries to apply some synchronous transaction. It is
written on the leader itself and N/2 other nodes, one of which is B.
The transaction has thus gathered quorum, N/2 + 1 acks.

Now A writes CONFIRM and commits the transaction, but dies before the
confirmation reaches any of its followers. B is elected the new leader and it
sees that the last A's transaction is present on N/2 nodes, so it doesn't have a
quorum (A was one of the N/2 + 1).

Current `clear_synchro_queue()` implementation makes B roll the transaction
back, leading to rollback after commit, which is unacceptable.

To fix the problem, make `clear_synchro_queue()` wait until all the rows from
the previous leader gather `replication_synchro_quorum` acks.

In case the quorum wasn't achieved during replication_synchro_timeout, rollback
nothing and wait for user's intervention.

Closes #5435

Co-developed-by: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>

Closes #5538

For Python 3, PEP 3106 changed the design of the dict builtin and the
mapping API in general to replace the separate list based and iterator
based APIs in Python 2 with a merged, memory efficient set and multiset
view based API. This new style of dict iteration was also added to the
Python 2.7 dict type as a new set of iteration methods. PEP-0469 [1]
recommends to replace d.iteritems() to iter(d.items()) to make code
compatible with Python 3.

1. https://www.python.org/dev/peps/pep-0469/

Part of #5538

The largest change in Python 3 is the handling of strings.
In Python 2, the str type was used for two different
kinds of values - text and bytes, whereas in Python 3,
these are separate and incompatible types.
Patch converts strings to byte strings where it is required
to make tests compatible with Python 3.

Part of #5538

In Python 2.x calling items() makes a copy of the keys that you can
iterate over while modifying the dict. This doesn't work in Python 3.x
because items() returns an iterator instead of a list and Python 3 raise
an exception "dictionary changed size during iteration". To workaround
it one can use list to force a copy of the keys to be made.

Part of #5538

- convert print statement to function. In a Python 3 'print' becomes a
function, see [1]. Patch makes 'print' in a regression tests compatible with
Python 3.
- according to PEP8, mixing using double quotes and quotes in a project looks
inconsistent. Patch makes using quotes with strings consistent.
- use "format()" instead of "%" everywhere

1. https://docs.python.org/3/whatsnew/3.0.html#print-is-a-function

Part of #5538

Report box.stat().*.total, box.stat.net().*.total and
box.stat.net().*.current via feedback daemon report.
Accompany this data with the time when report was generated so that it's
possible to calculate RPS from this data on the feedback server.

`box.stat().OP_NAME.total` reside in `feedback.stats.box.OP_NAME.total`, while
`box.stat.net().OP_NAME.total` reside in `feedback.stats.net.OP_NAME.total`
The time of report generation is located at `feedback.stats.time`

Closes #5589

We have a feedback server which gathers information about a running instance.
While general info is enough for now we may loose a precious information about
crashes (such as call backtrace which caused the issue, type of build and etc).

In the commit we add support of sending this kind of information to the feedback
server. Internally we gather the reason of failure, pack it into base64 form
and then run another Tarantool instance which sends it out.

A typical report might look like

 | {
 |   "crashdump": {
 |     "version": "1",
 |     "data": {
 |       "uname": {
 |         "sysname": "Linux",
 |         "release": "5.9.14-100.fc32.x86_64",
 |         "version": "#1 SMP Fri Dec 11 14:30:38 UTC 2020",
 |         "machine": "x86_64"
 |       },
 |       "build": {
 |         "version": "2.7.0-115-g360565efb",
 |         "cmake_type": "Linux-x86_64-Debug"
 |       },
 |       "signal": {
 |         "signo": 11,
 |         "si_code": 0,
 |         "si_addr": "0x3e800004838",
 |         "backtrace": "#0  0x630724 in crash_collect+bf\n...",
 |         "timestamp": "2020-12-23 14:42:10 MSK"
 |       }
 |     }
 |   }
 | }

There is no simple way to test this so I did it manually:
1) Run instance with

	box.cfg{log_level = 8, feedback_host="127.0.0.1:1500"}

2) Run listener shell as

	while true ; do nc -l -p 1500 -c 'echo -e "HTTP/1.1 200 OK\n\n $(date)"'; done

3) Send SIGSEGV

	kill -11 `pidof tarantool`

Once SIGSEGV is delivered the crashinfo data is generated and sent out. For
debug purpose this data is also printed to the terminal on debug log level.

Closes #5261

Co-developed-by: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

@TarantoolBot document
Title: Configuration update, allow to disable sending crash information

For better analysis of program crashes the information associated with
the crash such as

 - utsname (similar to `uname -a` output except the network name)
 - build information
 - reason for a crash
 - call backtrace

is sent to the feedback server. To disable it set `feedback_crashinfo`
to `false`.

Added corresponding test

Fixes: #5307

Part-of #5446

Co-developed-by: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

Prior to this patch, region on fiber was reset during select(), get(),
count(), max(), or min(). This would result in an error if one of these
operations was used in a user-defined function in SQL. After this patch,
these functions truncate region instead of resetting it.

Closes #5427

Accidentally, in built-in declaration list it was specified that
ifnull() can return only integer values, meanwhile it should return
SCALAR: ifnull() returns first non-null argument so type of return value
depends on type of arguments. Let's fix this and set return type of
ifnull() SCALAR.

After this patch, the persistent functions "box.schema.user.info" and
"LUA" will have the same rights as the user who executed them.

The problem was that setuid was unnecessarily set. Because of this,
these functions had the same rights as the user who created them.
However, they must have the same rights as the user who used them.

Fixes tarantool/security#1

Platform panic occurs when fiber.yield() is used within any active
(i.e. being executed now) hook.

It is a regression caused by 96dbc49d
('lua: prohibit fiber yield when GC hook is active').

This patch fixes false positive panic in cases when VM is not running
a GC hook.

Relates to #4518
Closes #5649

Reported-by: Michael Filonenko <filonenko.mikhail@gmail.com>

Found that test replication/skip_conflict_row.test.lua fails with
output message in results file:

  [035] @@ -139,7 +139,19 @@
  [035]  -- applier is not in follow state
  [035]  test_run:wait_upstream(1, {status = 'stopped', message_re = "Duplicate key exists in unique index 'primary' in space 'test'"})
  [035]  ---
  [035] -- true
  [035] +- false
  [035] +- id: 1
  [035] +  uuid: f2084d3c-93f2-4267-925f-015df034d0a5
  [035] +  lsn: 553
  [035] +  upstream:
  [035] +    status: follow
  [035] +    idle: 0.0024020448327065
  [035] +    peer: unix/:/builds/4BUsapPU/0/tarantool/tarantool/test/var/035_replication/master.socket-iproto
  [035] +    lag: 0.0046234130859375
  [035] +  downstream:
  [035] +    status: follow
  [035] +    idle: 0.086121961474419
  [035] +    vclock: {2: 3, 1: 553}
  [035]  ...
  [035]  --
  [035]  -- gh-3977: check that NOP is written instead of conflicting row.

Test could not be restarted with checksum because of changing values
like UUID on each fail. It happend because test-run uses internal
chain of functions wait_upstream() -> gen_box_info_replication_cond()
which returns instance information on its fails. To avoid of it this
output was redirected to log file instead of results file.

There was an option 'force_recovery' that makes tarantool
to ignore some problems during xlog recovery. This patch
change  this option behavior and makes tarantool to ignore
some errors during snapshot recovery just like during
xlog recovery.
Error types which can be ignored:
 - snapshot is someway truncated,
   but after necessary system spaces
 - snapshot has some garbage after it declared length
 - single tuple within snapshot has broken checksum
   and may be skipped without consequences (in this case
   we ignore all row with this tuple)

@TarantoolBot document
Title: Change 'force_recovery' option behavior
Change 'force_recovery' option behavior to allow
tarantool loading from broken snapshot

Closes #5422

Added test-run filter on box.snapshot error message:

  'Invalid VYLOG file: Slice [0-9]+ deleted but not registered'

to avoid of printing changing data in results file to be able to use
its checksums in fragile list of test-run to rerun it as flaky issue.

Found issues:

 1) vinyl/deferred_delete.test.lua
    https://gitlab.com/tarantool/tarantool/-/jobs/913623306#L4552

  [036] 2020-12-15 19:10:01.996 [16602] coio vy_log.c:2202 E> failed to process vylog record: delete_slice{slice_id=744, }
  [036] 2020-12-15 19:10:01.996 [16602] main/103/vinyl vy_log.c:2068 E> ER_INVALID_VYLOG_FILE: Invalid VYLOG file: Slice 744 deleted but not registered

 2) vinyl/gh-4864-stmt-alloc-fail-compact.test.lua
    https://gitlab.com/tarantool/tarantool/-/jobs/913810422#L4835

  [052] @@ -56,9 +56,11 @@
  [052]  --
  [052]  dump(true)
  [052]   | ---
  [052] - | ...
  [052] -dump()
  [052] - | ---
  [052] + | - error: 'Invalid VYLOG file: Slice 253 deleted but not registered'
  [052] + | ...

 3) vinyl/misc.test.lua
    https://gitlab.com/tarantool/tarantool/-/jobs/913727925#L5284

  [014] @@ -62,14 +62,14 @@
  [014]  ...
  [014]  box.snapshot()
  [014]  ---
  [014] -- ok
  [014] +- error: 'Invalid VYLOG file: Slice 1141 deleted but not registered'
  [014]  ...

 4) vinyl/quota.test.lua
    https://gitlab.com/tarantool/tarantool/-/jobs/914016074#L4595

  [025] 2020-12-15 22:56:50.192 [25576] coio vy_log.c:2202 E> failed to process vylog record: delete_slice{slice_id=522, }
  [025] 2020-12-15 22:56:50.193 [25576] main/103/vinyl vy_log.c:2068 E> ER_INVALID_VYLOG_FILE: Invalid VYLOG file: Slice 522 deleted but not registered

 5) vinyl/update_optimize.test.lua
    https://gitlab.com/tarantool/tarantool/-/jobs/913728098#L2512

  [051] 2020-12-15 20:18:43.365 [17147] coio vy_log.c:2202 E> failed to process vylog record: delete_slice{slice_id=350, }
  [051] 2020-12-15 20:18:43.365 [17147] main/103/vinyl vy_log.c:2068 E> ER_INVALID_VYLOG_FILE: Invalid VYLOG file: Slice 350 deleted but not registered

 6) vinyl/upsert.test.lua
    https://gitlab.com/tarantool/tarantool/-/jobs/913623510#L6132

  [008] @@ -441,7 +441,7 @@
  [008]  -- Mem has DELETE
  [008]  box.snapshot()
  [008]  ---
  [008] -- ok
  [008] +- error: 'Invalid VYLOG file: Slice 1411 deleted but not registered'
  [008]  ...

 7) vinyl/replica_quota.test.lua
    https://gitlab.com/tarantool/tarantool/-/jobs/914272656#L5739

  [023] @@ -41,7 +41,7 @@
  [023]  ...
  [023]  box.snapshot()
  [023]  ---
  [023] -- ok
  [023] +- error: 'Invalid VYLOG file: Slice 232 deleted but not registered'
  [023]  ...

 8) vinyl/ddl.test.lua
    https://gitlab.com/tarantool/tarantool/-/jobs/914309343#L4538

  [039] @@ -81,7 +81,7 @@
  [039]  ...
  [039]  box.snapshot()
  [039]  ---
  [039] -- ok
  [039] +- error: 'Invalid VYLOG file: Slice 206 deleted but not registered'
  [039]  ...

 9) vinyl/write_iterator.test.lua
    https://gitlab.com/tarantool/tarantool/-/jobs/920646297#L4694

  [059] @@ -80,7 +80,7 @@
  [059]  ...
  [059]  box.snapshot()
  [059]  ---
  [059] -- ok
  [059] +- error: 'Invalid VYLOG file: Slice 351 deleted but not registered'
  [059]  ...
  [059]  --
  [059]  -- Create a couple of tiny runs on disk, to increate the "number of runs"

 10) vinyl/gc.test.lua
     https://gitlab.com/tarantool/tarantool/-/jobs/920441445#L4691

  [050] @@ -59,6 +59,7 @@
  [050]  ...
  [050]  gc()
  [050]  ---
  [050] +- error: 'Invalid VYLOG file: Run 1176 deleted but not registered'
  [050]  ...
  [050]  files = ls_data()
  [050]  ---

 11) vinyl/gh-3395-read-prepared-uncommitted.test.lua
     https://gitlab.com/tarantool/tarantool/-/jobs/921944705#L4258

  [019] @@ -38,7 +38,7 @@
  [019]   | ...
  [019]  box.snapshot()
  [019]   | ---
  [019] - | - ok
  [019] + | - error: 'Invalid VYLOG file: Slice 634 deleted but not registered'
  [019]   | ...
  [019]
  [019]  c = fiber.channel(1)

If death timeout was decreased during waiting for leader death or
discovery to a new value making the current death waiting end
immediately, it could crash in libev.

Because it would mean the remaining time until leader death became
negative. The negative timeout was passed to libev without any
checks, and there is an assertion, that a timeout should always
be >= 0.

This commit makes raft code covered almost on 100%, not counting
one 'unreachable()' place.

Closes #5303