W231 (Local variable is set but never accessed)

Part of #5464

W213 (Unused loop variable)

Part of #5464

W212 (Unused argument)

Part of #5464

W211 (Unused local variable)

Part of #5464

W113 (Accessing an undefined global variable)

Part of #5464

Part of #5464
Closes #5741

This patch adds LuaJIT tests to every CI job type except the one for
static build testing routine on OSX: there is no way to run LuaJIT tests
for out of source build on OSX due to SIP[1].

[1]: https://developer.apple.com/library/archive/documentation/Security/Conceptual/System_Integrity_Protection_Guide/FileSystemProtections/FileSystemProtections.html



Follows up #4862

Reviewed-by: Sergey Kaplun <skaplun@tarantool.org>
Reviewed-by: Timur Safin <tsafin@tarantool.org>
Signed-off-by: Igor Munkin <imun@tarantool.org>

LuaJIT submodule is bumped to introduce the following changes:
* test: run luacheck static analysis via CMake
* test: fix warnings found with luacheck in misclib*
* test: run LuaJIT tests via CMake
* build: replace GNU Make with CMake
* build: preserve the original build system

Since LuaJIT build system is ported to CMake in scope of the changeset
mentioned above, the module building the LuaJIT bundled in Tarantool is
completely reworked. There is no option to build Tarantool against
another prebuilt LuaJIT due to a91962c0
('Until Bug#962848 is fixed, don't try to compile with external
LuaJIT'), so all redundant options defining the libluajit to be used in
Tarantool are dropped with the related auxiliary files.

To run LuaJIT related tests or static analysis for Lua files within
LuaJIT repository, <LuaJIT-test> and <LuaJIT-luacheck> targets are used
respectively as a dependency of the corresponding Tarantool targets.

As an additional dependency to run LuaJIT tests, prove[1] utility is
required, so the necessary binary packages are added to the lists with
build requirements.

[1]: https://metacpan.org/pod/TAP::Harness#prove



Closes #4862
Closes #5470
Closes #5631

Reviewed-by: Sergey Kaplun <skaplun@tarantool.org>
Reviewed-by: Timur Safin <tsafin@tarantool.org>
Signed-off-by: Igor Munkin <imun@tarantool.org>

If Lua source path given to <lua_source> function is relative, the
output file is generated in the binary directory. At the same time if
the given path to be compiled to *.lua.c is absolute, the output
file is generated in source directory instead of the binary one. This
patch fixes the latter case, providing the valid behaviour for out of
source build type.

Needed for #4862

Reviewed-by: Sergey Kaplun <skaplun@tarantool.org>
Reviewed-by: Timur Safin <tsafin@tarantool.org>
Signed-off-by: Igor Munkin <imun@tarantool.org>

row_count % WAL_ROWS_PER_YIELD == 0 is tested by both local recovery and relay
on each recovered row, so it makes sense to have this constant a power of two,
so that the compiler could replace '%' with bitwise and.

WAL_ROWS_PER_YIELD used to be 32000, so change it to the closest power
of two: 32768 == 2 ^ 15.

Follow-up #5762

While sending a WAL, relay only yields in `coio_write_xrow`, once it
sees the socket isn't ready for writes.
It may happen that the socket is always ready for a long period of time,
and relay doesn't yield at all while recovering a whole .xlog file. This
may take well more than a minute.
During this period of time, relay doesn't read replica's ACKs due to
relay reader fiber not being scheduled, and once the reader is finally
live it times out immediately, causing the replica to reconnect.

The problem mostly occurs when replica is syncing with master, so there
are always some WALs to read, and is amplified by the fact that replica
waits for replication_timeout to pass prior to reconnecting, which lets
master pile up even more ready WALs, and effectively making it impossible
for the replica to sync.

To fix the problem let's yield explicitly in relay_send_row every
WAL_ROWS_PER_YIELD rows. The same is already done in local recovery, and
serves the same purpose: to not block the event loop for too long.

Closes #5762

Tarantool has been integrated to OSS-Fuzz infrastructure, and
implemented fuzzers are passed now [1]. Patch adds a new workflow that
will execute fuzzers on each push to a master branch using OSS-Fuzz
infrastructure [2]. To reduce a load on testing machines job triggers on
changes in directory with Tarantool's source code, fuzzers source code,
corpus files and GH Actions workflow file for fuzzing.

OSS-Fuzz provides web interface with fuzzing statistics, found errors
and other useful information. Access to OSS-Fuzz web interface is
available for persons whose email addresses specified in project.yml,
committed to oss-fuzz repository [4]. Bugs found in OSS-Fuzz reported to
bugtracker [5].

Using fuzzing testing documented in Tarantool's wiki [6].

1. https://github.com/google/oss-fuzz/pull/4723
2. https://google.github.io/oss-fuzz/getting-started/continuous-integration/#integrating-into-your-repository
3. https://google.github.io/oss-fuzz/further-reading/clusterfuzz/
4. https://github.com/google/oss-fuzz
5. https://bugs.chromium.org/p/oss-fuzz/issues/list
6. https://github.com/tarantool/tarantool/wiki/Fuzzing

Follows up #1809

Instead of shifting flags inside txn_x_flag() helpers lets
define precompiled value. This allows us to operate with
several flags at onces and shrink code a bit.

Closes #5128

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

There was a bug in the netbox module related to access
to previously released memory. To understand the essence
of error, you need to understand how GC works in Lua:
- GC checks the reachability of objects in Lua in one cycle
  and cleans out those that were unreachable.
- Lua GC object is an entity whose memory is managed by the GC,
  for example: table, function, userdata, cdata.
  In our case it's cdata object, with struct error payload.
- ffi.gc allows us to clean up Lua GC object payload at the time
  of deleting the GC object.
- Finalizer in ffi.gc is hung on the Lua GC object.

So after ffi.cast in our case first err object becomes unreachable.
It will be cleaned after some time and if finalizer hangs on it,
payload will also be cleaned. So payload in new err object
(struct error in our case) becomes invalid.

Removed last OSX 14 job from testing, gitlab-ci configuration removed.

Found that in commits:

  c53c650e ("github-ci: port perf jobs from gitlab-ci")
  5c68884a ("github-ci: add jepsen* test jobs")

was mistakenly missed "or" check "if:" condition, fixed.

Follows up #5663
Follows up tarantool/tarantool-qa#79

http_parser() function resolves symbol to lower case using array, where
an char code maps to an appropriate symbol. Some symbols may have
negative ASCII code, but array index cannot be negative, it is an
undefined behaviour.

I read parts relevant to message headers in "Hypertext Transfer Protocol
(HTTP/1.1): Message Syntax and Routing" [1] and "Hypertext Transfer
Protocol -- HTTP/1.1" [2] and found that only ASCII symbols only allowed
in message headers. It means that symbols with negative ASCII code are
not allowed. Patch adds a check for each processed symbol to have
positive ASCII code, otherwise function returns HTTP_PARSE_INVALID.

1. RFC 7230 - 3.2 Header Fields, https://tools.ietf.org/html/rfc7230#section-3.2
2. RFC 2616 - Message Headers, https://www.ietf.org/rfc/rfc2616.txt

Bug found using LibFuzzer

Closes https://github.com/tarantool/security/issues/6
Needed for: https://github.com/google/oss-fuzz/pull/4723

Found more flaky tests running testings with real parallel even for
tests in fragile lists.

Added Github local Action 'perf' to be used for perf docker images
preparations and benchmarks runs. Ported jobs from gitlab-ci:

  perf_cbench
  perf_linkbench_ssd
  perf_nosqlbench_hash
  perf_nosqlbench_tree
  perf_sysbench
  perf_tpcc
  perf_tpch
  perf_ycsb_hash
  perf_ycsb_tree

Closes #5663

Pip and jq installation may fail if the host is real, in this way it
should be manually pre-installed.

Moved from gitlab-ci to github-ci 'jepsen*' jobs for
testing Tarantool with long Jepsen tests:

  jepsen-single-* jobs run by cron each 3 hours
  jepsen-cluster-* jobs run manualy or with API backend request.

Closes tarantool/tarantool-qa#79

Found issues and resolved:

- Added '$PROJECT_BINARY_DIR/build/' path creation before use.

- Set always ssh-agent restart before add new SSH key, because
  found that in some situations it may fail to add w/o it.

- Found mistake in TF_VAR_id variable setup, inverted ${CI_JOB_ID}
  variable check for it.

Also made improvements in script:

- Added all needed checks for all externally set mandatory variables.

- Added usage message with the list of mandatory and optional variables.

- Added comments to all script steps.

- Added step with checking nodes that they are reachable by SSH.

Found issue:

  root@hpalx:/source# make run-jepsen
  [  0%] Performing update step for 'jepsen-tests'

  *** Please tell me who you are.

  Run

    git config --global user.email "you@example.com"
    git config --global user.name "Your Name"

  to set your account's default identity.
  Omit --global to set the identity only in this repository.

  fatal: unable to auto-detect email address (got 'root@hpalx.(none)')
  Cannot save the current index state
  CMake Error at /source/jepsen-tests-prefix/tmp/jepsen-tests-gitupdate.cmake:83 (message):
    Failed to stash changes

  CMakeFiles/jepsen-tests.dir/build.make:95: recipe for target 'jepsen-tests-prefix/src/jepsen-tests-stamp/jepsen-tests-update' failed

added 'Nobody' User setup to git configuration if needed.

Ported from gitlab-ci FreeBSD 12 build as a standalone Github
Actions workflow. Used sh4/sh8 local hosts as self-runners with
label 'freebsd-sh'. These hosts based on CentOS 7 with FreeBSD
12.1 installed in VBox VM. It was tried to use runners from
Github Actions which based on OSX, but failed to build FreeBSD 12
there, created additional issue to investigate the issue [1].

Closes #5754

[1]: https://github.com/tarantool/tarantool-qa/issues/81

It is fixup for 8ac47898 ('memtx: fix
test for gh5304 issue and memtx_space_is_recovering function').

Without .md extension the entry will be missed by
`tools/gen-release-notes`.

Follows up #5304.

In previous version of patch we compared memtx state with
MEMTX_FINAL_RECOVERY to check that memtx recovery completed.
This is not quite true, memtx_state == MEMTX_FINAL_RECOVERY
means that the recovery from snapshot is finished, but recovery
from wals not. We need to compare memtx_state with MEMTX_OK
to check that recovery totally finished.
In previous test version on_replace trigger (created on
_user space) is never called. It's because is_recovery_finished()
always returns false: on_schema_init is invoked BEFORE
user's data recovery process (so trigger is not created at all
at this moment).
In new test version you can see correct user case:
we create on_replace trigger on _index system space,
which replaces/inserts/updates tuples in temp and loc spaces.
So each time user creates new space and index for it,
trigger replaces/inserts/updates tuples in temp and loc spaces.
Because trigger replaces/inserts/updates tuple with same
primary key, we get error when insert trigger called.

Follow-up #5304

previously such code:
s = box.schema.space.create('test')
s:create_index('pk', {parts = {1, 'int'}})
s:create_index('sk', {parts = {2, is_nullable=true}})

was resulting in is_nullable=false, which is wrong

Closes #5674

See https://github.com/tarantool/tarantool/discussions/5634

See https://github.com/tarantool/tarantool/discussions/5634

See https://github.com/tarantool/tarantool/discussions/5634

The processing of the message to be sent via Telegram is separated for
OSX and other platforms for two reasons:
 * there is no need to wrap newline symbol with a single quote marks
 * backslashes used in the message haven't to be escaped

The link to the failed job log is not saved in other GitHub Actions
links. At the same time this link can be obtained via the following
command using GitHub API:

 $ curl -s https://api.github.com/repos/tarantool/tarantool/actions/runs/${{  github.run_id }}/jobs | jq -r '.jobs[0].html_url'

The obtained link for the particular run has the following format:
https://github.com/tarantool/tarantool/runs/<id>, where 'id' is the
number of the job.

This link is added to the Telegram message for the failed job which ID
is set as suggested in GitHub docs[1].

[1]: https://docs.github.com/en/actions/reference/workflow-commands-for-github-actions#example-6

Moved from gitlab-ci to github-ci 'osx_10_15_lto' job for
building/testing Tarantool on OSX with LTO.

Moved from gitlab-ci to github-ci 'static_build*' jobs for
building/testing Tarantool on Linux and OSX. Added targets
in .travis.mk file for static build on OSX on Github hosts.

Closes #5661

Moved from gitlab-ci to github-ci 'out-of-source' job for
building/testing Tarantool in standalone build path while
source path is not writable.

Closes #5660

Moved from gitlab-ci to github-ci 'default_gcc_centos_7' job for
building/testing Tarantool on default GCC installed in CentOS 7.
Removed not used after movement templates from gitlab-ci config.

Artifacts paths for pack/deploy/osx based jobs corrected.

After commit:

  a598f3f5 ('test: update test-run (pass timeouts via env)')

The following variables added to github actions with testing:

  REPLICATION_SYNC_TIMEOUT
  TEST_TIMEOUT
  NO_OUTPUT_TIMEOUT

and cumulative variable to pass environment to pack/deploy jobs:

  PRESERVE_ENVVARS=REPLICATION_SYNC_TIMEOUT,TEST_TIMEOUT,NO_OUTPUT_TIMEOUT

These variables added within newly added local action:

  .github/actions/environment

Used Telegram public channel to send notifications for failed Github
Actions workflows. Created new Github Action based on Python:

  .github/actions/send-telegram-notify/action.yml

with the ability to send messages in MarkdownV2 format [1]. To make
able to run standalone Github Action used its 'composite' type [2].
Also added ability to logout Github environment variables in this
action.

[1] - https://core.telegram.org/bots/api#markdownv2-style
[2] - https://docs.github.com/en/actions/creating-actions/creating-a-composite-run-steps-action