This refactoring will:

1. Move macros from a header to the source file.
Macros should be used in header only with undef to avoid redefinitions.
Undef directive is not useful since we want to use these macros in the
source file.

2. Remove `using namespace lua_grammar` from header.
https://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines#Rs-using-directive

3. Moving serializer entry point and constant parameters into
luajit_fuzzer namespace.
It's a common practice in C++ to avoid name collisions.

4. Move serializer functions into anonymous namespace.
These functions are not a part of the interface so should have
static linkage.
https://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines#Rs-unnamed2

5. Fix ConvertToStringDefault function.
It was logically wrong so it would generate an identifier `123` from
`*123`.

NO_CHANGELOG=internal
NO_DOC=fuzzer fix

(cherry picked from commit 56488e15)
(cherry picked from commit 8ebe4851)

LuaJIT fuzzer used to stop due to timeout caused by infinite cycles and
recursions. Counters were introduced for every cycle and function to
address LuaJIT fuzzer timeouts.

The idea is to add unique counters for every cycle and function to
ensure finite code execution, if it wasn't already. For while, repeat,
for cycles, local and global named, anonymous functions, counters will
be initialized before the code generated from protobuf, and checked
in the first body statement. An entry point for the serializer was
created to count cycles and functions for counter initialization.

The idea was taken from a paper "Program Reconditioning: Avoiding
Undefined Behaviour When Finding and Reducing Compiler Bugs" [1].

Here is an example of a change in serialized code made by this commit.

Before:
```lua
while (true) do
    foo = 'bar'
end
function bar()
    bar()
end
```

After:
```lua
counter_0 = 0
counter_1 = 0
while (true) do
    if counter_0 > 5 then
        break
    end
    counter_0 = counter_0 + 1
    foo = 'bar'
end
function bar()
    if counter_1 > 5 then
        return
    end
    counter_1 = counter_1 + 1
    bar()
end
```
Protobuf structures that reproduce the timeout problem were added to
the LuaJIT fuzzer corpus.

[1] https://www.doc.ic.ac.uk/~afd/homepages/papers/pdfs/2023/PLDI.pdf

NO_CHANGELOG=internal
NO_DOC=fuzzer fix

(cherry picked from commit 4d004bbe)
(cherry picked from commit 9c59bbc8)

Cases in two switches had no breaks, so they were falling
through. Breaks were added to solve the problem. Code
generated by the LuaJIT fuzzer became more various.

NO_CHANGELOG=internal
NO_DOC=fuzzer fix

(cherry picked from commit 4430cac9)
(cherry picked from commit 61f7892e)

Function `datetime_strptime` decodes string with datetime according to
specified format, it accepts a datetime struct, buffer with datetime and
string with format in arguments. Fuzzing test used static string
"iso8601" as a format and it blocked fuzzing test to cover functions
used by datetime_strptime under the hood. Fuzz introspector shows that
code coveraged by a test is quite low.

Patch updates the test to make it more effective: buffer with datetime
and format string are generated using FDP (Fuzzing Data Provider).

Test file extension was changed to .cc, because FuzzingDataProvider is
used and we need building it by C++ compiler.

Function `tnt_strptime` uses assert, that triggered by fuzzing tests.
Therefore it was replaced with to if..then.

1. https://storage.googleapis.com/oss-fuzz-introspector/tarantool/

Fixes #8490

NO_CHANGELOG=fuzzing test
NO_DOC=fuzzing test
NO_TEST=fuzzing test

(cherry picked from commit a1bd6e0b)
(cherry picked from commit 504a0f88)

Fixes #8502
Needed for #8490

NO_DOC=bugfix
NO_TEST=covered by fuzzing test

(cherry picked from commit 783a7040)
(cherry picked from commit 70b0fc1f)

Patch adds a LuaJIT fuzzer based on libprotobuf-mutator and LibFuzzer.
Grammar is described via messages in protobuf format, serializer is
applied to convert .proto format to string.

For displaying generated code on the screen during fuzzing set
the environment variable 'LPM_DUMP_NATIVE_INPUT'.

For displaying error messages from lua functions set
the environment variable 'LUA_FUZZER_VERBOSE'.

Note: UndefinedBehaviourSanitizer is unsupported by LuaJIT (see #8473),
so fuzzing test is disabled when CMake option ENABLE_UB_SANITIZER is
passed.

Closes #4823

NO_DOC=<fuzzing testing of LuaJIT>
NO_TEST=<fuzzing testing of LuaJIT>

(cherry picked from commit a287c853)
(cherry picked from commit 87f4c930)

Added Google's 'libprotobuf-mutator' and 'protobuf' libraries
for developing grammar-based LuaJIT and SQL fuzzers based on
LibFuzzer.

It is needed to build protobuf module from source because
by default, the system-installed version of protobuf is used
by libprotobuf-mutator, and this version can be too old.

Part of #4823

NO_CHANGELOG=<dependencies>
NO_DOC=<dependencies>
NO_TEST=<dependencies>

(cherry picked from commit b11072a6)
(cherry picked from commit 70469594)

Added options for fuzzing and for getting more information
on debugging.

NO_CHANGELOG=<fuzzing options>
NO_DOC=<fuzzing options>
NO_TEST=<fuzzing options>

(cherry picked from commit 69f21e25)
(cherry picked from commit 85496d4a)

According to libFuzzer documentation [1] backslash should be escaped.

1. https://llvm.org/docs/LibFuzzer.html#dictionaries

```
$ swim_proto_meta_fuzzer -dict=swim_proto_meta_fuzzer.dict
ParseDictionaryFile: error in line 1
                "\001\000\000\004"
$ swim_proto_member_fuzzer -dict=swim_proto_member_fuzzer.dict
ParseDictionaryFile: error in line 1
                "\022\000\000\000\000\000\000\000"
```

NO_CHANGELOG=internal
NO_DOC=internal
NO_TEST=internal

(cherry picked from commit 62d03f15)
(cherry picked from commit 2b3eeda5)

Follows up #8488

NO_CHANGELOG=testing
NO_DOC=testing
NO_TEST=testing

(cherry picked from commit 64427eec)
(cherry picked from commit 787953b7)

Follows up #8488

NO_CHANGELOG=testing
NO_DOC=testing
NO_TEST=testing

(cherry picked from commit 1e584e89)
(cherry picked from commit d7f0a14b)

Seed corpus based on test data used in regression tests.
Dictionary was created using fuzzing test after 10^6 test executions.

Follows up #6731
Fixes #8488

NO_CHANGELOG=fuzzing corpus
NO_DOC=fuzzing corpus
NO_TEST=fuzzing corpus

(cherry picked from commit 46725004)
(cherry picked from commit af974bff)

Seed corpus based on test data used in regression tests.
Dictionary was created using fuzzing test after 10^6 test executions.

Follows up #6731
Part of #8488

NO_CHANGELOG=fuzzing corpus
NO_DOC=fuzzing corpus
NO_TEST=fuzzing corpus

(cherry picked from commit b0b11131)
(cherry picked from commit df31b8ca)

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

This patch adds a new method `log_set_format_by_name` which helps
FFI users set arbitrary logger's output format using its name
encoded as nul-terminated cstring.

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

godzie44 authored 1 year ago and Konstantin D committed 1 year ago

Add `current_cord_name` function to get a name of the current cord,
add `cord_is_main_dont_create` function.
Add exports for `cord_is_main`, `cord_is_main_dont_create` and
`current_cord_name` functions.

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

godzie44 authored 1 year ago and Konstantin D committed 1 year ago

Add `log_default_logger` to get a default logger.
Add exports for `log_set_format`, `log_set_level` and `log_default_logger` functions.

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

This patch makes schema_object_type to be exported.

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

NO_CHANGELOG=bug fix for unreleased feature
NO_DOC=bug fix for unreleased feature

1. Pass `docker build --build-arg TARANTOOL_VERSION=...` to ensure the
   resulting image contains the correct version.
2. Pass it in the CI trigger variables.
3. Fix CI jobs relationships: "docker" stage needs "deploy-packages".
4. Minor fixes in job names.

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

Nikita Zheleztsov authored 1 year ago and Дмитрий Кольцов committed 1 year ago

Commit/rollback triggers are run asynchronously, upon receiving the
write status from WAL. We can't run them in the original fiber that
submitted the WAL request, because it would open a time window between
writing a transaction to WAL and committing it in tx, which could lead
to violating the cascading rolback principles. As a result,
commit/rollback triggers run with admin privileges.

The issue was already solved for confirming async transaction, but
session and user are still not correct, when the transaction is
confirmed by the limbo. Let's fix this issue by temporarily setting
session and credentials to the original fiberfor running
commit/rollback triggers.

Closes #8742

NO_DOC=bugfix

(cherry picked from commit 8cd0cd09)

Nikita Zheleztsov authored 1 year ago and Дмитрий Кольцов committed 1 year ago

Currently some transactions on synchronous space fail to complete with
the `ER_CURSOR_NO_TRANSACTION` error, when on_rollback/on_commit triggers
are set.

This is caused due to the fact, that some rollback/commit triggers
require in_txn fiber variable to be set but it's not done when a
transaction is completed from the limbo. Callbacks, which are used to
work with iterators (`lbox_txn_pairs` and `lbox_txn_iterator_next`),
acquire tnx statements from the current transactions, but they cannot
do that, when this transaction is not assigned to the current fiber, so
`ER_CURSOR_NO_TRANSACTION` is thrown.

Let's assign in_txn variable when we complete transaction from the limbo.
Moreover, let's add assertions, which check whether in_txn() is correct,
in order to be sure, that `txn_complete_success/fail` always run with
in_txn set.

Closes #8505

NO_DOC=bugfix

(cherry picked from commit 6fadc8a0)

Dmitry Ivanov authored 1 year ago and Dmitry Rodionov committed 1 year ago

This patch helps us create and write to non-default
loggers provided by `say.h`. We'll use this mainly
for audit log in picodata.

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

Dmitry Ivanov authored 1 year ago and Dmitry Rodionov committed 1 year ago

Turns out Cyrus SASL doesn't use AM_MAINTAINER_MODE or anything to
prevent autotools from re-configuring the project at build time.
This patch "fixes" that by maiming autotools' paths in Makefile.

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

Kurdakov Alexander authored 1 year ago and Dmitry Rodionov committed 1 year ago

Add to console lua api function that overrides sql executor

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Feodor Alexandrov authored 1 year ago and Dmitry Rodionov committed 1 year ago

Picodata is built statically, and name crypto clashes between
`src/lib/crypto` and libcrypto from `openssl`. To resolve the
ambiguity this patch renames tarantool library to `tcrypto`.

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Dmitry Ivanov authored 1 year ago and Dmitry Rodionov committed 1 year ago

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Maksim Kaitmazian authored 1 year ago and Dmitry Rodionov committed 1 year ago

In the picodata-support branch, the tarantool.a library is built
instead of a tarantool executable. So tests cannot be launched because
of the lack of the executable. This commit adds this executable.

NO_DOC=just an executable is added
NO_TEST=just an executable is added
NO_CHANGELOG=just an executable is added

Georgy Moshkin authored 1 year ago and Dmitry Rodionov committed 1 year ago

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Georgy Moshkin authored 2 years ago and Dmitry Rodionov committed 1 year ago

Was broken because `tarantool_free` checks if the current process is the
main one and not the child, which was forked at some point (at what
point?). This check was implemented by saving the original process's id
in the static variable master_pid, which got set when the code got
loaded the first time into memory.

So we broke this when we started forking the process in picodata, which
resulted in `master_pid` being set to the pid of the picodata's
"supervisor" process, which doesn't even enter tarantool runtime.

The fix is simple - save master_pid first thing when running
tarantool_main.

Georgy Moshkin authored 2 years ago and Dmitry Rodionov committed 1 year ago

- tarantool is now a static library
- main(argc, argv) -> tarantool_main(argc, argv, cb, cb_data)
- callback is called before running lua script

Closes #9237

Add exports for fiber_set_name_n, fiber_name, fiber_id, fiber_csw &
fiber_find.

Also make fiber_set_joinable, fiber_set_ctx & fiber_get_ctx interpret
NULL as the current fiber.

@TarantoolBot document
Title: add basic fiber api to ffi exports.

5 basic functions can now be used via ffi api, which were previously
only accessible via lua api: fiber_set_name_n, fiber_name, fiber_id,
fiber_csw & fiber_find.

fiber_set_joinable now interprets NULL as current fiber.

NO_DOC=internal change
NO_TEST=internal change