Nikolay Shirokovskiy authored 1 year ago and Дмитрий Кольцов committed 1 year ago

Closes #3389
Closes #7689
Closes #4646

@TarantoolBot document
Title: new box.cfg parameter memtx_sort_threads

The parameter sets the number of threads used to sort keys of secondary
indexes on loading memtx database. The parameter cannot be changed
dynamically (as it does not make sense).

Maximum value is 256, minimum is 1. Default is to use all available cores.

Usage example:
```
box.cfg{memtx_sort_threads=4}
```

Nikolay Shirokovskiy authored 1 year ago and Дмитрий Кольцов committed 1 year ago

The algorithm runs sort in multiple threads and does not use OpenMP. It
has better threads utilization right from the beginning but probably
a worse constant than parallel qsort. See details in code comments.

Besides sort is not performed in calling thread but instead in spawned
worker threads. Calling thread yields waiting for worker threads to
finish. Exception is small data size, in this case sorting is executed
in calling thread saving time on spawning a thread. This should speed up
test execution. This is existing behaviour of qsort_arg but data size
threshold is reduced from 128000 to 1024.

Part of #3389

NO_CHANGELOG=internal
NO_DOC=internal

Maksim Kaitmazian authored 1 year ago and Дмитрий Кольцов committed 1 year ago

These exports allow us to perform user authentication and
network communication.

part of picodata/picodata/sbroad!292

NO_DOC=exports
NO_CHANGELOG=exports
NO_TEST=exports

Due to inconsistency of Tarantool type casting while using strict
data types as "double" or "unsigned" it is needed
to use "number" data type in a whole bunch of cases.
However "number" may contain "decimal" that will be serialized into
string by JSON builtin module.

This commit adds "encode_decimal_as_number" parameter to json.cfg{}.
That forces to encode `decimal` as JSON number to force type
consistency in JSON output.
Use with catious - most of JSON parsers assume that number is restricted
to float64.

NO_DOC=we do not host doc

* Mark CONV as non-weak, to prevent elimination of its side-effect.
* Fix ABC FOLD rule with constants.
* test: add test for conversions folding
* Add NaN check to IR_NEWREF.
* LJ_GC64: Fix lua_concat().
* test: introduce asserts assert_str{_not}_equal
* ci: enable codespell
* cmake: introduce target with codespell
* codehealth: fix typos
* tools: add cli flag to run profile dump parsers
* profilers: purge generation mechanism
* memprof: refactor symbol resolution
* sysprof: fix crash during FFUNC stream
* Fix last commit.
* Print errors from __gc finalizers instead of rethrowing them.
* x86/x64: Fix math.ceil(-0.9) result sign.
* test: fix flaky fix-jit-dump-ir-conv.test.lua
* IR_MIN/IR_MAX is non-commutative due to underlying FPU ops.
* Fix jit.dump() output for IR_CONV.
* Fix FOLD rule for x-0.
* FFI: Fix pragma push stack limit check and throw on overflow.
* Prevent compile of __concat with tailcall to fast function.
* Fix base register coalescing in side trace.
* Fix register mask for stack check in head of side trace.
* x64: Properly fix __call metamethod return dispatch.

Closes #8594
Closes #8767
Closes #9339
Part of #9145

NO_DOC=LuaJIT submodule bump
NO_TEST=LuaJIT submodule bump

* Revert "Update cur_L on exceptional path (arm)"
* Revert "arm64: fix cur_L restoration on error throw"
* Revert "Update cur_L on exceptional path"
* Revert "Fix cur_L tracking on exceptional path"
* Restore cur_L for specific Lua/C API use case.
* Fix Clang build.

Closes #6323
Part of #9145

NO_DOC=LuaJIT submodule bump
NO_TEST=LuaJIT submodule bump

luarocks version updated to version 3.9.2

Closes #6597

NO_DOC=The engine has been updated, the functionality has not changed
NO_TEST=see NO_DOC

(cherry picked from commit 1dc8cd81)

* test: fix fix-mips64-spare-side-exit-patching
* test: fix `fillmcode()` generator helper
* MIPS: Fix "bad FP FLOAD" assertion.
* Handle table unsinking in the presence of IRFL_TAB_NOMM.
* Fix handling of instable types in TNEW/TDUP load forwarding.
* Fix predict_next() in parser (again).
* Always exit after machine code page protection change fails.

Part of #8825

NO_DOC=LuaJIT submodule bump
NO_TEST=LuaJIT submodule bump

The patch updates curl module to the version 8.3.0 [1] and updates a
CMake module for building curl library.

Changes in CMake module:

- Option `CURL_STATICLIB` is gone and replaced with `BUILD_STATIC_LIBS`.
- Option `CURL_USE_GNUTLS` was added and disabled by default.
- NSS library support was removed and option `CURL_USE_NSS` has been
  removed as well.
- Option `CMAKE_UNITY_BUILD` was added and disabled by default.
- Option `CURL_DISABLE_FORM_API` was added and disabled by default. It is
  in fact depends on `CURL_DISABLE_MIME`, but anyway disabled explicitly.

Changelog: https://curl.se/changes.html#8_3_0

1. https://github.com/curl/curl/releases/tag/curl-8_3_0

NO_DOC=libcurl submodule bump
NO_TEST=libcurl submodule bump

Fixes #9086

(cherry picked from commit 979b4adb)

Historically, we encode strings that contain invalid or non-printable
utf-8 sequences in YAML as binary base64 blobs. We do that because of
limitations/bugs of the YAML encoder, which refuses to encode invalid
utf-8 strings. To work around this issue, we introduced the helper
utf8_check_printable, which is basically a copy of yaml_check_utf8,
and treat strings for which it fails as binary data (MP_BIN).

This commit updates the YAML submodule to the version where all known
issues with encoding invalid/unprintable utf-8 strings are fixed and
removes special treatment of such strings (drops utf8_check_printable).
Now unprintable or invalid utf-8 sequences are emitted as code points,
e.g. '\xFF' or '\uFFFF'. This change is a pre-requisite for introducing
the new varbinary type to Lua. Without it plain strings would be
implicitly converted to varbinary after decoding/encoding them in YAML,
which would be confusing.

Closes #8756

NO_DOC=bug fix

(cherry picked from commit 890a821c)

* Fix maxslots when recording BC_TSETM.
* Fix TDUP load forwarding after table rehash.
* Fix binary number literal parsing.
* Fix maxslots when recording BC_VARG, part 3.
* test: fix flaky <unit-jit-parse.test.lua> again
* Fix predict_next() in parser.
* Revert to trivial pow() optimizations to prevent inaccuracies.
* Fix pow() optimization inconsistencies.
* Improve assertions.
* Remove pow() splitting and cleanup backends.
* test: introduce `samevalues()` TAP checker
* MIPS: Add MIPS64 R6 port.
* DynASM/MIPS: Fix shadowed variable.
* MIPS64: Fix register allocation in assembly of HREF.
* Prevent integer overflow while parsing long strings.
* Fix LJ_MAX_JSLOTS assertion in rec_check_slots().
* Fix debug.getinfo() argument check.
* ARM: Fix GCC 7 -Wimplicit-fallthrough warnings.
* DynASM: Fix warning.
* Fix GCC 7 -Wimplicit-fallthrough warnings.
* Cleanup math function compilation and fix inconsistencies.
* FFI: Eliminate hardcoded string hashes.
* Windows: Add UWP support, part 1.
* build: fix non-Linux/macOS builds
* PPC: Add soft-float support to JIT compiler backend.
* PPC: Add soft-float support to interpreter.
* MIPS64: Add soft-float support to JIT compiler backend.
* MIPS: Fix handling of spare long-range jump slots.
* test: introduce mcode generator for tests
* MIPS: Use precise search for exit jump patching.
* sysprof: improve parser's memory footprint
* tools: add execution permission to sysprof parser
* sysprof: remove `split by vmstate` option

Part of #8825

NO_DOC=LuaJIT submodule bump
NO_TEST=LuaJIT submodule bump

This reverts commit ace88542.

That commit disabled repl in tarantool for debugger console session
because debugger console wasn't compatible with Tarantool console,
i.e. this code hang in terminal:

```lua
tarantool> dbg = require 'luadebug'
tarantool> dbg()
```

With the previous patch in the patchset, full readline support and
console compatibility was introduced. Thus, no need to disable repl.

Part of #7738

NO_TEST=internal
NO_DOC=internal, revert unpublished
NO_CHANGELOG=internal

(cherry picked from commit 3774e85d)

Implemented readline history and autocomplete by reusing
readline facilities of Tarantool console. They used to be
being hidden once Lua 'console' module is loaded. With c432e9e9
(lua: don't use public module name as internal one), now they
are available as 'console.lib'.

Closes #7738

NO_TEST=covered by refactored console_debugger_session_test.lua

@TarantoolBot document
Title: proper readline support in readline

Similar to tarantool interactive console, tdbg now uses readline
for its shell. It enables handier input editing, command history
and so on.

(cherry picked from commit 92222451)

* ci: support coveralls
* cmake: add code coverage support
* test: run flake8 static analysis via CMake
* test: fix E741 errors by pycodestyle
* test: fix E722 errors by pycodestyle
* test: fix E711 errors by pycodestyle
* test: fix E502 errors by pycodestyle
* test: fix E501 errors by pycodestyle
* test: fix E305 errors by pycodestyle
* test: fix E303 errors by pycodestyle
* test: fix E302 errors by pycodestyle
* test: fix E301 errors by pycodestyle
* test: fix E275 errors by pycodestyle
* test: fix E251 errors by pycodestyle
* test: fix E231 errors by pycodestyle
* test: fix E203 errors by pycodestyle
* test: fix E201 and E202 errors by pycodestyle
* test: suppress E131 errors by pycodestyle
* test: fix E128 errors by pycodestyle
* test: fix E122 errors by pycodestyle
* gdb: fix Python <assert> statement usage

NO_DOC=LuaJIT submodule bump
NO_TEST=LuaJIT submodule bump
NO_CHANGELOG=LuaJIT submodule bump

* ci: introduce testing workflow with sanitizers
* build: introduce LUAJIT_USE_ASAN option
* test: introduce test:done TAP helper
* memprof: remove invalid assertions
* ci: clean up workflow for exotic builds

Closes #5878

NO_DOC=LuaJIT submodule bump
NO_TEST=LuaJIT submodule bump
NO_CHANGELOG=LuaJIT submodule bump

* FFI: Fix recording of union initialization.
* Fix maxslots when recording BC_VARG, part 2.
* Fix maxslots when recording BC_VARG.
* Fix BC_UCLO insertion for returns.
* ci: update job concurrency group definition

Part of #8825

NO_DOC=LuaJIT submodule bump
NO_TEST=LuaJIT submodule bump

* test: fix flaky <unit-jit-parse.test.lua>
* Fix use-def analysis for vararg functions.
* Fix use-def analysis for BC_VARG.
* Fix TNEW load forwarding with instable types.
* Fix memory probing allocator to check for valid end address, too.
* Another fix for lua_yield() from C hook.
* Fix lua_yield() from C hook.
* Fix saved bytecode encapsulated in ELF objects.
* x64: Fix 64 bit shift code generation.
* Fix canonicalization of +-0.0 keys for IR_NEWREF.
* test: add utility for parsing `jit.dump`
* test: split utils.lua into several modules
* test: rewrite lj-49-bad-lightuserdata test in C
* test: rewrite misclib-sysprof-capi test in C
* test: rewrite misclib-getmetrics-capi test in C
* test: introduce utils.h helper for C tests
* test: introduce module for C tests
* test: fix setting of {DY}LD_LIBRARY_PATH variables
* build: fix build with LUAJIT_USE_GDBJIT enabled
* ci: update the branch name for Tarantool 2.11

Closes #8718
Part of #7900
Part of #8516

NO_DOC=LuaJIT submodule bump
NO_TEST=LuaJIT submodule bump

Bump the metrics submodule to 1.0.0-2-gea83227 version.

NO_DOC=metrics submodule bump
NO_TEST=metrics submodule bump
NO_CHANGELOG=metrics submodule bump

(cherry picked from commit 2780e8e0)

Bump the metrics submodule to 1.0.0 version.

NO_DOC=submodule bump
NO_TEST=submodule bump
NO_CHANGELOG=submodule bump

(cherry picked from commit 8bbc73ce)

* LJ_GC64: Make ASMREF_L references 64 bit.
* lldb: introduce luajit-lldb
* x64/LJ_GC64: Fix emit_rma().
* Limit path length passed to C library loader.

Part of #4808
Part of #8069
Part of #8516

NO_DOC=LuaJIT submodule bump
NO_TEST=LuaJIT submodule bump

Rework "running tests with built-in package" assert since now checks is
a callable table package with subpackages instead of a single function.

NO_DOC=No tagged version since checks initial embedding
NO_CHANGELOG=No tagged version since checks initial embedding

(cherry picked from commit a8bc6312)

tarantool/metrics [1] is a lua module (distributed as a separate rock)
for metrics aggregation and export. After this patch, it will be a part
of the tarantool binary.

1. https://github.com/tarantool/metrics

Part of #7725

@TarantoolBot document
Title: embedded metrics

Now tarantool has metrics module on its board. metrics is a lua module
previously distributed as a separate rock which is widely used by
tarantool applications. Metrics has its own documentation section:
https://www.tarantool.io/en/doc/latest/book/monitoring/ , but it doesn't
yet mention anything about the embedding.

(cherry picked from commit 82ebbb35)

* ARM64: Avoid side-effects of constant rematerialization.
* ARM64: Fix {AHUV}LOAD specialized to nil/false/true.
* ARM64: Fix pcall() error case.
* Fix math.min()/math.max() inconsistencies.
* test: add test case for math.modf

Closes #6163
Part of #8069
Follows up #7230

NO_DOC=LuaJIT submodule bump
NO_TEST=LuaJIT submodule bump

Updated third_party/zstd submodule from v1.5.2 to pre-1.5.5 version.
The new version fixes a rare bug that was introduced in 1.5.0 [1].

A v1.5.5 release version will be produced in March.

1. https://github.com/facebook/zstd/pull/3517

Fixes #8391

NO_DOC=build
NO_TEST=build

(cherry picked from commit ecbbc925)

* ci: add nojit flavor for exotic builds
* test: fix lua-Harness JIT-related tests
* test: adjust JIT-related tests in tarantool-tests
* build: fix build with JIT disabled
* Minor fixes.

NO_DOC=LuaJIT submodule bump
NO_TEST=LuaJIT submodule bump
NO_CHANGELOG=test

* test: make skipcond helper more convenient
* test: introduce test:skiprest TAP helper
* test: introduce test:skipall TAP helper
* test: stop using utils.selfrun in tests
* ci: use LuaJIT-test target in testing workflows

NO_DOC=LuaJIT submodule bump
NO_TEST=LuaJIT submodule bump
NO_CHANGELOG=test

Updating libcurl from 7.84.0 to 7.87.0 in commit 09f4eca1
("third_party: update libcurl from 7.84.0 to 7.87.0")
made it impossible to built tarantool with bundled curl with nghttp2,
because nghttp2 was not updated to fit libcurl 7.87.0, this led to
missing function
nghttp2_option_set_no_rfc9113_leading_and_trailing_ws_validation
during linking step.

https://github.com/nghttp2/nghttp2/releases/tag/v1.52.0

Closes #8268

NO_DOC=there is no known behavior changes, consider the change as not
       visible for a user
NO_TEST=can only reproduce in specific environment
NO_CHANGELOG=see NO_DOC

(cherry picked from commit a6fae421)

Added `-d` option for activation of debugger shell:
- it calls debugger shell in `luadebug.lua` instead of
  a standard interactive shell from `console.lua`;
- that option complements original way for starting a
  debugging shell via `require 'luadebug'()`, but is a
  little bit easier.

NB! At the moment when we enter debugging mode instead of
a standard Tarantool console, we change banner to:
```
Tarantool debugger 2.11.0-entrypoint-852-g9e6ed28ae
type 'help' for interactive help
```

Part of #7456

@TarantoolBot document
Title: Command-line option `-d` for console debugger.

Please see third_party/lua/README-luadebug.md for a full description
of different ways to activate debugging shell.

* ci: add LUAJIT_ENABLE_CHECKHOOK for exotic matrix
* ci: add ARM64 architecture to exotic testing
* ci: update action/checkout to v3
* Fix os.date() for wider libc strftime() compatibility.
* x86/x64: Fix loop realignment.
* ci: introduce workflow for exotic builds
* sysprof: fix interval parsing in dual-number mode
* test: add test for `string.format('%c', 0)`
* ci: drop obsolete arguments for LuaJIT integration

Part of #8069

NO_DOC=LuaJIT submodule bump
NO_TEST=LuaJIT submodule bump

We used to "normalize" breakpoint filename paths
down to their basic name, which worked fine for
unique file names, but is getting annoying when
you need to debug a code in one of many `init.lua`
modules, i.e.

    break init.lua:5

Will stop at _every_ `init.lua` file we will run
at the debugging session.

So we add machinery for partial paths lookup, to
make possible to set breakpoints using (unique-enough)
path suffixes, e.g. if we have multiple `init.lua` in
the application, i.e.
    ./init.lua
    ./A/init.lua
    ./B/init.lua

Then we could use syntax:

    break A/init.lua:5

To activate debugger breakpoint only in the particular
module, and not trigger it elsewhere.

Current suffix trees limitations and peculiarities
--------------------------------------------------

Please keep in mind that suffix-tree algorithm, used for
partial paths lookup, uses `$ref` and `$f` node names
for their own purposes. That means that it would not handle
well breakpoints with paths containing `$f` and `$ref`.
That's not a big problem given that majority of filesystems
we are running on do not usually allow `$` as part of
file name.

Also, due to suffix tree lookup behavior, if we would
activate multiple breakpoints:
    break init.lua:10
    break A/init.lua:10

Then _least specific_ `init.lua:10` breakpoint will trigger.
From users' prospective it makes no much difference, as we
need to stop in debugger shell in this line, but this is
the way how suffix tree is working.

Dot files treatment
-------------------

There is a special mechanism for `.` and `..` treatment
if they are at the header of a path provided. I.e.

    break ./main.lua:10
    break ../a/b/c/least.lua:15

For these cases `.` and `..` will be converted to the
full path to the given file location.

NO_TEST=hard to implement tests. Postponed till step.5

@TarantoolBot document
Title: Breakpoints in console debugger for Lua

Match partial path patterns in breakpoints
==========================================

Please see `third_party/lua/README-luadebug.md` for
a fuller description of partial path syntax one could use
in a breakpoints definition.

We used to "normalize" breakpoint filename paths
down to their basic name, which worked fine for
unique file names, but is getting annoying when
you need to debug a code in one of many `init.lua`
modules, i.e.

    break init.lua:5

Will stop at _every_ `init.lua` file we will run
at the debugging session.

So we add machinery for partial paths lookup, to
make possible to set breakpoints using (unique-enough)
path suffixes, e.g. if we have multiple `init.lua` in
the application, i.e.
    ./init.lua
    ./A/init.lua
    ./B/init.lua

Then we could use syntax:

    break A/init.lua:5

To activate debugger breakpoint only in the particular
module, and not trigger it elsewhere.

NB! If we will activate multiple breakpoints:
    break init.lua:10
    break A/init.lua:10

Then _least specific_ `init.lua:10` breakpoint will trigger.
From users' prospective it makes no much difference, as we
need to stop in debugger shell in this line, but this is
the way how suffix tree is working.

NO_TEST=hard to implement tests. Postponed till step.5

@TarantoolBot document
Title: Breakpoints in console debugger for Lua

Match partial path patterns in breakpoints
==========================================

Please see `third_party/lua/README-luadebug.md` for
a fuller description of partial path syntax one could use
in a breakpoints definition.

* Added breakpoints support in debugger;

NO_TEST=documentation added

@TarantoolBot document
Title: Breakpoints in console debugger for Lua

Breakpoints support
===================

Please see `third_party/lua/README-luadebug.md` for
a description of breakpoints support introduced to the
builtin console debugger in `luadebug.lua`.

Commonize mechanism activated by `dbg.cfg.auto_where` and an
automatic code listing we show for breakpoint context.

Make sure we do not show redundant listing, i.e. after
commands `where #`, or `up`, or `down` we do not output
automatic listing, as there was already shown some different
context.

NO_DOC=see later
NO_CHANGELOG=internal
NO_TEST=internal

Make message, which is shown for a `help` command,
to be more readable.

NO_DOC=see later commit
NO_CHANGELOG=internal
NO_TEST=internal

Introduced `dbg_write_error` to centralize the way
how errors reported in debugger shell.

Similarly `dbg_write_warn` is used for reporting of
highlighted non-fatal warnings.

NO_DOC=internal
NO_CHANGELOG=internal

We may use "caret" or "arrow" symbols either in
colored or uncolored contexts. To reduce further
confusion we get rid of colored prefixes.

NO_DOC=internal
NO_CHANGELOG=internal
NO_TEST=internal

In many cases we do not need to use all "Snl"
parameters in `debug.getinfo()`: so try to trim wherever
possible "n", which is calculating `name` and `namewhat`,
and "S" which is generating `source`, `short_src` and others.

NO_DOC=internal
NO_CHANGELOG=internal
NO_TEST=internal

Visualize properly locations with breakpoints
saved there. In addition to current line visualization
(using green '=>') we annotate corresponding line with
red "●".

NO_TEST=see the later commit
NO_DOC=internals
NO_CHANGELOG=internals

Introduced few new commands to handle breakpoints:
- `b file:NNN` to set new breakpoint;
- `bd file:NNN` to remove breakpoint;
- 'bl` to list all active breakpoints.

There is partial breakpoint notation supported when
for `b :N` or `b +N` breakpoint will be set to the
asked line in the _currently_ debugged file.

Changed `c` (continue) to stop on breakpoints, if there any active
one asssigned by user. Otherwise it still run in full speed, without
any hook check.

NO_DOC=see the later commit
NO_CHANGELOG=see the later commit

Historically luadebug.lua used `up` and `down` in
a strange order which was inherited from original
debugger.lua implementation. But that was counter
intuitive, and is confusing.

Swap their meaning to be more compatible with
that we accustomed in `gdb`/`lldb`.

NO_DOC=internal