- Apr 15, 2021
-
-
Mergen Imeev authored
The original vdbe_decode_msgpack_into_mem() returns a MEM that contains string and binary values as ephemeral. This patch renames this function to mem_from_mp_ephemeral() and introduces new function mem_from_mp(), which returns a MEM that contains string and binary values in newly allocated memory. This patch changes behavior for this query: CREATE TABLE t1(m VARBINARY primary key); INSERT INTO t1 VALUES(x'6178'), (x'6278'), (x'6379'); SELECT count(*), substr(m,2,1) AS mx FROM t1 GROUP BY mx; Before this patch: tarantool> SELECT count(*), substr(m,2,1) AS mx FROM t1 GROUP BY mx; --- - metadata: - name: COLUMN_1 type: integer - name: MX type: string rows: - [2, 'y'] - [1, 'y'] ... After this patch. tarantool> SELECT count(*), substr(m,2,1) AS mx FROM t1 GROUP BY mx; --- - metadata: - name: COLUMN_1 type: integer - name: MX type: string rows: - [2, 'x'] - [1, 'y'] ... Part of #5818 Closes #5890
-
Mergen Imeev authored
This patch reworks mem_move(). This function moves all content of source MEM to destination MEM. Source mem is set to NULL. Part of #5818
-
Mergen Imeev authored
This patch intoduces mem_copy_as_ephemeral(). This function copies value from source MEM to destination MEM. In case value is of string or binary type and its allocation type is not static, it copied as value with ephemeral allocation type. Part of #5818
-
Mergen Imeev authored
This patch introduces mem_copy(). This function copies value from source MEM to destination MEM. In case value is string or binary and have not static allocation type, it is copied to newly allocated memory. Part of #5818
-
Mergen Imeev authored
This patch introduces mem_is_*() functions that allows to check current MEM state. Part of #5818
-
Mergen Imeev authored
This patch introduces mem_destroy(). This function should be used to free and destroy all objects owned by MEM, if necessary. Part of #5818
-
Mergen Imeev authored
This patch introduces mem_create(). This function should be used to initialize a MEM. MEM should be initialized before usage. Part of #5818
-
Mergen Imeev authored
This patch introduces mem_str() which allows to receive value of MEM as a string. Due to the limitations of static_alloc(), this function cannot be used to safely retrieve a value of MEM converted to string. This function is suitable for debugging, displaying the value in an error message, etc. Part of #5818
-
Mergen Imeev authored
This patch disables unused code in sql/vdbemem.c. It will simplify refactoring. Part of #5818
-
Mergen Imeev authored
Part of #5818
-
Mergen Imeev authored
This patch simplifies port_vdbemem_dump_lua() and port_vdbemem_get_msgpack() functions. Part of #5818
-
Mergen Imeev authored
This patch moves all MEM-related functions in mem.c/mem.h. Part of #5818
-
Mergen Imeev authored
There is no need for NULL-termination, since MEMs converted to msgpack and msgpack do not require NULL-terminated strings. Needed for #5818
-
Mergen Imeev authored
This patch disables unused code in sql/legacy.c. It will simplify refactoring. Needed for #5818
-
Mergen Imeev authored
This patch disables unused code in sql/analyze.c. It will simplify refactoring. Needed for #5818
-
Mergen Imeev authored
Currently, vdbe_decode_msgpack_into_mem() creates a MEM that is not properly initialized in case msgpack contains MP_EXT, MP_MAP, or MP_ARRAY fields. This patch makes it so that after execution of this function, all created MEMs are properly initialized. Closes #5011 Closes #5704 Closes #5913 Needed for #5818
-
Alexander Turenko authored
The reason of the update is to protect us against possible MITM attack from a malicious HTTPS proxy server with trusted certificate when TLS 1.3 is used (CVE-2021-22890, [1]). libcurl versions prior to 7.76.0 can skip a TLS handshake with a target host in this circumstances. Other vulnerabilities fixed in the (7.71.1; 7.76.0] version range do not look relevant to our built-in http client. See [2] for the full list. The CMake version requirement is updated from 3.1 to 3.2, because curl's CMakeLists.txt has the following clause at beginning: | cmake_minimum_required(VERSION 3.2...3.16 FATAL_ERROR) (It was there in vanilla curl 7.71.1 too and we had to remove it in order to support CMake 2. Now we don't support CMake 2, so it is good time to get rid of the extra patch upward vanilla curl repository.) According to the CMake versions table in 8a7702b1 ('github-ci: purge Debian Jessie from CI'), CMake 3.2+ is available on all supported OSes. [1]: https://curl.se/docs/CVE-2021-22890.html [2]: https://curl.se/docs/vulnerabilities.html @TarantoolBot document Title: Now we require CMake 3.2 to build tarantool In https://github.com/tarantool/doc/issues/1780 we requested to update the CMake minimum version to 3.1. Now it is time for 3.2. See details in the linked commit. Please, update the 'Building from source' manual.
-
- Apr 14, 2021
-
-
Roman Khabibov authored
Ship libcurl headers to system path "${PREFIX}/include/tarantool" in the case of libcurl included as bundled library or static build. It is needed to use SMTP client with tarantool's libcurl instead of system libcurl. See related issue: https://github.com/tarantool/smtp/issues/24 Closes #4559
-
Roman Khabibov authored
Enable smtp and smtps protocols in bundled libcurl. It is needed to use SMTP client with tarantool's libcurl instead of system libcurl. See related issue: https://github.com/tarantool/smtp/issues/24 Part of #4559
-
Alexander V. Tikhonov authored
Found that after patch set of 5 commits: aa97a185 ("feedback_daemon: count and report some events") e9c9832a ("feedback_daemon: generate report right before sending") bc15e0f0 ("feedback_daemon: send feedback on server start") 670acf0d ("feedback_daemon: rename `send_test` to `send`") c5d595bc ("feedback_daemon: include server uptime in the report") began to leak feedback demon on using cURL. To avoid of it decided to add LSAN suppresions: 1. for /lib/x86_64-linux-gnu/libcrypto.so leak:libcrypto.so* 2. for /lib/x86_64-linux-gnu/libssl.so leak:libssl.so* Needed for tarantool/tarantool-qa#116
-
Sergey Kaplun authored
LuaJIT submodule is bumped to introduce the following changes: * test: disable too deep recursive PUC-Rio test * test: disable PUC-Rio hanging GC test * test: disable PUC-Rio test checking -h option * test: disable PUC-Rio test for checking arg layout * test: disable PUC-Rio tests for several -l options * test: disable PUC-Rio test for syntax level error * test: disable PUC-Rio test for non-ascii variable * test: disable PUC-Rio test for fast function name * test: disable PUC-Rio test for variables in error * test: disable PUC-Rio test for getfenv in tailcall * test: remove string.gfind assert in PUC-Rio test * test: use math.fmod in PUC-Rio tests * test: disable locale-dependent PUC-Rio tests * test: adapt PUC-Rio test for %q in string.format * test: disable PUC-Rio test for per-coroutine hooks * test: adapt PUC-Rio test with activeline check * test: disable PUC-Rio test for tailcall info * test: adapt PUC-Rio test with count hooks * test: adapt PUC-Rio test for debug in vararg func * test: adapt PUC-Rio tests with vararg functions * test: disable PUC-Rio suite tests for line hook * test: adapt PUC-Rio tests counting GC steps * test: disable PUC-Rio tests for bytecode header * test: disable PUC-Rio tests confused by -v output * test: adapt PUC-Rio test for arg presence * test: remove quotes in progname from PUC-Rio * test: adapt PUC-Rio suite for out-of-source build * test: build auxiliary C libs from PUC-Rio Lua 5.1 * test: add PUC-Rio Lua 5.1 test suite Within this changeset PUC-Rio Lua 5.1 suite[1] is added to Tarantool testing. Considering Tarantool specific changes in runtime the suite itself is adjusted in LuaJIT submodule. <test/luajit-test-init.lua> pretest runner is adjusted by setting custom `_loadstring()` and `_dofile()` global function to run tests correctly for out-of-source build. Also, this patch excludes PUC-Rio-Lua-5.1 test suite from ASAN checks. [1]: https://www.lua.org/tests/lua5.1-tests.tar.gz Closes #5845 Closes #5686 Closes #5694 Closes #5701 Closes #5708 Closes #5710 Closes #5711 Closes #5712 Part of #4473 Reviewed-by:
Igor Munkin <imun@tarantool.org> Signed-off-by:
Igor Munkin <imun@tarantool.org>
-
Aleksandr Lyapunov authored
Add an argument which will receive successor of the inserted tuple. Part of #5628
-
Aleksandr Lyapunov authored
Common insertion function in BSP tree can make a replacement or insertion. When a replacement happens, a replaced element is returned via special agrument. Let's add another argument of the function for the case of insetion: an element before which then insertion happened will be returned via that argument. Part of #5628
-
Aleksandr Lyapunov authored
When MVCC engine is enabled, an index can contain temporary non-committed tuples that are not visible outside their transaction. That's why internal index size can be greater than visible by standalone observer. In order to fix space:count and index:count etc we have to use special adjustment from MVCC engine. Fixes #5972
-
Aleksandr Lyapunov authored
TX manager has a problem, kwown as phantom read. If a transaction reads a value by key and founds nothing, even no dirty tuples, then the read is not recorded anywhere. Implement a special storage for point reads that founds nothing, and use it in conflict resolution. Part of #5628
-
Aleksandr Lyapunov authored
Part of #5628
-
Aleksandr Lyapunov authored
See test for details. Part of #5628
-
Aleksandr Lyapunov authored
Before this patch there was a way to build history of a key when the terminal element in list is a clean tuple, not its story. That was a kind of optimization desigend to avoid excess story creation. Unfortunately that kind of otimization is possible to use only for one-index spaces, and even in that case it doesn't work. Let's remove optimization that doesn't work. Part of #5628
-
Aleksandr Lyapunov authored
Since DUP_REPLACE mode is always used with given old_tuple, there's no reason to check in runtime Part of #5628
-
Aleksandr Lyapunov authored
Part of #5628
-
Aleksandr Lyapunov authored
Part of #5628
-
Aleksandr Lyapunov authored
Since some operations with an ephemeral space are done without a transaction - we cannot make it consistent and should not even try. Part of #5628
-
Aleksandr Lyapunov authored
Part of #5958
-
Aleksandr Lyapunov authored
Part of #5958
-
Aleksandr Lyapunov authored
It seems that it'll never need it greater that UINT16_MAX. Maybe it will allow us to save some memory. Part of #5958
-
Aleksandr Lyapunov authored
Usually during final recovery, when .xlog files are loaded and applied, a special memtx_space_replace_primary_key function is used for faster recovery. The function only updates primary key while all other indexes are scheduled to build at the end of recovery. This approach doesn't work with MVCC engine and synchro spaces: When transactions appear in xlog and they may or may not be committed, we should process the in normal mode throught all indexes in order to leave transaction history in each index. Fixes #5610 Fixes #5973
-
Alexander V. Tikhonov authored
Found that 'actions/checkout' does not remove all temporary files from previous runs in submodules [1], it runs only 'git clean --xffd' [2]: libgcov profiling error:/home/ubuntu/actions-runner/_work/tarantool/tarantool/src/lib/small/CMakeFiles/small.dir/small/small_class.c.gcda:overwriting an existing profile data with a different timestamp To avoid of it added: git submodules foreach --recursive 'git clean -xffd' to 'actions/environment' which is run after each 'actions/checkout'. Part of #5986 [1]: https://github.com/tarantool/tarantool/runs/2318244478?check_suite_focus=true#step:5:4012 [2]: https://github.com/actions/checkout/issues/358
-
Sergey Kaplun authored
LuaJIT submodule is bumped to introduce the following changes: * tools: introduce --leak-only memprof parser option Within this changeset the new Lua module providing post-processing routines for parsed memory events is introduced: * memprof/process.lua: post-process the collected events The changes provide an option showing only heap difference. One can launch memory profile parser with the introduced option via the following command: $ tarantool -e 'require("memprof")(arg)' - --leak-only filename.bin Closes #5812 Reviewed-by:
Igor Munkin <imun@tarantool.org> Signed-off-by:
Igor Munkin <imun@tarantool.org>
-
Cyrill Gorcunov authored
Part-of #4642 Co-developed-by:
Vladislav Shpilevoy <v.shpilevoy@tarantool.org> Signed-off-by:
Cyrill Gorcunov <gorcunov@gmail.com>
-
Cyrill Gorcunov authored
Currently to run "C" function from some external module one have to register it first in "_func" system space. This is a problem if node is in read-only mode (replica). Still people would like to have a way to run such functions even in ro mode. For this sake we implement "box.lib" lua module. Unlike `box.schema.func` interface the `box.lib` does not defer module loading procedure until first call of a function. Instead a module is loaded immediately and if some error happens (say shared library is corrupted or not found) it pops up early. The need of use stored C procedures implies that application is running under serious loads most likely there is modular structure present on Lua level (ie same shared library is loaded in different sources) thus we cache the loaded library and reuse it on next load attempts. To verify that cached library is up to day the module_cache engine test for file attributes (device, inode, size, modification time) on every load attempt. Since both `box.schema.func` and `box.lib` are using caching to minimize module loading procedure the pass-through caching scheme is implemented: - box.lib relies on module_cache engine for caching; - box.schema.func does snoop into box.lib hash table when attempt to load a new module, if module is present in box.lib hash then it simply referenced from there and added into own hash table; in case if module is not present then it loaded from the scratch and put into both hashes; - the module_reload action in box.schema.func invalidates module_cache or fill it if entry is not present. Closes #4642 Co-developed-by:
Vladislav Shpilevoy <v.shpilevoy@tarantool.org> Signed-off-by:
Cyrill Gorcunov <gorcunov@gmail.com> @TarantoolBot document Title: box.lib module Overview ======== `box.lib` module provides a way to create, delete and execute `C` procedures from shared libraries. Unlike `box.schema.func` methods the functions created with `box.lib` help are not persistent and live purely in memory. Once a node get turned off they are vanished. An initial purpose for them is to execute them on nodes which are running in read-only mode. Module functions ================ `box.lib.load(path) -> obj | error` ----------------------------------- Loads a module from `path` and return an object instance associate with the module, otherwise an error is thrown. The `path` should not end up with shared library extension (such as `.so`), only a file name shall be there. Possible errors: - IllegalParams: module path is either not supplied or not a string. - SystemError: unable to open a module due to a system error. - ClientError: a module does not exist. - OutOfMemory: unable to allocate a module. Example: ``` Lua -- Without error handling m = box.lib.load('path/to/library) -- With error handling m, err = pcall(box.lib.load, 'path/to/library') if err ~= nil then print(err) end ``` `module:unload() -> true | error` --------------------------------- Unloads a module. Returns `true` on success, otherwise an error is thrown. Once the module is unloaded one can't load new functions from this module instance. Possible errors: - IllegalParams: a module is not supplied. - IllegalParams: a module is already unloaded. Example: ``` Lua m = box.lib.load('path/to/library') -- -- do something with module -- m:unload() ``` If there are functions from this module referenced somewhere in other places of Lua code they still can be executed because the module continue sitting in memory until the last reference to it is closed. If the module become a target to the Lua's garbage collector then unload is called implicitly. `module:load(name) -> obj | error` ---------------------------------- Loads a new function with name `name` from the previously loaded `module` and return a callable object instance associated with the function. On failure an error is thrown. Possible errors: - IllegalParams: function name is either not supplied or not a string. - IllegalParams: attempt to load a function but module has been unloaded already. - ClientError: no such function in the module. - OutOfMemory: unable to allocate a function. Example: ``` Lua -- Load a module if not been loaded yet. m = box.lib.load('path/to/library') -- Load a function with the `foo` name from the module `m`. func = m:load('foo') ``` In case if there is no need for further loading of other functions from the same module then the module might be unloaded immediately. ``` Lua m = box.lib.load('path/to/library') func = m:load('foo') m:unload() ``` `function:unload() -> true | error` ----------------------------------- Unloads a function. Returns `true` on success, otherwise an error is thrown. Possible errors: - IllegalParams: function name is either not supplied or not a string. - IllegalParams: the function already unloaded. Example: ``` Lua m = box.lib.load('path/to/library') func = m:load('foo') -- -- do something with function and cleanup then -- func:unload() m:unload() ``` If the function become a target to the Lua's garbage collector then unload is called implicitly. Executing a loaded function =========================== Once function is loaded it can be executed as an ordinary Lua call. Lets consider the following example. We have a `C` function which takes two numbers and returns their sum. ``` C int cfunc_sum(box_function_ctx_t *ctx, const char *args, const char *args_end) { uint32_t arg_count = mp_decode_array(&args); if (arg_count != 2) { return box_error_set(__FILE__, __LINE__, ER_PROC_C, "%s", "invalid argument count"); } uint64_t a = mp_decode_uint(&args); uint64_t b = mp_decode_uint(&args); char res[16]; char *end = mp_encode_uint(res, a + b); box_return_mp(ctx, res, end); return 0; } ``` The name of the function is `cfunc_sum` and the function is built into `cfunc.so` shared library. First we should load it as ``` Lua m = box.lib.load('cfunc') cfunc_sum = m:load('cfunc_sum') ``` Once successfully loaded we can execute it. Lets call the `cfunc_sum` with wrong number of arguments ``` Lua cfunc_sum() | --- | - error: invalid argument count ``` We will see the `"invalid argument count"` message in output. The error message has been set by the `box_error_set` in `C` code above. On success the sum of arguments will be printed out. ``` Lua cfunc_sum(1, 2) | --- | - 3 ``` The functions may return multiple results. For example a trivial echo function which prints arguments passed in. ``` Lua cfunc_echo(1,2,3) | --- | - 1 | - 2 | - 3 ``` Module and function caches ========================== Loading a module is relatively slow procedure because operating system needs to read the library, resolve its symbols and etc. Thus to speedup this procedure if the module is loaded for a first time we put it into an internal cache. If module is sitting in the cache already and new request to load comes in -- we simply reuse a previous copy. In case if module is updated on a storage device then on new load attempt we detect that file attributes (such as device number, inode, size, modification time) get changed and reload module from the scratch. Note that newly loaded module does not intersect with previously loaded modules, the continue operating with code previously read from cache. Thus if there is a need to update a module then all module instances should be unloaded (together with functions) and loaded again. Similar caching technique applied to functions -- only first function allocation cause symbol resolving, next ones are simply obtained from a function cache.
-