- Mar 26, 2024
-
-
Andrey Saranchin authored
Current func_adapter implementation has several drawbacks. Firstly, the interface has 21 virtual method (except for virtual destructor) which means you need to implement all these methods if you need a new func_adapter. Secondly, if you need to pass arguments from Lua to func_adapter, you need to iterate over Lua stack and push the value to func_adapter with appropriate type. We have internal interface for calling persistent functions - port. Let's use it for triggers too. It will allow to easily implement new func_adapter since it will have only call method and virtual destructor. Also, it will allow to easily call any func_adapter from Lua using port_lua - now module trigger relies on the fact that there are only Lua triggers. This change reduces code complexity as well - since we are using port_c for triggers, that allows to be dumped several times, we can create helper `event_run_triggers` which incapsulates work with `event_trigger_iterator` and simply calls all the triggers from the event passing the same port with arguments. Since func_adapter is used mostly for triggers, let's make both ports arguments optional - many triggers have no arguments and almost all triggers ignore returned values. Along the way, fix a possible crash when iterator passed to transactional trigger is saved and used after the trigger is over. NO_CHANGELOG=internal NO_DOC=internal
-
Andrey Saranchin authored
The method allows to represent port contents as a list of port_c_entry. It is handy to easily iterate over the port contents since port_c_entry is actually a list of variants of supported by Tarantool types. It is needed to easily process values returned from triggers. Returned entries mustn't be destroyed since they don't own any resources. Their lifetime is similar to the MsgPack obtained by port_get_msgpack - it may be returned directly from the port, or it can be allocated on fiber()->gc, so the caller is responsible for cleaning the region up. NO_CHANGELOG=internal NO_DOC=internal
-
Andrey Saranchin authored
We are going to use port_c for user-defined triggers, so we need to add an ability to store iterator in port_c. Also, this port already allows to be dumped several times - we should keep this invariant to call chains of triggers using only one port. Therefore, this commit populates port_c with new type called `iterable`. An iterable object is an object that can create iterators - an iterator is created on each dump. We decided that such approach is more convenient than copying an iterator in initial state on each dump. Iterable object and created iterator have no destructor and any invalidation logic, so author of pushed object must think about iterator invalidation by himself. What's about destructors, user of port_c must ensure that the object is alive as long as the port can be dumped. When port_c is not needed anymore, the object can be deleted by caller. Created iterator has no destructor because we think that iterator shouldn't own any resources - its state is allocated when the iterable object is dumped, and the release of the state lies on the side that dumped the object. Rationale: if you dump iterable object to Lua, the iterator is most likely to be freed only when Lua VM will run out of memory (GC will be called then), so Tarantool resources will be occupied for a long time. Let's try to avoid such situation. NO_CHANGELOG=internal NO_DOC=internal
-
Andrey Saranchin authored
We are going to use ports for triggers. The problem is there is no port implementation that meets all requirements to be used to pass arguments to triggers from C, so let's extend existing port_c. This port is already allows to be dumped several times - let's keep this invariant to call chains of triggers using only one port. Let's store trivial types directly, not packed in MsgPack. Then, we won't need to make a second mempool allocation to store them and we won't need to decode them to read. It's worth noting that port_c has two MsgPack types now - MP and MP_OBJECT. The first now is a MsgPack packet that will be unpacked on dump if it's possible. For example, we can pass a field of tuple, which is a MsgPack packet, to field constraint, and it will be unpacked when it will be dumped to Lua, so the constraint argument will have a Lua value, not MsgPack. On the other hand, we want to dump MsgPack as a MsgPack object sometimes. For instance, we use MsgPack object to pass request header and body to iproto override handlers and space recovery triggers. NO_CHANGELOG=internal NO_DOC=internal
-
Andrey Saranchin authored
Methods `get_msgpack` and `dump_msgpack` generally do the same things except for some details (array header, type of destination buffer), so let's factor out their core part into a separate function - it doesn't make much sense right now, but we are going to populate port_c with new types, and these two functions would become very similar without this refactoring. Along the way, let's panic in these methods when we are out of memory - it's the approach we are taking right now in Tarantool. Corresponding error injection is dropped as well. NO_TEST=refactoring NO_CHANGELOG=refactoring NO_DOC=refactoring
-
Andrey Saranchin authored
For easier maintenance, let's panic when there is no memory for allocation needed to add a value to port - that's the approach we're taking right now in Tarantool and we don't need to check return code of `port_c_add*()` functions because they became non-failing. Functions from C API still return int (which is always zero actually) for backward compatibility. NO_TEST=no behavior changes NO_CHANGELOG=no behavior changes NO_DOC=no behavior changes
-
Andrey Saranchin authored
Since we have a special mp_ctx type for mp_object, incapsulating tuple formats and key translation, let's replace iproto_key_translation with iproto_mp_ctx. NO_TEST=no behavior changes NO_CHANGELOG=internal NO_DOC=internal
-
Andrey Saranchin authored
We are going to populate port_c with mp_object, which requires mp_ctx. However, we want to have an ability to dump one port_c instance several times, so we need to add virtual copy to mp_ctx. It would be fair to say that copy constructor must accept `src` as a constant object, but this approach is inconvinient in C, and such method would be embarrassing to use in our code. So `src` is not constant in copy constructor. Method copy of mp_box_ctx is not implemented because it's not trivial, it probably will require reference counting in tuple_format_map. Anyway, it is not needed now, so that's not a problem. Also, mp_ctx_move required dst to be default initialized. However, since it provides semantics of move constructor, we shouldn't expect dst to be initialized at all, since constructor is used to construct an uninitialized object. So let's drop this requirement along the way. NO_TEST=trivial NO_CHANGELOG=internal NO_DOC=internal
-
Andrey Saranchin authored
Since fiber lua state can be used for internal purposes, we cannot truncate it, so we need an abitily to create port_lua with lower index boundary. The patch populates the port with field bottom - dump will start from this index at Lua stack. Also, the patch modifies port_lua destructor - all unused values are popped from lua stack now to prevent Lua stack overflow. NO_CHANGELOG=internal NO_DOC=internal
-
Andrey Saranchin authored
Since we are going to pass arguments to triggers with ports, we need to move all functions that fire triggers to box because ports are implemented there. Making functions that run triggers external (similar to cord_on_yield) leads to more circular dependencies and problems with linker - I didn't cope with box and box_error circular dependency, so let's use core triggers as callbacks to avoid this problem. NO_TEST=no behavior changes NO_CHANGELOG=no behavior changes NO_DOC=no behavior changes
-
Andrey Saranchin authored
Since we are going to rewrite func_adapter to use ports for arguments and return values, submodule port must be initialized before `box.cfg{}` because triggers are allowed to be used before database configuration. So let's init and free port submodule in `box_init` and `box_free` functions instead of `box_storage_init` and `box_storage_free` - `box_storage` is initialized on `box.cfg{}` call. NO_TEST=no behavior changes NO_CHANGELOG=no behavior changes NO_DOC=no behvior changes
-
Nikolay Shirokovskiy authored
We are going keep existing format messages for most of existing error codes as mentioned in more details in the previous commit. But sometimes we want error constructor argument to be formatted and to be added to payload differently. One example is `ER_TUPLE_FOUND` we want tuples to be formatted as string as before in formatted string but want to add them as `TUPLE`. For this purpose let's use "" payload field name. In this case error constructor argument will be used for formatted string but will not be added to the error payload. Follow up #9109 NO_CHANGELOG=internal NO_DOC=internal
-
Nikolay Shirokovskiy authored
We are going keep existing format messages for most of existing error codes. This way we will keep all the details in the error messages until `compat.BOX_ERROR_SERIALIZE_VERBOSE` is switched to 'new' so details will not disappear from logs/console output. Also we won't need to change existing Tarantool and integration tests. Follow up #9109 NO_CHANGELOG=internal NO_DOC=internal
-
Alexander Turenko authored
This commit increases a time to wait of the process termination. It may take longer than 5 seconds, when tarantool is built with an address sanitizer. The address sanitizer generates a report at the process termination and it is not always a fast thing. NO_DOC=test fix NO_CHANGELOG=see NO_DOC
-
- Mar 25, 2024
-
-
Mergen Imeev authored
Closes #9643 @TarantoolBot document Title: config: changes in `credentials` section Now the privileges that were not granted by the configuration, as well as privileges that were not granted solely by the configuration, are not revoked on reload. Privileges that have been granted only by the config module will still be revoked if they are removed from the `credentials` section on reload.
-
Mergen Imeev authored
Currently, if we try to run Tarantool 3.0 with config using old snapshot, we may get a SCHEMA_NEEDS_UPGRADE error because granting and revoking privileges are DDL operations. This leads to a situation where loading Tarantool to perform an upgrade becomes quite problematic. To avoid the issue, this patch causes 'credentials.lua' to issue a warning instead of an error in case of the SCHEMA_NEEDS_UPGRADE error during granting and revoking privileges. Note that it was still possible to startup and perform the upgrade by removing the 'credentials' section from the config or without using config. This is only a part of the solution, the issue will be fixed in #9849. Part of #9849 Needed for #9643 NO_DOC=will be added later NO_CHANGELOG=will be added later
-
- Mar 23, 2024
-
-
Gleb Kashkin authored
Before this patch, Tarantool, when started with empty configuration file, used to fail with a non verbose error: ``` $ ./src/tarantool -n instance-001 -c single.yaml LuajitError: [cluster_config] Unexpected data type for a record: "nil" fatal error, exiting the event loop ``` Now Tarantool instance can be set up with configuration from remote configuration sources available in Tarantool Enterprise Edition and run with empty config file. When there is no configuration provided for the instance an according error is thrown. Closes #9845 NO_DOC=bugfix
-
- Mar 22, 2024
-
-
Vladimir Davydov authored
Currently, the error location (file, line) is always retrieved from the current stack frame. The new argument `level` allows to change this. It has the same semantics as the `level` argument of the built-in Lua function `error`. Closes #9792 @TarantoolBot document Title: Document `level` argument of `box.error` and `box.error.new` Now, when used with a table argument, `box.error` and `box.error.new` also accept an optional second argument called `level`. It has the same meaning as the `level` argument of the built-in Lua function `error`: it specifies how to get the error location. With level 1 (the default), the error location is where `box.error` / `box.error.new` was called. Level 2 points the error to where the function that called `box.error` / `box.error.new` was called; and so on. Passing level 0 avoids addition of location information to the error. Example of using `level` with `box.error`: ```lua local json = require('json') local function inner(level) box.error({message = 'test'}, level) -- line:4 end local function outer(level) inner(level) -- line:8 end local ok, err ok, err = pcall(outer) print(json.encode(err.trace)) -- prints line:4 ok, err = pcall(outer, 1) print(json.encode(err.trace)) -- prints line:4 ok, err = pcall(outer, 2) print(json.encode(err.trace)) -- prints line:8 ok, err = pcall(outer, 0) print(json.encode(err.trace)) -- prints empty table ``` Example of using `level` with `box.error.new`: ```lua local json = require('json') local function inner(level) local err = box.error.new({message = 'test'}, level) -- line:4 return err end local function outer(level) local err = inner(level) -- line:9 return err end local err err = outer() print(json.encode(err.trace)) -- prints line:4 err = outer(1) print(json.encode(err.trace)) -- prints line:4 err = outer(2) print(json.encode(err.trace)) -- prints line:9 ok, err = pcall(outer, 0) print(json.encode(err.trace)) -- prints empty table ``` It is also possible to specify `level` when using `box.error` to re-raise an error created earlier with `box.error.new`, for example: ```lua local json = require('json') local err0 = box.error.new{message = 'test'} -- line:3 local function raise(err, level) box.error(err, level) -- line:6 end ok, err = pcall(raise, err0) print(json.encode(err.trace)) -- prints line:3 ok, err = pcall(raise, err0, 1) print(json.encode(err.trace)) -- prints line:6 ok, err = pcall(raise, err0, 0) print(json.encode(err.trace)) -- prints empty table ```
-
- Mar 20, 2024
-
-
Nikolay Shirokovskiy authored
Follow-up #9109 NO_CHANGELOG=internal NO_DOC=internal
-
Mergen Imeev authored
This patch adds a new option '_origin' to the 'box.schema.user.grant()', 'box.schema.role.grant()', 'box.schema.user.revoke()' and 'box.schema.role.revoke()' functions. This is a private option that allows to differentiate privileges based on their origin. If this option is not provided an empty string is used by default. Privileges that were granted using 'grant()' can be revoked using 'revoke()' with the same '_origin'. The motivation for this change is to avoid actually revoking a privilege granted by two actors if one actor revokes it. Needed for #9643 NO_DOC=internal NO_CHANGELOG=internal
-
Mergen Imeev authored
This patch reduces the number of tests in the in2.test.lua test file. This patch also reduces the number of inserted values. This shouldn't affect the test since it's not really an original Tarantool test, but it will reduce the execution time of this test. Currently this test often fails due to a timeout. NO_DOC=test NO_CHANGELOG=test
-
Mergen Imeev authored
Prior to this patch, the OP_SetDiag opcode accepted an error code and error description, which already included the description from the error code. But this makes #9108 more difficult, so now the accepted error description does not include the description from the error code. Note that this change is possible because all accepted error codes use exactly one argument. Additionally, this patch removes p4 from the OP_Halt opcode in one place, since OP_Halt only uses p1 and p2 and does not use p4. Moreover, the address specified in p4 may not be available by the time OP_Halt is executed. Needed for #9108 NO_DOC=internal NO_TEST=internal NO_CHANGELOG=internal
-
Sergey Kaplun authored
* test: set dependencies in BuildTestCLib macro * Add 'cc' file type for saving bytecode. * Fix C file generation in jit.bcsave. * Throw any errors before stack changes in trace stitching. * Fix recording of __concat metamethod. * Check frame size limit before returning to a lower frame. * build: purge sysprof.collapse module * build: fix tool components handling * memprof: refactor `heap_chunk` data structure * memprof: remove unused arguments * memprof: introduce the `--human-readable` option * profilers: introduce event reader module * ci: extend tarantool integration testing Part of #9595 Part of #5994 Follows up #8700 Needed for #9217 NO_DOC=LuaJIT submodule bump NO_TEST=LuaJIT submodule bump
-
Maxim Kokryashkin authored
After the commit 3daf2399 ("ci: fix step parameters for reusable runs") it is now possible to create Tarantool integration workflows for any repository in the Tarantool organization. Considering this, we don't need to run Tarantool tests under the old target for the LuaJIT integration in .test.mk and we can leave only LuaJIT tests in this target for the sake of exotic Tarantool builds testing. NO_DOC=CI NO_TEST=CI NO_CHANGELOG=CI
-
- Mar 19, 2024
-
-
Alexander Turenko authored
Fixes #9680 @TarantoolBot document Title: config: access configuration of other cluster members ## `config:instances()` List all instances of the cluster. Returns a table of the following format. ```lua { [<instance_name>] = { instance_name = <...>, replicaset_name = <...>, group_name = <...>, }, <...> } ``` If an action should be performed for each instance of the given cluster, it can be written this way: ```lua for instance_name in pairs(config:instances()) do action(instance_name) end ``` If replicaset/group names matter, the instances may be filtered this way: ```lua -- Perform an action for all instances of the given replicaset. for instance_name, def in pairs(config:instances()) do if def.replicaset_name == box.info.replicaset.name then action(instance_name) end end ``` ## `config:get(<...>, {instance = '<...>'})` The new `instance` option of the `config:get()` method allows to retrieve a configuration value of another instance from the given cluster. Example: ```lua -- Collect all the enabled roles within the cluster. local enabled_roles = {} for instance_name in pairs(config:instances()) do local roles = config:get('roles', {instance = instance_name}) for _, role in ipairs(roles) do enabled_roles[role] = true end end ``` Note: There is a difference between a missing `instance` option and the `instance` option that is equal to the given instance name. The former returns an instance configuration taking into account instance configuration sources (environment variables). The latter takes into account only cluster configuration, so the environment variables are ignored.
-
Georgiy Lebedev authored
Due to a regression introduced in c13b3a31, the worker fiber is started synchronously, while it should be started asynchronously, in order for the `wait_connected = false` option of `connect` to work correctly. We already explicitly wait from Lua for the connection to become active via `wait_state` when `wait_connected = true`. Closes #9489 NO_DOC=<bugfix>
-
- Mar 18, 2024
-
-
Mergen Imeev authored
Prior to this patch, privileges in the "credentials" section of the configuration could be requested for non-existent spaces, functions, and sequences. However, this was not possible for roles, although essentially the same mechanism is used. The problem is resolved with this patch. Part of #9643 @TarantoolBot document Title: Assign a non-existent credential role A non-existent credential role can be assigned to a user or role in `credential` section of the config. The actual assignment will occur when the assigned credential role is created.
-
Gleb Kashkin authored
Closes #9809 @TarantoolBot document Title: New labels in config schema The new labels are basically maps with string keys and values, that are merged down to instance level. It means that if a replicaset and an instance inside it have label with same key and different value, the actual value is the one instance provided, e.g. ```yaml groups: group-001: replicasets: replicaset-001: labels: foo: 'true' bar: 'true' instances: instance-001: labels: baz: 'true' foo: 'false' ``` Results in: ``` ./instance-001.iproto> require('config'):get('labels') --- - baz: 'true' foo: 'false' bar: 'true' ... ```
-
Ilya Verbin authored
If the number of tuple fields is less than `format->min_field_count`, then some required field is missed, i.e., there is no need to update the `required_fields` bitmap during msgpack decoding. This optimization is valid only if tuple format doesn't contain fields accessed by JSON paths. This patch improves bench_tuple_new by 15-50%, depending on field count. NO_WRAP $ taskset 0x2 ~/benchmark/tools/compare.py benchmarks \ ./tuple.perftest.old ./tuple.perftest.new \ --benchmark_min_warmup_time=10 \ --benchmark_repetitions=30 \ --benchmark_report_aggregates_only=true \ --benchmark_filter=tuple_new [...] Comparing ./tuple.perftest.old to ./tuple.perftest.new Benchmark Time CPU Time Old Time New CPU Old CPU New ------------------------------------------------------------------------------------------------------------------------------------ bench_tuple_new<FORMAT_BASIC>_mean -0.1469 -0.1470 126 107 126 107 bench_tuple_new<FORMAT_BASIC>_median -0.1428 -0.1429 124 106 124 106 bench_tuple_new<FORMAT_BASIC>_stddev +0.0589 +0.0600 4 5 4 5 bench_tuple_new<FORMAT_BASIC>_cv +0.2412 +0.2427 0 0 0 0 bench_tuple_new<FORMAT_SPARSE>_mean -0.3754 -0.3753 3104 1939 3104 1939 bench_tuple_new<FORMAT_SPARSE>_median -0.3749 -0.3747 3071 1920 3071 1920 bench_tuple_new<FORMAT_SPARSE>_stddev -0.3482 -0.3482 85 55 85 55 bench_tuple_new<FORMAT_SPARSE>_cv +0.0434 +0.0434 0 0 0 0 NO_WRAP Needed for tarantool/tarantool-ee#711 NO_DOC=perf improvement
-
Ilya Verbin authored
It is possible to skip MP_NIL by mp_decode_nil(), which is faster than mp_next(). This patch improves bench_tuple_new<FORMAT_SPARSE> by 2.2x. NO_WRAP $ taskset 0x2 ~/benchmark/tools/compare.py benchmarks \ ./tuple.perftest.old ./tuple.perftest.new \ --benchmark_min_warmup_time=10 \ --benchmark_repetitions=30 \ --benchmark_report_aggregates_only=true \ --benchmark_filter=tuple_new\<FORMAT_SPARSE\> [...] Comparing ./tuple.perftest.old to ./tuple.perftest.new Benchmark Time CPU Time Old Time New CPU Old CPU New ------------------------------------------------------------------------------------------------------------------------------------ bench_tuple_new<FORMAT_SPARSE>_mean -0.5525 -0.5525 6985 3126 6985 3126 bench_tuple_new<FORMAT_SPARSE>_median -0.5445 -0.5444 6838 3115 6838 3115 bench_tuple_new<FORMAT_SPARSE>_stddev -0.8368 -0.8367 541 88 541 88 bench_tuple_new<FORMAT_SPARSE>_cv -0.6354 -0.6352 0 0 0 0 NO_WRAP Needed for tarantool/tarantool-ee#711 NO_DOC=perf improvement NO_TEST=perf improvement NO_CHANGELOG=next commit
-
Ilya Verbin authored
Implement `class MpData<FORMAT_SPARSE>`, which generates 1000 fields, 10 of them contain unsigned integers, while the remaining are null. Needed for tarantool/tarantool-ee#711 NO_DOC=perf test NO_TEST=perf test NO_CHANGELOG=perf test
-
Ilya Verbin authored
Currently `class MpData` generates msgpack data with a predefined format, let's call it `FORMAT_BASIC`. This patch allows to extend it with other formats. No functional changes. Needed for tarantool/tarantool-ee#711 NO_DOC=perf test NO_TEST=perf test NO_CHANGELOG=perf test
-
Ilya Verbin authored
The test fails when Tarantool with a renewed schema version recovers from a gh_8937_data/ snapshot, because `luatest/server_instance.lua` executes: box.schema.user.grant('guest', 'super', nil, nil, {if_not_exists = true}) while insert into system spaces is forbidden until schema is upgraded. Fix it by adding `box.schema.user.grant()` to the snapshot. NO_DOC=test NO_CHANGELOG=test
-
- Mar 15, 2024
-
-
Astronomax authored
Fixed a bug when it was possible that the `on_relay_thread_start` trigger handler would continue to use `data` allocated on the stack of `box_collect_confirmed_vclock` after returning from it. Closes #9505 NO_DOC=bugfix NO_TEST=asan
-
Mergen Imeev authored
Closes #9780 @TarantoolBot document Title: Expansion of `config:info()` Before this patch, the `meta` field of the `config:info()` function contained the meta data of the last load, even if it was unsuccessful reload. This can be inconvenient because if the reload fails, we expect that the last configuration that was successfully applied is used, but its metadata was lost. Now `config:info()` can take one argument — the version of the information that should be returned by the function. If no arguments are specified or the argument is the string `v1`, the return value has the same structure as before. If the argument is the string `v2`, the `meta` field in the return value has fields - `active` (meta for the last successfully applied configuration) and `last` (meta of the last loaded configuration). Currently only these values are accepted.
-
- Mar 14, 2024
-
-
Yaroslav Lobankov authored
Bump version of the `codeql-action` action to v3 for fixing an annoying warning that appears in GitHub WebUI: Node.js 16 actions are deprecated. Please update the following actions to use Node.js 20 NO_DOC=ci NO_TEST=ci NO_CHANGELOG=ci
-
Yaroslav Lobankov authored
This patch follows up #9795 where bump of the version of the `checkout` action was skipped for the etcd_integration.yml and static_build_pack_test_deploy.yml workflows. Follows up #9795 NO_DOC=ci NO_TEST=ci NO_CHANGELOG=ci
-
Yaroslav Lobankov authored
It's impossible to run Node.js 20 actions on CentOS 7 due to issue [1]. So if we want to upgrade to Node.js 20 actions, we have to drop testing on CentOS 7 in the packaging workflow. Otherwise, the testing job fails with the following error: /__e/node20/bin/node: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /__e/node20/bin/node) /__e/node20/bin/node: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /__e/node20/bin/node) /__e/node20/bin/node: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /__e/node20/bin/node) /__e/node20/bin/node: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /__e/node20/bin/node) /__e/node20/bin/node: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by /__e/node20/bin/node) /__e/node20/bin/node: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by /__e/node20/bin/node) [1] https://github.com/actions/checkout/issues/1487 NO_DOC=ci NO_TEST=ci NO_CHANGELOG=ci
-
Yaroslav Lobankov authored
Bump version of the `upload-artifact` and `download-artifact` actions to v4 for fixing an annoying warning that appears in GitHub WebUI: Node.js 16 actions are deprecated. Please update the following actions to use Node.js 20 NO_DOC=ci NO_TEST=ci NO_CHANGELOG=ci
-
Georgiy Lebedev authored
Let's add an `__autocomplete` method to enable error autocompletion and make the error object easier to use. It will suggest error object fields (including own and inherited payload fields) and methods. Closes #9107 NO_DOC=<not a documentable feature>
-