- Nov 13, 2019
-
-
Ilya Kosarev authored
alter_space_new doesn't throw anymore. It's usages are updated. Part of #4247
-
Ilya Kosarev authored
user_has_data is used in on_replace_dd_user therefore it has to be cleared from exceptions. Now it doesn't throw any more. It means we also need to clear from exceptions it's subsidiary function: space_has_data. Their usages are updated. Part of #4247
-
Ilya Kosarev authored
space_def_new_from_tuple is used in on_replace_dd_space therefore it has to be cleared from exceptions. Now it doesn't throw any more. It means we also need to clear from exceptions it's subsidiary functions: space_opts_decode, field_def_decode & space_format_decode. Their usages are updated. Part of #4247
-
- Nov 12, 2019
-
-
Nikita Pettik authored
284 int 285 json_path_cmp(const char *a, int a_len, const char *b, int b_len, 286 int index_base) 287 { ... 304 /* Paths a and b must be valid. */ 305 assert(rc_b == 0 && rc_b == 0); Obviously (according to the comment) author implied that both rc_a == 0 and rc_b == 0. Let's fix this small typo.
-
Vladislav Shpilevoy authored
Before the patch a struct xrow_update_field object didn't account array header in its .size and .data members. Indeed, it was not needed, because anyway updates could be only 'flat'. For example, consider the tuple: [mp_array, mp_uint, mp_uint, mp_uint] ^ ^ pos1 pos2 Struct xrow_update_field.size and .data accounted memory from pos1 to pos2, without the array header. Number of fields was stored inside a rope object. This is why it made no sense to keep array header pointer. But now updates are going to be not flat, and not only for array. There will be an update tree. Each node of that tree will describe update of some part of a tuple. Some of the nodes will need to know exact borders of their children, including headers. It is going to be used for fast copying of neighbours of such children. Consider an example. Tuple with one field consisting of nested maps: tuple = {} tuple[1] = { a = { b = { c = { d = {1, 2, 3} } } } } Update: {{'+', '[1].a.b.c.d[1]', 1}, {'+', '[1].a.b.c.d[2]', 1}} To update such a tuple a simple tree will be built: root: [ [1] ] | isolated path: [ 'a.b.c' ] | leaves: [ [1] [2] [3] ] +1 +1 - Root node keeps the whole tuple borders. It is a rope with single field. This single field is a deeply updated map. Such deep multiple updates with long common prefixes are stored as an isolated path + map/array in the end. Here the isolated path is 'a.b.c'. It ends with the terminal array update. Assume, that operations are applied and it is time to save the result. Save starts from the root. Root rope will encode root array header, and will try to save the single field. The single field is an isolated update. It needs to save everything before old {1,2,3}, the new array {2,2,3}, and everything after the old array. The simplest way to do it - know exact borders of the old array {1,2,3} and memcpy all memory before and after. This is exactly what this patch allows to do. Everything before xrow_update_field.data, and after xrow_update_field.data + .size can be safely copied, and is not related to the field. To copy adjacent memory it is not even needed to know field type. Xrow_update_field.data and .size have the same meaning for all field types. Part of #1261
-
Vladislav Shpilevoy authored
They are needed in incoming JSON updates, which are going to solve a task of comparison of two JSON paths, their simultaneous parsing, and digging into a tuple. json_token_cmp() existed before this patch, but it was trying to compare parent pointers too, which is not needed in the JSON updates, since they won't use JSON trees. Needed for #1261
-
Ilya Kosarev authored
There were some _xc functions used in triggers. Now they all are replaced with their non _xc versions. If corresponding _xc version hadn't had any other usages, it was removed. Part of #4247
-
Ilya Kosarev authored
sequence_field_from_tuple is used in set_space_sequence & on_replace_dd_space_sequence therefore it has to be cleared from exceptions. Now it doesn't throw any more. It's usages are updated. Part of #4247
-
Ilya Kosarev authored
sequence_def_new_from_tuple is used in on_replace_dd_sequence therefore it has to be cleared from exceptions. Now it doesn't throw any more. It's usages are updated. Part of #4247
-
Vladislav Shpilevoy authored
Bootstrap and recovery work on behalf of admin. Without the universe access they are not able to even fill system spaces with data. It is better to forbid this ability until someone made their cluster unrecoverable.
-
Vladislav Shpilevoy authored
The admin user has universal privileges before bootstrap or recovery are done. That allows to, for example, bootstrap from a remote master, because to do that the admin should be able to insert into system spaces, such as _priv. But after the patch on online credentials update was implemented (#2763, 48d00b0e) the admin could loose its universal access if, for example, a role was granted to him before universal access was recovered. That happened by two reasons: - Any change in access rights, even in granted roles, led to rebuild of universal access; - Any change in access rights updated the universal access in all existing sessions, thanks to #2763. What happened: two tarantools were started. One of them master, granted 'replication' role to admin. Second node, slave, tried to bootstrap from the master. The slave created an admin session and started loading data. After it loaded 'grant replication role to admin' command, this nullified admin universal access everywhere, including this session. Next rows could not be applied. Closes #4606
-
- Nov 11, 2019
-
-
Alexander V. Tikhonov authored
After the issue #4537 fixed for the data segment size limit, the temporary blocked tests because of it unblocked. Part of #4271
-
- Nov 09, 2019
-
-
Ilya Kosarev authored
func_def_new_from_tuple is used in on_replace_dd_func therefore it has to be cleared from exceptions. Now it doesn't throw any more. It means we also need to clear from exceptions it's subsidiary function func_def_get_ids_from_tuple. Their usages are updated. Part of #4247
-
Ilya Kosarev authored
index_def_new_from_tuple is used in on_replace_dd_index therefore it has to be cleared from exceptions. Now it doesn't throw any more. It means we also need to clear from exceptions it's subsidiary functions: index_def_check_sequence, index_def_check_tuple, index_opts_decode, func_index_check_func. Their usages are updated. Part of #4247
-
Serge Petrenko authored
Implement a new function in Lua fiber library: top(). It returns a table containing fiber cpu usage stats. The table has two entries: "cpu_misses" and "cpu". "cpu" itself is a table listing all the alive fibers and their cpu consumtion. The patch relies on CPU timestamp counter to measure each fiber's time share. Closes #2694 @TarantoolBot document Title: fiber: new function `fiber.top()` `fiber.top()` returns a table of all alive fibers and lists their cpu consumption. Let's take a look at the example: ``` tarantool> fiber.top() --- - cpu: 107/lua: instant: 30.967324490456 time: 0.351821993 average: 25.582738345233 104/lua: instant: 9.6473633128437 time: 0.110869897 average: 7.9693406131877 101/on_shutdown: instant: 0 time: 0 average: 0 103/lua: instant: 9.8026528631511 time: 0.112641118 average: 18.138387232255 106/lua: instant: 20.071174377224 time: 0.226901357 average: 17.077908441831 102/interactive: instant: 0 time: 9.6858e-05 average: 0 105/lua: instant: 9.2461986412164 time: 0.10657528 average: 7.7068458630827 1/sched: instant: 20.265286315108 time: 0.237095335 average: 23.141537169257 cpu_misses: 0 ... ``` The two entries in a table returned by `fiber.top()` are `cpu_misses` and `cpu`. `cpu` itself is a table whose keys are strings containing fiber ids and names. The three metrics available for each fiber are: 1) instant (per cent), which indicates the share of time fiber was executing during the previous event loop iteration 2) average (per cent), which is calculated as an exponential moving average of `instant` values over all previous event loop iterations. 3) time (seconds), which estimates how much cpu time each fiber spent processing during its lifetime. More info on `cpu_misses` field returned by `fiber.top()`: `cpu_misses` indicates the amount of times tx thread detected it was rescheduled on a different cpu core during the last event loop iteration. fiber.top() uses cpu timestamp counter to measure each fiber's execution time. However, each cpu core may have its own counter value (you can only rely on counter deltas if both measurements were taken on the same core, otherwise the delta may even get negative). When tx thread is rescheduled to a different cpu core, tarantool just assumes cpu delta was zero for the latest measurement. This loweres precision of our computations, so the bigger `cpu misses` value the lower the precision of fiber.top() results. Fiber.top() doesn't work on arm architecture at the moment. Please note, that enabling fiber.top() slows down fiber switching by about 15 per cent, so it is disabled by default. To enable it you need to issue `fiber.top_enable()`. You can disable it back after you finished debugging using `fiber.top_disable()`. "Time" entry is also added to each fibers output in fiber.info() (it duplicates "time" entry from fiber.top().cpu per fiber). Note, that "time" is only counted while fiber.top is enabled.
-
Vladislav Shpilevoy authored
Before the patch update was implemented as a set of operations applicable for arrays only. It was ok until field names and JSON paths appearance, because tuple is an array on the top level. But now there are four reasons to allow more complex updates of tuple field internals by JSON paths: - tuple field access by JSON path is allowed so for consistency JSON paths should be allowed in updates as well; - JSON indexes are supported. JSON update should be able to change an indexed field without rewriting half of a tuple, and its full replacement; - Tarantool is going to support documents in storage so JSON path updates is one more step forward; - JSON updates are going to be faster and more compact in WAL than get + in-memory Lua/connector update + replace (or update of a whole tuple field). The patch reworks the current update code in such a way, that now update is not just an array of operations, applied to tuple's top level fields. Now it is a tree, just like tuples are. The concept is to build a tree of xrow_update_field objects. Each updates a part of a tuple. Leafs in the tree contain update operations, specified by a user, as xrow_update_op objects. To make the code support and understanding simpler, the patch splits update implementation into several independent files-modules for each type of an updated field. One file describes how to update an array field, another file - how to update a map field, etc. This commit introduces only array. Just because it was already supported before the patch. Next commits will introduce more types one by one. Besides, the patch makes some minor changes, not separable from this commit: - The big comment about xrow updates in xrow_update.c is updated. Now it describes the tree-idea presented above; - Comments were properly aligned by 66 symbols in all the moved or changed code. Not affected code is kept as is so as not to increase the diff even more; - Added missing comments to moved or changed structures and their attributes such as struct xrow_update, struct xrow_update_op_meta, struct xrow_update_op. - Struct xrow_update_field was significantly reworked. Now it is not just a couple of pointers at tuple's top level array. From now it stores type of the updated field, range of its source data in the original tuple, and a subtree of other update fields applied to the original data. - Added missing comments to some functions which I moved and decided worth commenting alongside, such as xrow_update_op_adjust_field_no(), xrow_update_alloc(). - Functions xrow_update_op_do_f, xrow_update_op_read_arg_f, xrow_update_op_store_f are separated from struct xrow_update, so as they could be called on any updated field in the tree. From this moment they are methods of struct xrow_update_op. They take an op as a first argument (like 'this' in C++), and are applied to a given struct xrow_update_field. Another notable, but not separable, change is a new naming schema for the methods of struct xrow_update_field and struct xrow_update_op. This is motivated by the fact that struct xrow_update_field now has a type, and might be not a terminal. There are now two groups of functions. Generic functions working with struct xrow_update_field of any type: xrow_update_field_sizeof xrow_update_field_store xrow_update_op_do_field_<operation> And typed functions: xrow_update_<type>_sizeof xrow_update_<type>_store xrow_update_op_do_<type>_<operation> Where operation = insert/delete/set/arith ... type = array/map/bar/scalar ... Common functions are used when type of a field to update is not known in advance. For example, when an operation is applied to one of fields of an array, it is not known what a type this field has: another array, scalar, not changed field, map, etc. Common functions do nothing more than just a switch by field type to choose a more specific function. Typed functions work with a specific type. They may change the given field (add a new array element, replace it with a new value, ...), or may forward an operation deeper in case they see that its JSON path is not fully passed yet. Part of #1261
-
Vladislav Shpilevoy authored
That patch finishes transformation of tuple_update public API to xrow_update. Part of #1261
-
Vladislav Shpilevoy authored
Tuple_update is a too general name for the updates implemented in these files. Indeed, a tuple can be updated from Lua, from SQL, from update microlanguage. Xrow_update is a more specific name, which is already widely used in tuple_update.c. Part of #1261
-
- Nov 08, 2019
-
-
Cyrill Gorcunov authored
When invalid command is passed we should send an error message to a client. Instead a nil dereference occurs that causes abnormal exit of a console. This is the regression from 96dbc49d ('box/console: Refactor command handling'). Reported-by:
Mergen Imeev <imeevma@tarantool.org> Signed-off-by:
Cyrill Gorcunov <gorcunov@gmail.com> Reviewed-by:
Alexander Turenko <alexander.turenko@tarantool.org>
-
Alexander V. Tikhonov authored
Added build + test jobs in GitLab-CI and build + test + deploy jobs on Travis-CI for CentOS 8. Updated testing dependencies in the RPM spec to follow the new Python 2 package naming scheme that was introduced in CentOS 8: it uses 'python2-' prefix rather then 'python-'. CentOS 8 does not provide python2-gevent and python2-greenlet packages, so they were pushed to https://packagecloud.io/packpack/backports repository. This repository is enabled in our build image (packpack/packpack:el-8) by default. Those dependencies are build-time, so nothing was changed for a user. The source RPM packages were gathered from https://cbs.centos.org . Disabled app-tap/pwd.test.lua on CentOS 8 due to systemd-nss issue, which was not worked around properly. Filed #4592 to resolved it in the future. Eliminated libunwind runtime dependency (and libunwind-devel build dependency) on CentOS 8, because the base system does not provide it. fiber.info() backtraces and printing of a backtrace after a crash will not be available on this system. Hopefully we'll fix it in the future, filed #4611 on this. Closes #4543 Reviewed-by:
Alexander Turenko <alexander.turenko@tarantool.org> Reviewed-by:
Igor Munkin <imun@tarantool.org>
-
Alexander Turenko authored
After ea5929db ('build: fix OpenSSL linking problems on FreeBSD') we set CFLAGS explicitly (possibly to an empty value) when invoking a configure script for curl. When this parameter is set the script does not use a value of environment variable CFLAGS. Before this commit LDFLAGS environment variable can affect build of curl submodule. This can lead to a problem when a user or a tool set CFLAGS and LDFLAGS both and some linker flag assumes that some compilation flag is present. Here we set empty LDFLAGS explicitly to avoid using of the environment variable. A distributive build tool such as rpmbuild or emerge usually sets CFLAGS and LDFLAGS. The problem with incompatible compiler / linker options has been reveal under rpmbuild on CentOS 8 with hardened build enabled (which is so when backtraces are disabled). It is not clear whether we should follow environment variables or values determined by CMake for CFLAGS, CPPFLAGS and LDFLAGS when building a submodule (such as luajit and curl). Let's decide about this later. Part of #4543. Reviewed-by:
Alexander V. Tikhonov <avtikhon@tarantool.org> Reviewed-by:
Igor Munkin <imun@tarantool.org>
-
- Nov 07, 2019
-
-
Vladislav Shpilevoy authored
Type was displayed in error messages, was returned in meta headers, and a type string is a result of typeof() SQL function. Typeof() always returns lower case type string; meta contained upper case type; error messages contained both. It was necessary to choose one case for everything, and the lower one was chosen. It allows not to break typeof() function which actually might be used by someone.
-
- Nov 05, 2019
-
-
Vladislav Shpilevoy authored
There was a bug that netbox at any schema update called on_connect() triggers. This was due to overcomplicated logic of handling of changes in the netbox state machine. On_connect() was fired each time the machine entered 'active' state, even if its previous states were 'active' and then 'fetch_schema'. The latter state can be entered many times without reconnects. Another bug was about on_disconnect() - it could be fired even if the connection never entered active state. For example, if its first 'fetch_schema' has failed. Now there is an explicit flag showing the machine connect state. The triggers are fired only when it is changed, on 'active' and on any error states. Intermediate states (fetch_schema, auth) do not matter anymore. Thanks @mtrempoltsev for the initial investigation and a draft fix. Closes #4593
-
Mergen Imeev authored
This patch fixes memory leak in lbox_tuple_format_new(). Closes #4588
-
- Nov 01, 2019
-
-
Vladislav Shpilevoy authored
Box.session.su() worked like following: check user existence, create its credentials on the stack, check the function, call the function, destroy the credentials, restore the old credentials. After creating the credentials on the stack the function check could raise a Lua error. It led to the credentials object not being destroyed. As a result, user.credentials_list was pointing at invalid memory. Now there is no errors between creating the temporary credentials and its destruction. Closes #4597
-
Vladislav Shpilevoy authored
This function is supposed to return NULL on an error. For exceptions there is user_find_by_name_xc.
-
Vladislav Shpilevoy authored
Some guest user privileges were not revoked in the end.
-
- Oct 31, 2019
-
-
Vladislav Shpilevoy authored
Func_delete() called credentials_destroy() after func->vtab->destroy(). But appeared, that vtab->destroy() is actually delete, and it frees the func object. Now the func's owner credentials are destroyed before the function is freed. Closes #4597 Follow up #2763
-
- Oct 30, 2019
-
-
Vladislav Shpilevoy authored
Explicit cast uses uppercase, and the patch makes the implicit cast the same. Upper case is the standard according to SQL standard 2011, cast specification 6.13, general rules 11.e: General rules 11) If TD is variable-length character string or large object character string, then let MLTD be the maximum length in characters of TD. e) If SD (source type) is boolean, then Case: i) If SV is True and MLTD is not less than 4, then TV is 'TRUE'. ii) If SV is False and MLTD is not less than 5, then TV is 'FALSE'. iii) Otherwise, an exception condition is raised: data exception — invalid character value for cast. Part of #4462
-
Vladislav Shpilevoy authored
Before the patch LENGTH didn't take boolean argument into account. Now it does and treats like any other non-string argument - stringify and calculate length. It is worth mentioning, that in future LENGTH will discard any non-string argument, see #3929. Part of #4462
-
Vladislav Shpilevoy authored
Argparse module stores unspecified parameter values as boolean true. It led to a problem, that a command line '--value' with 'value' defined as a number or a string, showed a strange error message: Expected number/string, got "true" Even though a user didn't pass any value. Now it shows 'nothing' instead of '"true"'. That is clearer. Follow up #4076
-
Vladislav Shpilevoy authored
There was a complaint that tarantoolctl --show-system option is very hard to use. It incorrectly parsed passed values, and provided strange errors. tarantoolctl cat --show-system true Bad input for parameter "show-system". Expected boolean, got "true" tarantoolctl cat --show-system 1 Bad input for parameter "show-system". Expected boolean, got "1" tarantoolctl cat --show-system=true Bad input for parameter "show-system". Expected boolean, got "true" First of all, appeared that the complaining people didn't read documentation in 'tarantoolctl --help'. It explicitly says, that '--show-system' should go after a file name, and does not have a value. Secondly, even having taken the documentation into account, the errors indeed look ridiculous. 'Expected boolean, got "true"' looks especially weird. The problem appeared to be with argparse module, how it parses boolean parameters, and how stores parameter values not specified in a command line. All parameters were parsed into a dictionary: parameter name -> value. If a name is alone (no value), then it is boolean true. Otherwise it was always a string value. An attempt to specify an explicit parameter value 'true' led to storing string 'true' in that dictionary. Consequential check for boolean parameters was trivial: type(value) == 'boolean', which was obviously wrong, and didn't pass for 'true' string, but passed for an empty value. Closes #4076
-
Vladislav Shpilevoy authored
Credentials is a cache of user universal privileges. And that cache can become outdated in case user privs were changed after creation of the cache. The patch makes user update all its credentials caches with new privileges, via a list of all creds. That solves a couple of real life problems: - If a user managed to connect after box.cfg started listening port, but before access was granted, then he needed a reconnect; - Even if access was granted, a user may connect after box.cfg listen, but before access *is recovered* from _priv space. It was not possible to fix without a reconnect. And this problem affected replication. Closes #2763 Part of #4535 Part of #4536 @TarantoolBot document Title: User privileges update affects existing sessions and objects Previously if user privileges were updated (via `box.schema.user.grant/revoke`), it was not reflected in already existing sessions and objects like functions. Now it is. For example: ``` box.cfg{listen = 3313} box.schema.user.create('test_user', {password = '1'}) function test1() return 'success' end c = require('net.box').connect(box.cfg.listen, { user = 'test_user', password = '1' }) -- Error, no access for this connection. c:call('test1') box.schema.user.grant('test_user', 'execute', 'universe') -- Now works, even though access was granted after -- connection. c:call('test1') ``` A similar thing happens now with `box.session.su` and functions created via `box.schema.func.create` with `setuid` flag. In other words, now user privileges update is reflected everywhere immediately. (cherry picked from commit 06dbcec597f14fae6b3a7fa2361f2ac513099662)
-
Vladislav Shpilevoy authored
Struct credentials is a cache of user's universal privileges. It is static and is never changed after creation. That is a problem. If a user privileges are updated, it is not reflected in his existing credentials caches. This patch reworks credentials API so as now this struct is not just a container for several numbers. It is an object with standard methods like create(), destroy(). A credentials object still is not updated together with its source user, but now at least the API allows to fix that. Next patch will link all struct credentials of a user into a list via which the user will be able to keep the credentials up to date. Part of #2763 (cherry picked from commit a8c3ebdbfc97b72832ebc5d87b681a310cce9589)
-
- Oct 28, 2019
-
-
Alexander Turenko authored
Added --exclude option (#54).
-
Vladislav Shpilevoy authored
First reason - update is a too general name. Tarantool has SQL update, update in Lua, configuration update. So name 'just update' can't be used. 'Xrow update' fits because it works directly with MessagePack tuple internals and because the updates are persisted in WAL in that format. Second reason, without which the first one would not matter - next patches are going to split tuple_update.c into multiple module files. That will make some structures and functions of tuple_update.c be declared in header files, what makes them a public API of xrow update. Public API methods should be prefixed with their subsystem name, and here it is 'xrow_update_'. Part of #1261
-
Vladislav Shpilevoy authored
Rope is a library to define a custom rope data structure with specified type of a stored value, and some rope functions such as split, alloc. It is possible to choose a unique name for a defined rope structure. It was implemented as #define rope_api(x) rope_##rope_name##_##x #define rope rope_##rope_name But with such rope_api definition it was always expanded to `rope_rope_name_<x value>`. So rope_name was basically a constant 'rope_name' regardless what was defined under it. The patch fixes it and makes name generation just like bps_tree.h. Additionally, the name template is changed a bit, now it is <rope_name>_ + rope + _<method> instead of <rope> + _<rope_name>_ + <method> It just appeared to look better. For example, consider rope name 'xrow_update' and method 'size': new: xrow_update_rope_size() old: rope_xrow_update_size() The second name would be generated by the old template and looks wrong. The new name not only looks better, but also conforms with our code style.
-
Kirill Yukhin authored
Now errors are returned using errcode in triggers and this functions wasn't updated to do so. It doesn't throw, so it set to return 0 unconditionally.
-
Kirill Yukhin authored
-
Alexander Turenko authored
This allows to overcome problems when CMake chooses one toolchain to build tarantool, but a library (libluajit.a or libcurl.a) is built using another (incompatible) toolchain. Fixes #4587.
-