- Dec 05, 2019
-
-
Olga Arkhangelskaia authored
Turns on LUAJIT_ENABLE_PAIRSMM flag for tarantool build. Now __pairs/__ipairs metamethods are available. Closes #4650 (cherry picked from commit b504ca1a096a839f3a4fddc72a33457a3f0dc700)
-
Alexander V. Tikhonov authored
Added build + test jobs in GitLab-CI and build + test + deploy jobs on Travis-CI for Fedora 31. Updated testing dependencies in the RPM spec to follow the new Python 2 package naming scheme that was introduced in Fedora 31: it uses python2-' prefix rather then 'python-'. Fedora 31 does not provide python2-gevent and python2-greenlet packages, so they were pushed to https://packagecloud.io/packpack/backports repository. This repository is enabled in our build image (packpack/packpack:fedora-31) by default. Those dependencies are build-time, so nothing was changed for a user. The source RPM packages were gathered from https://rpms.remirepo.net/rpmphp/ . Closes #4612 Reviewed-by:
Alexander Turenko <alexander.turenko@tarantool.org> (cherry picked from commit 9e09b07c)
-
Alexander Turenko authored
Strengthen test_run:cmd() against temporary connection failures (#193). We recently added 'replication/box_set_replication_stress' test that may exceed file descriptor limit. When test_run:cmd() function executes a command ('switch master' in the case), it tries to create a new socket and connect it to test-run's inspector, but it may fail to do so in the case, because of the file descriptor limit. The sockets that the test produces are closed in background, so if we'll keep trying to create and connect a socket we'll succeed once. This is exactly that the test-run's patch doing: it fails test_run:cmd() function only if a socket cannot be connected during 100 seconds. I guess that the reason why sockets are not closed immediately is that relays wait until replicas will close its side of a socket and only then closes its side. Didn't investigate it deeper, to be honest. (cherry picked from commit 5fccf003)
-
- Dec 03, 2019
-
-
Maria authored
Inside json_decode() struct luaL_serializer is allocated on stack, but json context stores pointer to it: 998 static int json_decode(lua_State *l) 999 { ... 1007 if (lua_gettop(l) == 2) { 1008 struct luaL_serializer user_cfg = *luaL_checkserializer(l); 1009 luaL_serializer_parse_options(l, &user_cfg); 1010 lua_pop(l, 1); 1011 json.cfg = &user_cfg; 1012 } Later (for instance in json_decode_descend()), it can be dereferenced which in turn results in stack-use-after-scope (object turns into garbage right after scope is ended). To fix it let's simply avoid allocating and copying luaL_serializer on stack and instead use pointer to it. Bug is found by ASAN: test app-tap/json.test.lua fails with enabled ASAN. Current fix allows to pass all tests. Thanks to @Korablev77 for the initial investigation. Closes #4637 (cherry picked from commit 6508ddb7)
-
- Dec 02, 2019
-
-
Alexander Turenko authored
This reverts commit a0b196dd. This commit was pushed occasionally and points to a draft commit in test-run repository. See also https://github.com/tarantool/test-run/issues/195 (cherry picked from commit 4acdeeda)
-
Ilya Kosarev authored
There were some pass conditions in quorum test which could take some time to be satisfied. Now they are wrapped using test_run:wait_cond to make the test stable. Closes #4586 (cherry picked from commit f6775e86)
-
Ilya Kosarev authored
In replicaset_follow we iterate anon replicas list: list of replicas that haven't received an UUID. In case of successful connect replica link is being removed from anon list. If it happens immediately, without yield in applier, iteration breaks. Now it is fixed by rlist_foreach_entry_safe instead of common rlist_foreach_entry. Relevant test case is added. Part of #4586 Closes #4576 Closes #4440 (cherry picked from commit 6f038f4b)
-
Ilya Kosarev authored
During pruning of appliers some anon replicas might connect from replicaset_follow called in another fiber. Therefore we need to prune appliers of anon replicas first and, moreover, prune them one by one instead of iterating them, as far as any of them might connect while we are stopping the other one and it will break iteration. Part of #4586 Closes #4643 (cherry picked from commit 36ff3c89)
-
Ilya Kosarev authored
Stabilize tcp_connect in test_run:cmd() (tarantool/test-run#193) (cherry picked from commit a0b196dd)
-
- Nov 26, 2019
-
-
Vladislav Shpilevoy authored
Binary session disconnect trigger yield could lead to use after free of the session object. That happened because iproto thread sent two requests to TX thread at disconnect: - Close the session and run its on disconnect triggers; - If all requests are handled, destroy the session. When a connection is idle, all requests are handled, so both these requests are sent. If the first one yielded in TX thread, the second one arrived and destroyed the session right under the feet of the first one. This can be solved in two ways - in TX thread, and in iproto thread. Iproto thread solution (which is chosen in the patch): just don't send destroy request until disconnect returns back to iproto thread. TX thread solution (alternative): add a flag which says whether disconnect is processed by TX. When destroy request arrives, it checks the flag. If disconnect is not done, the destroy request waits on a condition variable until it is. The iproto is a bit tricker to implement, but it looks more correct. Closes #4627 (cherry picked from commit 6da9d395)
-
Vladislav Shpilevoy authored
Bootstrap.snap is created from a normal snapshot file, but with erased VClock option in the header: SNAP 0.13 Version: 2.2.1-122-g1146bb78d Instance: 03d3836a-e608-421c-9f8d-ad9beefe7440 VClock: {} In a normal snapshot it is 'VClock: {1: ...}'. To erase the option usually developers use 'vim'. But when a binary file is opened in vim without any arguments, like this: vim bootstrap.snap on close it will edit some parts of the file in unexpected ways, depending on local vim settings. To forbid any implicit changes binary mode should be used: vim -b bootstrap.snap The patch regenerates bootstrap.snap and drops VClock using binary mode vim. Closes #4510
-
- Nov 22, 2019
-
-
Kirill Yukhin authored
Add LUAJIT_ENABLE_PAIRSMM flag as a build option for luajit. If the flag is set, pairs/ipairs metamethods are available in Lua 5.1. For Tarantool this option is enabled by default. (cherry picked from commit 93e710d5de0d723086bda6fedc9cb383a8e5e477)
-
- Nov 21, 2019
-
-
Vladislav Shpilevoy authored
Replication's applier encoded an auth request with exactly the same parameters as extracted by the URI parser. I.e. when no password was specified, the parser returned it as NULL, and it was not encoded. The relay, received such an auth request, complained that IPROTO_TUPLE field is not specified (this is password). Such an error confuses - a user didn't do anything illegal, he just used URI like 'login@host:port', without a password after the login. The patch makes the applier use an empty string as a default password. An alternative was to force a user always set a password even if it is an empty string, like that: 'login:@host:port'. And if a password was not found in an auth request, then reject it with a password mismatch error. But in that case a URI of kind 'login@host:port' becomes useless - it can never pass. In addition, netbox already uses an empty string as a default password. So the only way to make it consistent, and don't break anything - repeat netbox logic for replication URIs. Closes #4605 Conflicts: test/replication/suite.cfg (cherry picked from commit 6c01ca48) Conflicts: test/replication/suite.cfg
-
Vladislav Shpilevoy authored
Functions are stored in lists inside module objects. Module objects are stored in a hash table, where key is a package name. But the key was a pointer at one of module's function definition object. Therefore, when that function was deleted, its freed package name memory was still in the hash key, and could be accessed, when another function was deleted. Now module does not use memory of its functions, and keep a copy of the package name. (cherry picked from commit fa2893ea)
-
- Nov 15, 2019
-
-
Alexander Turenko authored
The problem appears after 6c627af3 ('test: tarantoolctl: verify delayed box.cfg()'), where the test case was changed and it doesn't more assume an error at the instance start. So we need to stop it to prevent a situation when instances are stay after `make test`. Fixes #4600. Reviewed-by:
Vladislav Shpilevoy <v.shpilevoy@tarantool.org> (cherry picked from commit 8d363c43)
-
- Nov 14, 2019
-
-
Alexander Turenko authored
Before commit 03f85d4c ('app: fix boolean handling in argparse module') the module does not expect a value after a 'boolean' argument. However there was the problem: a 'boolean' argument can be passed only at end of an argument list, otherwise it wrongly consumes a next argument and gives a confusing error message. The mentioned commit fixes this behaviour in the following way: it still allows to pass a 'boolean' argument at end of the list w/o a value, but requires a value ('true', 'false', '1', '0') if a 'boolean' argument is not at the end to be provided using {'--foo=true'} or {'--foo', 'true'} syntax. Here this behaviour is changed: a 'boolean' argument does not assume an explicitly passed value despite its position in an argument list. If a 'boolean' argument appears in the list, then argparse.parse() returns `true` for its value (a list of `true` values in case of 'boolean+' argument), otherwise it will not be added to the result. This change also makes the behaviour of long (--foo) and short (-f) 'boolean' options consistent. The motivation of the change is simple: it is easier and more natural to type, say, `tarantoolctl cat --show-system 00000000000000000000.snap` then `tarantoolctl cat --show-system true 00000000000000000000.snap`. This commit adds several new test cases, but it does not mean that we guarantee that the module behaviour will not be changed around some corner cases, say, handling of 'boolean+' arguments. This is internal module. Follows up #4076. Reviewed-by:
Vladislav Shpilevoy <v.shpilevoy@tarantool.org> (cherry picked from commit e47f2c91)
-
- Nov 12, 2019
-
-
Vladislav Shpilevoy authored
The admin user has universal privileges before bootstrap or recovery are done. That allows to, for example, bootstrap from a remote master, because to do that the admin should be able to insert into system spaces, such as _priv. But after the patch on online credentials update was implemented (#2763, 48d00b0e) the admin could loose its universal access if, for example, a role was granted to him before universal access was recovered. That happened by two reasons: - Any change in access rights, even in granted roles, led to rebuild of universal access; - Any change in access rights updated the universal access in all existing sessions, thanks to #2763. What happened: two tarantools were started. One of them master, granted 'replication' role to admin. Second node, slave, tried to bootstrap from the master. The slave created an admin session and started loading data. After it loaded 'grant replication role to admin' command, this nullified admin universal access everywhere, including this session. Next rows could not be applied. Closes #4606 (cherry picked from commit 95237ac8)
-
- Nov 11, 2019
-
-
Alexander V. Tikhonov authored
After the issue #4537 fixed for the data segment size limit, the temporary blocked tests because of it unblocked. Part of #4271 (cherry picked from commit e6866550)
-
- Nov 08, 2019
-
-
Cyrill Gorcunov authored
When invalid command is passed we should send an error message to a client. Instead a nil dereference occurs that causes abnormal exit of a console. This is the regression from 96dbc49d ('box/console: Refactor command handling'). Reported-by:
Mergen Imeev <imeevma@tarantool.org> Signed-off-by:
Cyrill Gorcunov <gorcunov@gmail.com> Reviewed-by:
Alexander Turenko <alexander.turenko@tarantool.org> (cherry picked from commit ada8c97c)
-
Alexander V. Tikhonov authored
Added build + test jobs in GitLab-CI and build + test + deploy jobs on Travis-CI for CentOS 8. Updated testing dependencies in the RPM spec to follow the new Python 2 package naming scheme that was introduced in CentOS 8: it uses 'python2-' prefix rather then 'python-'. CentOS 8 does not provide python2-gevent and python2-greenlet packages, so they were pushed to https://packagecloud.io/packpack/backports repository. This repository is enabled in our build image (packpack/packpack:el-8) by default. Those dependencies are build-time, so nothing was changed for a user. The source RPM packages were gathered from https://cbs.centos.org . Disabled app-tap/pwd.test.lua on CentOS 8 due to systemd-nss issue, which was not worked around properly. Filed #4592 to resolved it in the future. Eliminated libunwind runtime dependency (and libunwind-devel build dependency) on CentOS 8, because the base system does not provide it. fiber.info() backtraces and printing of a backtrace after a crash will not be available on this system. Hopefully we'll fix it in the future, filed #4611 on this. Closes #4543 Reviewed-by:
Alexander Turenko <alexander.turenko@tarantool.org> Reviewed-by:
Igor Munkin <imun@tarantool.org> (cherry picked from commit e3d9d8c9)
-
Alexander Turenko authored
After ea5929db ('build: fix OpenSSL linking problems on FreeBSD') we set CFLAGS explicitly (possibly to an empty value) when invoking a configure script for curl. When this parameter is set the script does not use a value of environment variable CFLAGS. Before this commit LDFLAGS environment variable can affect build of curl submodule. This can lead to a problem when a user or a tool set CFLAGS and LDFLAGS both and some linker flag assumes that some compilation flag is present. Here we set empty LDFLAGS explicitly to avoid using of the environment variable. A distributive build tool such as rpmbuild or emerge usually sets CFLAGS and LDFLAGS. The problem with incompatible compiler / linker options has been reveal under rpmbuild on CentOS 8 with hardened build enabled (which is so when backtraces are disabled). It is not clear whether we should follow environment variables or values determined by CMake for CFLAGS, CPPFLAGS and LDFLAGS when building a submodule (such as luajit and curl). Let's decide about this later. Part of #4543. Reviewed-by:
Alexander V. Tikhonov <avtikhon@tarantool.org> Reviewed-by:
Igor Munkin <imun@tarantool.org> (cherry picked from commit 0bead600)
-
- Nov 05, 2019
-
-
Vladislav Shpilevoy authored
There was a bug that netbox at any schema update called on_connect() triggers. This was due to overcomplicated logic of handling of changes in the netbox state machine. On_connect() was fired each time the machine entered 'active' state, even if its previous states were 'active' and then 'fetch_schema'. The latter state can be entered many times without reconnects. Another bug was about on_disconnect() - it could be fired even if the connection never entered active state. For example, if its first 'fetch_schema' has failed. Now there is an explicit flag showing the machine connect state. The triggers are fired only when it is changed, on 'active' and on any error states. Intermediate states (fetch_schema, auth) do not matter anymore. Thanks @mtrempoltsev for the initial investigation and a draft fix. Closes #4593 (cherry picked from commit d56d869a)
-
Mergen Imeev authored
This patch fixes memory leak in lbox_tuple_format_new(). Closes #4588 (cherry picked from commit 96199855)
-
- Nov 01, 2019
-
-
Vladislav Shpilevoy authored
Box.session.su() worked like following: check user existence, create its credentials on the stack, check the function, call the function, destroy the credentials, restore the old credentials. After creating the credentials on the stack the function check could raise a Lua error. It led to the credentials object not being destroyed. As a result, user.credentials_list was pointing at invalid memory. Now there is no errors between creating the temporary credentials and its destruction. Closes #4597 (cherry picked from commit 2bb8d1ea)
-
Vladislav Shpilevoy authored
This function is supposed to return NULL on an error. For exceptions there is user_find_by_name_xc. (cherry picked from commit 8b6bdb43)
-
Vladislav Shpilevoy authored
Some guest user privileges were not revoked in the end. (cherry picked from commit 44d4555b)
-
- Oct 31, 2019
-
-
Vladislav Shpilevoy authored
Func_delete() called credentials_destroy() after func->vtab->destroy(). But appeared, that vtab->destroy() is actually delete, and it frees the func object. Now the func's owner credentials are destroyed before the function is freed. Closes #4597 Follow up #2763 (cherry picked from commit 330ea240)
-
- Oct 30, 2019
-
-
Vladislav Shpilevoy authored
Argparse module stores unspecified parameter values as boolean true. It led to a problem, that a command line '--value' with 'value' defined as a number or a string, showed a strange error message: Expected number/string, got "true" Even though a user didn't pass any value. Now it shows 'nothing' instead of '"true"'. That is clearer. Follow up #4076 (cherry picked from commit c214d086)
-
Vladislav Shpilevoy authored
There was a complaint that tarantoolctl --show-system option is very hard to use. It incorrectly parsed passed values, and provided strange errors. tarantoolctl cat --show-system true Bad input for parameter "show-system". Expected boolean, got "true" tarantoolctl cat --show-system 1 Bad input for parameter "show-system". Expected boolean, got "1" tarantoolctl cat --show-system=true Bad input for parameter "show-system". Expected boolean, got "true" First of all, appeared that the complaining people didn't read documentation in 'tarantoolctl --help'. It explicitly says, that '--show-system' should go after a file name, and does not have a value. Secondly, even having taken the documentation into account, the errors indeed look ridiculous. 'Expected boolean, got "true"' looks especially weird. The problem appeared to be with argparse module, how it parses boolean parameters, and how stores parameter values not specified in a command line. All parameters were parsed into a dictionary: parameter name -> value. If a name is alone (no value), then it is boolean true. Otherwise it was always a string value. An attempt to specify an explicit parameter value 'true' led to storing string 'true' in that dictionary. Consequential check for boolean parameters was trivial: type(value) == 'boolean', which was obviously wrong, and didn't pass for 'true' string, but passed for an empty value. Closes #4076 (cherry picked from commit 03f85d4c)
-
Vladislav Shpilevoy authored
Credentials is a cache of user universal privileges. And that cache can become outdated in case user privs were changed after creation of the cache. The patch makes user update all its credentials caches with new privileges, via a list of all creds. That solves a couple of real life problems: - If a user managed to connect after box.cfg started listening port, but before access was granted, then he needed a reconnect; - Even if access was granted, a user may connect after box.cfg listen, but before access *is recovered* from _priv space. It was not possible to fix without a reconnect. And this problem affected replication. Closes #2763 Part of #4535 Part of #4536 @TarantoolBot document Title: User privileges update affects existing sessions and objects Previously if user privileges were updated (via `box.schema.user.grant/revoke`), it was not reflected in already existing sessions and objects like functions. Now it is. For example: ``` box.cfg{listen = 3313} box.schema.user.create('test_user', {password = '1'}) function test1() return 'success' end c = require('net.box').connect(box.cfg.listen, { user = 'test_user', password = '1' }) -- Error, no access for this connection. c:call('test1') box.schema.user.grant('test_user', 'execute', 'universe') -- Now works, even though access was granted after -- connection. c:call('test1') ``` A similar thing happens now with `box.session.su` and functions created via `box.schema.func.create` with `setuid` flag. In other words, now user privileges update is reflected everywhere immediately. (cherry picked from commit 06dbcec597f14fae6b3a7fa2361f2ac513099662) (cherry picked from commit 2b599c0efa9ae265fb7464af6abae3f6a192e30e)
-
Vladislav Shpilevoy authored
Struct credentials is a cache of user's universal privileges. It is static and is never changed after creation. That is a problem. If a user privileges are updated, it is not reflected in his existing credentials caches. This patch reworks credentials API so as now this struct is not just a container for several numbers. It is an object with standard methods like create(), destroy(). A credentials object still is not updated together with its source user, but now at least the API allows to fix that. Next patch will link all struct credentials of a user into a list via which the user will be able to keep the credentials up to date. Part of #2763 (cherry picked from commit a8c3ebdbfc97b72832ebc5d87b681a310cce9589) (cherry picked from commit 6b15dce614cfc3b14a12b66819737263a5089eaf)
-
- Oct 28, 2019
-
-
Alexander Turenko authored
Added --exclude option (#54). (cherry picked from commit c17c10a4)
-
Alexander Turenko authored
This allows to overcome problems when CMake chooses one toolchain to build tarantool, but a library (libluajit.a or libcurl.a) is built using another (incompatible) toolchain. Fixes #4587. (cherry picked from commit 1eead75e)
-
Alexander Turenko authored
FreeBSD has OpenSSL as part of the base system: libraries are located in /usr/lib, headers are in /usr/include. However a user may install the library into /usr/local/{lib,include} from ports / pkg. In this case tarantool did choose /usr/local version, while libcurl will pick up a base system library. This is fixed by passing --with-ssl option with an argument (/usr/local or /usr if custom -DOPENSSL_ROOT_DIR=<...> is not passed). Now the behaviour is the following. If -DOPENSSL_ROOT_DIR=<...> is passed, then try to use OpenSSL from it. Otherwise find the library in /usr/local and then in /usr. This is right as for tarantool's crypto module as well as for libcurl submodule. There is a flaw here: a user is unable to choose a base system library if a ports / pkg version of OpenSSL is installed. The reason here is that tarantool's crypto module depends on other libraries and -I/usr/local/include may be added to build options. I have no good solution for that, so `cmake . -DOPENSSL_ROOT_DIR=/usr` will give a warning on FreeBSD and `gmake` likely will fail if libraries are of different versions (see cmake/os.cmake comments for more information). See also a [discussion][1] in FreeBSD community about all those /usr and /usr/local problems. There were two other problems that may fail tarantool build on FreeBSD: they are fixed in this commit and described below. First, libcurl's configure script chooses GCC by default if it exists (say, installed from ports / pkg). It is unexpected behaviour when tarantool sources itself are built with clang. Now it is fixed by passing a compiler explicitly to the libcurl's configure script: the library will use base system clang by default or one that a user pass to tarantool's cmake. Side note: GCC has /usr/local/include in its default headers search paths; libcurl's configure script chooses GCC as a compiler and OpenSSL from a base system by default (when CC and --with-ssl=<...> are not set) that leads to OpenSSL header / library mismatch. It is the primary reason of the build fail that was fixed in 1f2338bd ('build: FreeBSD packages installation'). It is not much relevant anymore, because we don't try to link with a base system OpenSSL if /usr/local one exists (however if it is asked explicitly with -DOPENSSL_ROOT_DIR=<...> we'll do, but will give a warning). Anyway, it is important to know such details if we'll change build scripts in a future. Second, backtraces are not supported on FreeBSD, but were enabled if libunwind headers is found. This leads to an error on cmake stage, because of inability to find a right library (this is a bug). Now we disable backtraces on FreeBSD by default even if libunwind is found. See When CC is passed to libcurl's configure script, the new problem opens on Mac OS. CMake chooses XCode toolchain by default (at least on a particular system where I tried it), which requires -isysroot=<SDK_PATH> option to be passed to a preprocessor and a compiler in order to find system headers. See [2] for more information. [1]: https://wiki.freebsd.org/WarnerLosh/UsrLocal [2]: https://developer.apple.com/documentation/xcode_release_notes/xcode_10_release_notes#3035623 Follows up #4490. (cherry picked from commit ea5929db)
-
Vladislav Shpilevoy authored
Before the patch there was a race in replication password configuration. It was possible that a replica connects to a master with a custom password before that password is actually set. The replica treated the error as critical and exited. But in fact it is not critical. Replica even can withstand absence of a user and keeps reconnecting. Wrong password situation arises from the same problem of non atomic configuration and is fixed the same - keep reconnect attempts if the password was wrong. Closes #4550 (cherry picked from commit aa2e2c56)
-
Vladislav Shpilevoy authored
The previous patch introduced a way to set box.cfg options in a strict order, even on a reconfiguration. It was used to set listen before replication. The same order problem existed for replication settings. A user could do box.cfg{ replication_connect_quorum = 0, replication = {...} } and expect, that due to quorum 0 the cfg() will return immediately. But actually the behaviour was undefined - due to arbitrary order of keys in a Lua table, replication could be applied before quorum. The patch makes all replication settings be applied before replication. Follow up #4433 Part of #3760 (cherry picked from commit 00c6c437)
-
Vladislav Shpilevoy authored
Before the patch the nil UUID was ignored and a new random one was generated. This was because internally box treats nil UUID as its absence. Now a user will see an explicit message that nil UUID is a reserved value. Closes #4282 (cherry picked from commit a8ebd334)
-
- Oct 24, 2019
-
-
Alexander V. Tikhonov authored
Added Ubuntu 19.10 Eoan Ermine into CI. Close #4583 Reviewed-by:
Alexander Turenko <alexander.turenko@tarantool.org> (cherry picked from commit c6cd2e62)
-
Oleg Babin authored
Before this patch RUN_TESTS condition in Dockerfile.staticbuild was ignored and always was true. However adding of brackets solves only part of problem. If RUN_TESTS is empty `sh -c` returns 1 and build fails. However if we run tests we should fail build if tests are not passed. Ternary logic was rewritten to fair if-else. This patch fixes it and allows build tarantool statically without running tests. @Totktonada: Fixed .gitlab.mk to pass RUN_TESTS environment variable to docker build arguments. Reviewed-by:
Alexander Turenko <alexander.turenko@tarantool.org> (cherry picked from commit 8c85dbc7)
-
- Oct 23, 2019
-
-
Serge Petrenko authored
After we started using bundled version of libyaml by default (see commit 47b91e90), we can remove it from building dependencies for RPM and DEB packages. Closes #4442 Reviewed-by:
Alexander Turenko <alexander.turenko@tarantool.org> (cherry picked from commit 1d4e584a)
-