- Jan 31, 2022
-
-
Andrei Sidorov authored
Fix static build for macOS 11.5 or higher. On macOS SDK ver. 11.5 some `*.dylib` files was replaced with `*.tbd`. So we replace `libunwind.dylib` on `libunwind.tbd`. Because of macOS 10.15 support being dropped conditional is not needed. Closes #6052
-
Vladimir Davydov authored
Because of the typo, the test fails on my localhost like this: There is no test with name 'test_ignore_with_force_recovery' but hook 'after_test' is defined for it (I use luatest installed with luarocks). Follow-up #6794 Follow-up 8c8b7739 ("recovery: fix recovering from 1.6 xlogs/snaps")
-
Serge Petrenko authored
Tarantool 1.6 didn't rotate xlogs on snap creation. This means, that Tarantool 1.6 data dir may contain files like 0...0.xlog and 0...01.snap, where 0...0.xlog covers changes both before and after the snap creation. New Tarantool versions fail to recover from such files: they recover the data correctly, but fail to update the vclock accordingly. Let's allow recovering from such files and update the vclock accordingly, but only when force_recovery is set. Patch 634f59c7 ("recovery: panic in case of recovery and replicaset vclock mismatch") added the panic, which helped to discover the issue. Before that patch Tarantool would silently recover with an outdated vclock. Also, while we're at it, refactor the check leading to a panic: it never happens after hot_standby, so move it into "else" branch of hot_standby. Closes #6794
-
- Jan 30, 2022
-
-
Georgiy Lebedev authored
Privileges passed to box.schema.user.grant are resolved in quite a naïve manner: we do not check that all of them have been resolved. Also, the privilege_resolve function itself is poorly designed (instead of generic string processing, we have a bunch of if's): extract parsing of the privileges string into privilege_parse and process it through a generic FSA. Closes #6199
-
- Jan 29, 2022
-
-
Alexander Turenko authored
Well, the diff is not nice, so I'll summarize the changes for humans. General changes: * Moved patch submission and so on to the end of the document. * Reformatted to usual 80 columns, inlined and actualized links. * Reformatted lists: start from a capital letter, end with a period. * Minor wording tweaks to make it sound better. Badges: * Travis CI and GitLab CI -> GitHub Actions. * Removed Slack and Gitter. * Google Groups -> GitHub Discussions. * Added Stack Overflow. Define Tarantool as in-memory computing platform as on the website. Added license information. Appserver: * Rewrote the sentence regarding compatibility with Lua 5.1 and highlighted key LuaJIT features instead. There are doubts within the team about declaring 100% compatibility with Lua 5.1, let's go to the safe ground so. * Splitted the list in a more granular way. * Added links to mysql and pg connectors. * Added queue, vshard, cartridge with links. * Added a link to the modules page. Database: * Replaced MsgPack with MessagePack. It is called so on the official website. * Replaced 'optional persistence' with 'complete WAL-based persistence' in the memtx description to make it less confusing. There are users, which believe that memtx does not offer persistency. * Moved SQL down. * Added JSON path indexes. * Added synchronous quorum-based replication. * Added RAFT-based automatic leader election. * Added a link to the connectors list. Platforms: * Dropped OpenBSD from the list. It is not verified in CI, so I feel it like 'we have no commitments here'. * Replaced x86 with more explicit x86_64. * Moved Mac OS before FreeBSD (we have more users here). * Added aarch64/M1. Tarantool is ideal for... * Kept intact, just reformatted to 80 columns. Download: * Added 'or using Docker' in addition to 'as a binary package to your OS'. Added a link to the 'Awesome Tarantool' list. Report bugs / send feedback or patch: * Replaced Google Groups with GitHub Discussions. * Added Stack Overflow. * Highlighted that we accept pull requests and linked 'How to get involved' guide. Fixes #6579
-
- Jan 28, 2022
-
-
Igor Munkin authored
* Actually implement maxirconst trace limit. * Fix string.char() recording with no arguments. Closes #6371 Part of #6548
-
Vladimir Davydov authored
This commit adds a new script, tools/check-commits. The script takes git revisions in the same format as git-rev-list (actually, it feeds all its arguments to git-rev-list) and runs some checks on each of the commits. For example, to check the head commit, run: ./tools/check-commits -1 HEAD To check the last five commits, run: ./tools/check-commits HEAD~5..HEAD Currently, there are only two checks, but in future we may add more checks, e.g. check diffs for trailing spaces: - The commit message contains a documentation request. Can be suppressed with NO_DOC=<reason> in the commit message. - A new changelog entry is added to changelog/unreleased by the commit. Can be suppressed with NO_CHANGELOG=<reason> in the commit message. This commit also adds a new workflow triggered on pull request, lint/commits. The workflow runs the tools/check-commits script on all the commits added by the pull request. The workflow doesn't run on push, because it's problematic to figure out what commits are new on a branch. Besides, we don't want to run it on push to release branches, because it's a pure dev workflow. Example output: Checking commit a33f3cc7 PASS Checking commit 6f29f9d7 FAIL SHA: 6f29f9d7 SUBJECT: iproto: introduce graceful shutdown protocol ERROR: Changelog not found in changelog/unreleased. If this commit doesn't require changelog, please add NO_CHANGELOG=<reason> to the commit message. Checking commit fbc25aae FAIL SHA: fbc25aae SUBJECT: Update small submodule ERROR: Missing documentation request ('@TarantoolBot document' not found in the commit message). If this commit doesn't need to be documented, please add NO_DOC=<reason> to the commit message. ERROR: Changelog not found in changelog/unreleased. If this commit doesn't require changelog, please add NO_CHANGELOG=<reason> to the commit message. NO_DOC=ci NO_CHANGELOG=ci
-
- Jan 27, 2022
-
-
Yaroslav Lobankov authored
Now the tarantool/testing:debian-stretch image from Docker Hub is used. Closes tarantool/tarantool-qa#136
-
- Jan 26, 2022
-
-
Nick Volynkin authored
-
Vladimir Davydov authored
Closes #5924 @TarantoolBot document Title: Document graceful shutdown of net.box connections If a Tarantool server supports the IPROTO graceful shutdown protocol (`connection.peer_protocol_features` contains `graceful_shutdown` or `peer_protocol_version` is >= 4), then when the server is asked to exit (`os.exit()` is called on the server or `SIGTERM` signal is received), it won't terminate net.box connections immediately. Instead here's what will happen: 1. The server stops accepting new connections and sends a special 'shutdown' packet to all connection that support the graceful shutdown protocol. 2. Upon receiving a 'shutdown' packet, a net.box connection executes shutdown triggers. The triggers are installed by `on_shutdown()` method of a net.box connection. The method follows the same protocol as `on_connect()`, `on_disconnect()`, and `on_schema_reload()`. Triggers are executed asynchronously in a new fiber. The connection remains active while triggers are running so a trigger callback may send new requests over the net.box connection. 3. After shutdown triggers have returned, the connection is switched to `graceful_shutdown` state, in which all new requests fail with an error. The connection will remain in this state until all requests have been completed. 4. Once all in-progress requests have completed, the connection is closed and switched to `error` or `error_reconnect` state, depending on whether `reconnect_after` option is set. 5. Once all connections that support the graceful shutdown protocol are closed, the server exits. Note, the graceful shutdown protocol is best-effort: there's no guarantee that the server doesn't exit before all active connections are gracefully closed; the server may still exit on timeout or just be killed. The timeout is configured by `box.ctl.set_on_shutdown_timeout()` on a server. Please also update the net.box state machine diagram: ``` initial -> auth -> fetch_schema <-> active fetch_schema, active -> graceful_shutdown (any state, on error) -> error_reconnect -> auth -> ... \ -> error (any state, but 'error') -> closed ```
-
Vladimir Davydov authored
This commit adds the graceful shutdown feature to the IPROTO protocol. Net.box is patched by the next commit, which also adds tests. Part of #5924 @TarantoolBot document Title: Document IPROTO graceful shutdown A new IPROTO request type was introduced - `IPROTO_SHUTDOWN`, code 77. When asked to shut down (`os.exit()` is called or `SIGTERM` signal is received), a server stops accepting new connections and sends a packet of this type to each of its clients that support the graceful shutdown feature (see below how a server figures out if a client supports the feature). The server won't exit until all the clients that were sent the packets close connections. If all the clients don't close connections within the shutdown timeout, the server will exit anyway. The default shutdown timeout is 3 seconds, and it can be configured with `box.ctl.set_on_shutdown_timeout()`, which also determines the timeout of `box.ctl.on_shutdown()` triggers. An `IPROTO_SHUTDOWN` packet doesn't have any keys in its headers (not even sync number or schema version) nor a body. A client isn't supposed to reply to an `IPROTO_SHUTDOWN` packet. Instead it's supposed to close its connection as soon as possible. A client may wait for pending requests to complete and even send new requests after receiving an `IPROTO_SHUTDOWN` packet. The server will serve them as usual until it exits on timeout. Clients that support the graceful shutdown feature are supposed to set the `IPROTO_FEATURE_GRACEFUL_SHUTDOWN` feature (bit 4) when sending an `IPROTO_ID` request to a server. Servers that support the feature set the same bit in reply to an `IPROTO_ID` request. Introduction of this feature bumped the IPROTO protocol version up to 4.
-
Vladimir Davydov authored
Needed to fix rlist_foreach_entry ASAN runtime error in case the rlist member is aligned.
-
- Jan 25, 2022
-
-
Vladislav Shpilevoy authored
Raft needs to know cluster size in order to detect and handle split vote. The patch uses registered server count as cluster size. It is not documented nor has a changelog file because this is an optimization. Can't be observed except in logs or with a watch. Closes #5285
-
Vladislav Shpilevoy authored
Split vote is a situation when during election nobody can win and will not win in this term for sure. Not a single node could get enough votes. For example, with 4 nodes one could get 2 votes and other also 2 votes. Nobody will get quorum 3 in this term. The patch makes raft able to notice that situation and speed up the term bump. It is not bumped immediately though, because nodes might do that simultaneously and will get the split vote again after voting for self. There is a random delay. But it is just max 10% of election timeout, so it should speed up split vote resolution on 90% at least. Part of #5285
-
Vladislav Shpilevoy authored
To detect split vote a node needs to see that number of free votes is not enough for anyone to win even if it gets them all. Hence every node needs to count votes for all other nodes. The patch makes raft store votes in a bit more complicated manner than a simple bitmap for just the current instance. Part of #5285
-
Vladislav Shpilevoy authored
ev_timer.at was used as timeout. But after ev_timer_start() it turns into the deadline - totally different value. The patch makes sure ev_timer.at is not used in raft at all. To test that the fakeev subsystem is patched to start its time not from 0. Otherwise ev_timer.at often really matched the timeout even for an active timer.
-
Vladislav Shpilevoy authored
It used to crash if done during election on a node voted for anybody, it is a candidate, it doesn't know a leader yet, but has a WAL write in progress. Thus it could only happen if the term was bumped by a message from a non-leader node and wasn't flushed to the disk yet. The patch makes the reconfig check if there is a WAL write in progress. Then don't do anything. Could also check for volatile vote instead of persistent, but it would create the same problem for the case when started writing vote for self and didn't finish yet. Reconfig would crash.
-
artembo authored
Add GitHub Actions workflows for Fedora 35 Closes: #6692
-
- Jan 20, 2022
-
-
Serge Petrenko authored
The test has a function wait_repl() which tries to wait until the data is replicated to the other node for a tiny timeout (0.2 seconds) This didn't cause issues in the past, but it became an issue after applier in thread introduction. Let's use the default test_run:wait_cond() approach instead of a custom one. This way we increase waiting timeout (making the test pass with a higher chance). Follow-up #6329
-
Serge Petrenko authored
Follow-up #6329
-
Serge Petrenko authored
It's reported that in master-slave setup replicas often have higher CPU usage than their replication masters. Moreover, it's easy for a replica to start falling behind the master. The reason for that is the additional work load on replica's tx thread as compared to master. While master typically splits request processing into 2 threads: iproto, which decodes the requests, and tx, which applies them, replica performs both tasks in the tx thread. This is due to replication architecture: replica handles master connection by a single fiber in the tx thread. The fiber first decodes the incoming requests and then applies them. Teach replicas to decode the incoming replication stream in a separate thread. This way tx thread doesn't waste processing time on row decoding. Each applier thread may serve several appliers, and the total number of applier threads is controlled by a new configuration option - `replication_threads`, with default value `1` (meaning, a single thread is spawned to handle all the appliers. Closes #6329 @TarantoolBot document Title: New configuration option - `replication_threads` It's now possible to specify how many threads will be spawned to decode the incoming replication stream. The default value is '1', meaning a single thread will handle all incoming replication streams. You may set the value to anything from 1 to 1000. When there are multiple replication threads, connections to serve are evenly distributed between the threads.
-
- Jan 19, 2022
-
-
Vladimir Davydov authored
This commit adds SSL context to iostream_ctx and a new kind of error for SSL connections. The SSL context is created if the target URI has transport=ssl parameter. If transport=plain or absent, it will be set to NULL and plain streams will be created as before. For other values of the transport parameter, an error is raised. Note, this is just a stub - an attempt to create an SSL context always currently fails with "SSL not available" error. It is supposed to be implemented in EE build. The new kind of error is currently only used for decoding errors from MsgPack and never raised. It will be raised by the actual implementation of SSL stream.
-
Vladimir Davydov authored
Use table.equals for comparing table-valued configuration parameters instead of ipairs. Needed to trigger reconfiguraiton if the new URI differs from the old one only by parameters, e.g.: box.cfg{listen = {uri = 3301, params = {transport = 'plain'}}} box.cfg{listen = {uri = 3301, params = {transport = 'ssl'}}} No test is added, because currently there's no URI parameters handled by box. The next commit adds the 'transport' parameter and a test that wouldn't pass without this commit.
-
Vladimir Davydov authored
IO stream context (iostream_ctx) stores information needed to construct an iostream object. A context is created from URI and mode, which is either server or client. Currently, both parameters are ignored and a newly created context always constructs plain streams. However, in future we will create SSL streams for certain URIs. That's when we'll need the mode (SSL connection is asymmetrical).
-
Vladimir Davydov authored
In SSL implementation iostream::ctx would point to a SSL object, while a SSL_CTX object would be used for creating streams. Let's rename ctx to data to avoid confusion.
-
Vladimir Davydov authored
The function is used only once, in box_sync_replication. Inlining it makes the code more robust - it allows us to throw from applier_new.
-
Vladimir Davydov authored
Currently, we always create a plain stream, but soon there will be an option to create an encrypted stream so let's pass iostream, which will encapsulate this information, instead of fd to the accept callback. The iostream will be created by evio in accordance with the settings passed in uri parameters so uri won't be needed in the accept callback.
-
Vladimir Davydov authored
It's been many years since it was last used. I'm tired of patching it whenever I need to update evio_service. Besides, it decreases code coverage. So let's finally drop it. After all, it's just a few lines of trivial code that can be rewritten from scratch any time we need.
-
Vladimir Davydov authored
The function creates a plain iostream so let's rename it accordingly.
-
Vladimir Davydov authored
There are two acceptable ways of calling `connect` - as a method of the net.box module or as a function defined in the module: ```lua local net = require('net.box') net.connect(...) net:connect(...) ``` We determine which one is used by checking the type of the first argument: if it's a table, we assume that the function is called as a method. However, the first argument can be a table itself - in case it's a URI given in the extended format. Let's fix this check by comparing the first argument against the net.box module. Let's also add a test checking all possible combinations of `connect` arguments. Follow-up 0badb413 ("net.box: pass uri instead of host and port to state machine").
-
- Jan 18, 2022
-
-
Vladimir Davydov authored
The commit fixes the following test failure: ``` [011] engine/errinj_ddl.test.lua memtx [ fail ] [011] [011] Test failed! Result content mismatch: [011] --- engine/errinj_ddl.result Tue Jan 18 15:28:21 2022 [011] +++ var/rejects/engine/errinj_ddl.reject Tue Jan 18 15:28:26 2022 [011] @@ -343,7 +343,7 @@ [011] s:create_index('sk', {parts = {2, 'unsigned'}}) -- must fail [011] --- [011] - error: Duplicate key exists in unique index "sk" in space "test" with old tuple [011] - - [101, 101, "xxxxxxxxxxxxxxxx"] and new tuple - [100, 101] [011] + - [100, 101] and new tuple - [101, 101, "xxxxxxxxxxxxxxxx"] [011] ... [011] ch:get() [011] --- ``` The test is inherently racy: a conflicting tuple may be inserted to the new index either by the index build procedure or by the test fiber doing DML in the background. The error messages will disagree regarding what tuple should be considered old and which one new. Let's match the error message explicitly. The failure was introduced by d11fb306 ("box: change ER_TUPLE_FOUND message") which enhanced error messages with conflicting tuples.
-
Georgiy Lebedev authored
Testing tuple field count overflow handling requires creating a severely large tuple with box.schema.FIELD_MAX (INT32_MAX) fields: introduce an error injection for testing this corner case. Fixes #6684
-
- Jan 17, 2022
-
-
Kirill Yukhin authored
* Enable parallel invocation of fragile tests
-
- Jan 14, 2022
-
-
Vladimir Davydov authored
The commit fixes the following test failure: ``` [082] vinyl/gh-4810-dump-during-index-build.test.lua Test timeout of 310 secs reached [ fail ] [082] [082] Test failed! Result content mismatch: [082] --- vinyl/gh-4810-dump-during-index-build.result Thu Dec 9 05:31:17 2021 [082] +++ /build/usr/src/debug/tarantool-2.10.0~beta1.324.dev/test/var/rejects/vinyl/gh-4810-dump-during-index-build.reject Thu Dec 9 06:51:03 2021 [082] @@ -117,34 +117,3 @@ [082] for i = 1, ch:size() do [082] ch:get() [082] end; [082] - | --- [082] - | ... [082] - ... ``` The test hangs waiting for the test fibers to exit. There are two test fibers - one builds an index, another populates the test space. The latter uses pcall so it always returns. The one that builds an index, however, doesn't. The problem is index build may fail because it builds a unique index while the fiber populating the space may insert non-unique values. Fix this by building a non-unique index instead, which should never fail. To reproduce the issue the test checks is fixed one can build any index, unique or non-unique, so it should be fine. Closes #5508
-
Vladimir Davydov authored
The commit fixes the following test failure: ``` [005] vinyl/gh.test.lua [ fail ] [005] [005] Test failed! Result content mismatch: [005] --- vinyl/gh.result Mon Dec 13 15:03:45 2021 [005] +++ /root/actions-runner/_work/tarantool/tarantool/test/var/rejects/vinyl/gh.reject Fri Dec 17 10:41:24 2021 [005] @@ -716,7 +716,7 @@ [005] ... [005] test_run:wait_cond(function() return finished == 2 end) [005] --- [005] -- true [005] +- false [005] ... [005] s:drop() [005] --- ``` The reason of the failure is that the fiber doing checkpoints fails, because a checkpoint may be already running by the checkpoint daemon. Invoke box.snapshot() under pcall to make the test more robust. Part of #5141
-
Vladimir Davydov authored
The commit fixes the following test failure: ``` [019] vinyl/deferred_delete.test.lua [ fail ] [019] [019] Test failed! Result content mismatch: [019] --- vinyl/deferred_delete.result Tue Jan 11 11:10:22 2022 [019] +++ /build/usr/src/debug/tarantool-2.10.0~beta2.37.dev/test/var/rejects/vinyl/deferred_delete.reject Fri Jan 14 11:45:26 2022 [019] @@ -964,7 +964,7 @@ [019] ... [019] sk:stat().disk.dump.count -- 1 [019] --- [019] -- 1 [019] +- 0 [019] ... [019] sk:stat().rows - dummy_rows -- 120 old REPLACEs + 120 new REPLACEs + 120 deferred DELETEs [019] --- ``` The test checks that compaction of a primary index triggers dump of secondary indexes of the same space, because it generates deferred DELETE statements. There's no guarantee that by the time compaction completes, secondary index dump have been completed as well, because compaction may ignore the memory quota (it uses vy_quota_force_use in vy_deferred_delete_on_replace). Make the check more robust by using wait_cond. Follow-up #5089
-
Vladimir Davydov authored
It's better than hand-written busy-wait.
-
Vladimir Davydov authored
The commit fixes the following test failure: ``` [013] vinyl/gc.test.lua [ fail ] [013] [013] Test failed! Result content mismatch: [013] --- vinyl/gc.result Fri Dec 24 12:27:33 2021 [013] +++ /build/usr/src/debug/tarantool-2.10.0~beta2.18.dev/test/var/rejects/vinyl/gc.reject Thu Dec 30 10:29:29 2021 [013] @@ -102,7 +102,7 @@ [013] ... [013] check_files_number(2) [013] --- [013] -- true [013] +- null [013] ... [013] -- All records should have been purged from the log by now [013] -- so we should only keep the previous log file. ``` The reason of the failure is that vylog files are deleted asynchronously (`box.snapshot()` doesn't wait for `unlink` to complete) since commit 8e429f4b ("wal: remove old xlog files asynchronously"). So to fix the test, we just need to make the test wait for garbage collection to complete. Follow-up #5383
-
mechanik20051988 authored
When `slab_get` is called from `region_alloc` or `ibuf_alloc` first of all we try to calculate order of appropriate slab. But there is no check that requested size with slab meta is <= UINT32_MAX, that leads to assertion failer in `slab_order` function. There is no need for this assertion we should return `cache->order_max + 1` for this case. Closes #6726
-
- Jan 13, 2022
-
-
Serge Petrenko authored
This required some test refactoring for tests which rely on quorum being "1" by default. Closes #5966 Prerequisite #6471
-