- Sep 15, 2022
-
-
Georgiy Lebedev authored
`directly_replaced` stories can potentially get garbage collected in `memtx_tx_handle_gap_write`, which is unexpected and leads to 'use after free': in order to fix this, limit garbage collection points only to external API calls. Wrap all possible garbage collection points with explicit warnings (see c9981a56). Closes #7449 NO_DOC=bugfix
-
- Sep 14, 2022
-
-
Sergey Bronnikov authored
NO_CHANGELOG=internal NO_DOC=internal NO_TEST=internal
-
Sergey Bronnikov authored
Commit 9adedc1f ("test: add new `make` test targets") introduced new targets for running test-run.py with unit tests. However, this target doesn't depend on changes in tested libraries and changes in unit tests. Proposed patch introduces a function that creates a build targets for unit tests and fixes the described problem with dependencies. Additionally, patch added missed dependence for popen and popen-child. NO_CHANGELOG=internal NO_DOC=internal
-
Sergey Bronnikov authored
NO_CHANGELOG=internal NO_DOC=internal NO_TEST=refactoring Needed for the next commit
-
Alexander Turenko authored
All merge sources (including the merger itself) share the same `<merge source>:pairs()` implementation, which returns `gen, param, state` triplet. `gen` is `lbox_merge_source_gen()`, `param` is `nil`, `state` in the merge source. The `lbox_merge_source_gen()` returns `source, tuple`. The returned source is supposed to be the same object as a one passed to the function (`gen(param, state)`), so the function assumes the object as alive and don't increment source's refcounter at entering, don't decrease it at exitting. This logic is perfect, but there was a mistake in the implementation: the function returns a new cdata object (which holds the same pointer to the merge source structure) instead of the same cdata object. The new cdata object neither increases the source's refcounter at pushing to Lua, nor decreases it at collecting. At result, if we'll loss the original merge source object (and the first `state` that is returned from `:pairs()`), the source structure may be freed. The pointer in the new cdata object will be invalid so. A sketchy code that illustrates the problem: ```lua gen, param, state0 = source:pairs() assert(state0 == source) source = nil state1, tuple = gen(param, state0) state0 = nil -- assert(state1 == source) -- would fails collectgarbage() -- The cdata object that is referenced as `source` and as `state` -- is collected. The GC handler is called and dropped the merge -- source structure refcounter to zero. The structure is freed. -- The call below will crash. gen(param, state1) ``` In the fixed code `state1 == source`, so the GC handler is not called prematurely: we have the merge source object alive till the end of the iterator or till the stop of the traversal. Fixes #7657 NO_DOC=a crash is definitely not what we want to document
-
Timur Safin authored
NO_DOC=internal NO_CHANGELOG=internal
-
Timur Safin authored
NO_DOC=internal NO_CHANGELOG=internal
-
Timur Safin authored
NO_DOC=internal NO_CHANGELOG=internal
-
Timur Safin authored
NO_DOC=internal NO_CHANGELOG=internal
-
Timur Safin authored
NO_DOC=internal NO_CHANGELOG=internal
-
Timur Safin authored
NO_DOC=internal NO_CHANGELOG=internal
-
Timur Safin authored
NO_DOC=internal NO_CHANGELOG=internal
-
Timur Safin authored
NO_DOC=internal NO_CHANGELOG=internal
-
Timur Safin authored
NO_DOC=internal NO_CHANGELOG=internal
-
Timur Safin authored
NO_DOC=internal NO_CHANGELOG=internal
-
Timur Safin authored
NO_DOC=internal NO_CHANGELOG=internal
-
Timur Safin authored
NO_DOC=internal NO_CHANGELOG=internal
-
Timur Safin authored
NO_DOC=internal NO_CHANGELOG=internal
-
Timur Safin authored
NO_DOC=internal NO_CHANGELOG=internal
-
Timur Safin authored
NO_DOC=internal NO_CHANGELOG=internal
-
Timur Safin authored
NO_DOC=internal NO_CHANGELOG=internal
-
Igor Munkin authored
* From Lua 5.3: assert() accepts any type of error object. Closes #7457 Part of #7230 NO_DOC=LuaJIT submodule bump NO_TEST=LuaJIT submodule bump
-
- Sep 13, 2022
-
-
Georgiy Lebedev authored
When conflicting transactions that made full scans in `memtx_tx_handle_gap_write`, we need to also track that the conflicted transaction has read the inserted tuple, just like we do in gap tracking for ordered indexes — otherwise another transaction can overwrite the inserted tuple in which case no gap tracking will be handled. Closes #7493 NO_DOC=bugfix
-
Georgy Moshkin authored
Before this change there was no way to create a fiber that accepts parameters without yielding from the current fiber using the c api. You could pass the function arguments when calling fiber_start, but that forces you to yield, which is not acceptable in some scenarios (e.g. within a transaction). This commit introduces 2 new functions to the api: fiber_set_ctx for setting an pointer to a context of the given fiber and fiber_get_ctx for accessing that context. Closes https://github.com/tarantool/tarantool/issues/7669 @TarantoolBot document Title: fiber: add fiber_set_ctx & fiber_get_ctx functions Add 2 api functions: `fiber_set_ctx` & `fiber_get_ctx` which can be used for passing data to a fiber. Previously this could be done via the `fiber_start` function, except that this would force the current fiber to yield, which is not acceptable in some scenarios (e.g. during a transaction). Now you can create a fiber with `fiber_new`, set it's contents with `fiber_set_ctx`, make it ready for execution with `fiber_wakeup` and keep executing the current fiber.
-
Yaroslav Lobankov authored
- Remove unused imports - Remove unnecessary creation of 'replica' instance objects - Use `<instance>.iproto.uri` object attribute instead of calling `box.cfg.listen` via admin connection NO_DOC=testing stuff NO_TEST=testing stuff NO_CHANGELOG=testing stuff
-
Yaroslav Lobankov authored
Bump test-run to new version with the following improvements: - Report job summary on GitHub Actions [1] - Free port auto resolving for TarantoolServer and AppServer [2] Also, this patch includes the following changes: - removing `use_unix_sockets` option from all suite.ini config files due to permanent using Unix sockets for admin connection recently introduced in test-run - switching replication-py tests to Unix sockets for iproto connection - fixing replication-py/swap.test.py and swim/swim.test.lua tests [1] tarantool/test-run#341 [2] tarantool/test-run#348 NO_DOC=testing stuff NO_TEST=testing stuff NO_CHANGELOG=testing stuff
-
- Sep 12, 2022
-
-
Vladimir Davydov authored
- Filter out temporary spaces on read view creation unless the read_view_opts::needs_temporary_spaces flag is set and drop temporary space filter from checkpoint and join code. - Pass read_view_opts to engine_create_read_view. In case of memtx, delay garbage collection of temporary tuples if the flag is set. Needed for https://github.com/tarantool/tarantool-ee/issues/213 NO_DOC=internal NO_TEST=ee NO_CHANGELOG=internal
-
Sergey Bronnikov authored
@TarantoolBot document Title: Document a body decoding in http response New method "response:decode()" has been introduced for a HTTP response. It allows to decode body to a Lua object. Decoding depends on content type passed in HTTP header and decoding function for it defined in table `http.decoders` or `http.new().decoders`. Function `response:decode()` will raise an error when content type is present in the response, but there is no appropriate decoder. By default decoders for the following content types are defined: - to JSON with content-type "application/json" - to MsgPack with content-type "application/msgpack" - to YAML with content-type "application/yaml" If a content type is not present in the response, :decode() assumes "application/json". ``` tarantool> resp = require('http.client').put( 'http://127.0.0.1:8080', '{"productId": 123456, "quantity": 100}') --- ... tarantool> resp.body --- - '{"productId": 123456, "quantity": 100}' ... tarantool> tarantool> resp:decode() --- - productId: 123456 - quantity: 100 ... tarantool> ``` For content types missed in `http.decoders` user could define it's own by defining a new record with a key equal to desired MIME type in lowercase and it's decoding function that must accept HTTP body, content type and must return a decoded value: ``` local http = require("http") local xml = require("luarapidxml") http.decoders = { ['application/xml'] = function(body, _content_type) return xml.decode(body) end, } ``` Closes #6833
-
Sergey Bronnikov authored
client_object:request() could pass a body as a string value to HTTP request when body is used. User must encode body to a string value as a preparation step. However, we can do encoding for user automatically when it possible. @TarantoolBot document Title: Document body encoding in http request Now user could pass a "body" option as a value with standard Lua types (table, nil, string, userdata, cdata, boolean or number) as well as Tarantool's own data types like decimal, uuid, datetime. In such case that value will be encoded to a string automatically where it possible. Encoding depends on body's data type: `nil` converted to an empty string, types (`string`, `number`, `boolean` and `nil`) will be converted to a string using standard Lua function `tostring()`, types cdata, userdata and tables will be encoded by functions defined in `http.encoders` or `http.new().encoders` table. In latter case encoding function depends on a content type passed to HTTP request using HTTP header. Default content type "application/json" and appropriate encoder are used when content type is missed in HTTP request. By default encoders for the following content types are defined: - "application/json" - "application/msgpack" - "application/yaml" For content types missed in `http.encoders` user could define it's own encoder by defining a new record with a key equal to desired MIME type in lowercase and it's encoding function that must accept HTTP body and content type and must return a string with serialized data: ``` local http = require("http") local xml = require("luarapidxml") http.encoders = { ['application/xml'] = function(body, _content_type) return xml.encode(body) end, } ``` Be careful with defining your own encoding function - header with content type could contain content type like "text/html" as well as a header options [1], like "text/html; charset=UTF-8". Content type is passed to encoder function *with* options. JSON encoder will use default configuration settings defined in JSON module (see `json.cfg()`). Header with content type will be set to "application/json" if it was not defined by user in request's options. ``` local http = require("http") local client = http.new() local body = { a = 1, b = 2, c = 3, } client:request("POST", uri, body) client:post(uri, body) ``` 1. https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Type Part of #6833
-
Vladimir Davydov authored
strerror() is MT-Unsafe, because it uses a static buffer under the hood. We should use strerror_r() instead, which takes a user-provided buffer. The problem is there are two implementations of strerror_r(): XSI and GNU. The first one returns an error code and always writes the message to the beginning of the buffer while the second one returns a pointer to a location within the buffer where the message starts. Let's introduce a macro HAVE_STRERROR_R_GNU set if the GNU version is available and define tt_strerror() which writes the message to the static buffer, like tt_cstr() or tt_sprintf(). Note, we have to export tt_strerror(), because it is used by Lua via FFI. We also need to make it available in the module API header, because the say_syserror() macro uses strerror() directly. In order to avoid adding tt_strerror() to the module API, we introduce an internal helper function _say_strerror(), which calls tt_strerror(). NO_DOC=bug fix NO_TEST=code is covered by existing tests
-
- Sep 09, 2022
-
-
Alexander Turenko authored
In brief: `vfork()` on Mac OS 12 and newer doesn't suspend the parent process, so we should wait for `setpgrp()` to use `killpg()`. See more detailed description of the problem in a comment of the `popen_wait_group_leadership()` function. The solution is to spin in a loop and check child's process group. It looks as the most simple and direct solution. Other possible solutions requires to estimate cons and pros of using extra file descriptor or assigning a signal number for the child -> parent communication. There are the following alternatives and variations: * Create a pipe and notify the parent from the child about the `setpgrp()` call. It costs extra file descriptor, so I decided to don't do that. However if we'll need some channel to deliver information from the child to the parent for another task, it'll worth to reimplement this function too. One possible place, where we may need such channel is delivery of child's errors to the parent. Now the child writes them directly to logger's fd and it requires some tricky code to keep and close the descriptor at right points. Also it doesn't allow to catch those errors in the parent, but we may need it for #4925. * Notify the parent about `setpgrp()` using a signal. It seems too greedly to assign a specific signal for such local problem. It is also unclear how to guarantee that it'll not break any user's code: a user can load a dynamic library, which uses some signals on its own. However we can consider using this approach here if we'll design some common interprocess notification system. * We can use the fiber cond or the `popen_wait_timeout()` function from PR #7648 to react to the child termination instantly. It would complicate the code and anyway wouldn't allow to react instantly on `setpgrp()` in the child. Also it assumes yielding during the wait (see below). * Wait until `setpgrp()` in `popen_send_signal()` instead of `popen_new()`. It would add yielding/waiting inside `popen_send_signal()` and likely will extend a set of its possible exit situations. It is undesirable: this function should have simple and predictable behavior. * Finally, we considered yielding in `popen_wait_group_leadership()` instead of sleeping the whole tx thread. `<popen handle>:new()` doesn't yield at the moment and a user's code may lean on this fact. Yielding would allow to achieve better throughtput (amount of parallel requests per second), but we don't take much care to performance on Mac OS. The primary goal for this platform is to offer the same behavior as on Linux to allow development of applications. I didn't replace `vfork()` with `fork()` on Mac OS, because `vfork()` works and I don't know consequences of calling `pthread_atfork()` handlers in a child created by popen. See the comment in `popen_new()` near to `vfork()` call: it warns about possible mutex double locks. This topic will be investigated further in #6674. Fixes #7658 NO_DOC=fixes incorrect behavior, no need to document the bug NO_TEST=already tested by app-tap/popen.test.lua
-
- Sep 07, 2022
-
-
Vladislav Shpilevoy authored
If a node persisted a foreign term + vote request at the same time, it increased split-brain probability. A node could vote for a candidate having smaller vclock than the local one. For example, via the following scenario: - Node1, node2, node3 are started; - Node1 becomes a leader; - The topology becomes node1 <-> node2 <-> node3 due to network issues; - Node1 sends a synchro txn to node2. The txn starts a WAL write; - Node3 bumps term and votes for self. Sends it all to node2; - Node2 votes for node3, because their vclocks are equal; - Node2 finishes all pending WAL writes, including the txn from node1. Now its vclock is > node3's one and the vote was wrong. - Node3 wins, writes PROMOTE, and it conflicts with node1 writing CONFIRM. This patch makes so a node can't persist a vote in a new term in the same WAL write as the term bump. Term bump is written first and alone. It serves as a WAL sync after which the node's vclock is not supposed to change except for the 0 (local) component. The vote requests are re-checked after term bump is persisted to see if they still can be applied. Part of #7253 NO_DOC=bugfix
-
Vladislav Shpilevoy authored
If the limbo was fenced during CONFIRM WAL write, then the confirmed txn was committed just fine, but its author-fiber kept hanging. This is because when it was woken up, it checked if the limbo is frozen and went to infinite waiting before actually checking if the txn is completed. The fiber would unfreeze if would be woken up explicitly as a workaround. The fix is simple - change the checks order. Part of #7253 NO_DOC=bugfix
-
Vladislav Shpilevoy authored
box.ctl.promote() bumps the term, makes the node a candidate, and waits for the term outcome. The waiting used to be until there is a leader elected or the node lost connection quorum or the term was bumped again. There was a bug that a node could hang in box.ctl.promote() even when became a voter. It could happen if the quorum was still there and a leader couldn't be elected in the current term at all. For instance, others could have `election_mode='off'`. The fix is to stop waiting for the term outcome if the node can't win anyway. NO_DOC=bugfix
-
Vladislav Shpilevoy authored
If box.ctl.promote() was called on more than one instance, then it could lead to infinite or extremely long elections bumping thousands of terms in just a few seconds. This was because box.ctl.promote() used to be a loop. The loop retried term bump + voted for self until the node won. Retry happened immediately as the node saw the term was bumped again and there was no leader elected or the connection quorum was lost. If 2 nodes would start box.ctl.promote() almost at the same time, they could bump each other's terms, not see any winner, bump them again, and so on. For example: - Node1 term=1, node2 term=2; - Promote is called on both; - Node1 term=2, node2 term=3. They receive the messages. Node2 ignores node1's old term. Node1 term is bumped and it votes for node2, but it didn't win, so box.ctl.promote() bumps its term to 4. - Node2 receives term 4 from node1. Its own box.ctl.promote() sees the term was bumped and no winner, so it bumps it to 5 and the process continues for a long time. It worked good enough in tests - the problem happened sometimes, terms could roll like 80k times in a few seconds, but the tests ended fine anyway. One of the next commits will make term bump + vote written in separate WAL records. That aggravates the problem drastically. Basically, this mutual term bump loop could end only if one node would receive vote for self from another node and send back the message 'I am a leader' before the other node's box.ctl.promote() notices the term was bumped externally. This will get much harder to achieve. The patch simply drops the loop. Let box.ctl.promote() fail if the term was bumped outside. There was an alternative to keep running it in a loop with a randomized election timeout like it works inside of raft. But the current solution is just simpler. NO_DOC=bugfix NO_TEST=election_split_vote_test.lua catches it already
-
Vladimir Davydov authored
There may be a space upgrade in progress at the time when a read view is created. In this case, we should apply the upgrade function to tuples retrieved from the space read view. We can't just use the space upgrade function as is, because it may be dropped while the read view is still in use. So we need to create a special upgrade function for the read view. This commit adds stubs for this, which will be implemented in the EE repository. Needed for https://github.com/tarantool/tarantool-ee/issues/163 NO_DOC=internal NO_TEST=ee NO_CHANGELOG=internal
-
Vladimir Davydov authored
Although currenlty feasible, using a read view from multiple threads simultaneously is dangerous on its own because of inevitably arising object life time issues. With introduction of space upgrade handling, it'll become hardly possible, because we'll have to attach Lua function to a read view, which can only be used from one thread. Let's add some assertions to guarantee that a read view is used only from one thread. To do that, we need to introduce the concept of an active read view: from now on, a read view must be activated before it can be used and deactivated before it can be closed; both operations must be done from the same thread - the thread that accesses the read view. https://github.com/tarantool/tarantool-ee/issues/163 NO_DOC=debug NO_TEST=debug NO_CHANGELOG=debug
-
Ilya Verbin authored
Currently this function is not used inside Tarantool, however it is available via C module API. Deprecate it, because it is very confusing and has nothing to do with the fiber cancellation. Closes #7166 @TarantoolBot document Title: fiber: get rid of fiber_set_cancellable Product: Tarantool Since: 2.11 Audience/target: dev Root document: https://www.tarantool.io/en/doc/latest/dev_guide/reference_capi/fiber/#c.fiber_set_cancellable SME: @Gumix This function is a no-op since 2.11 and should be dropped from the documentation.
-
Ilya Verbin authored
It will yield until the is_shutting_down flag is set by tarantool_exit(). This allows to get rid of the FIBER_IS_CANCELLABLE flag, which is no longer used anywhere in Tarantool. Part of #7166 NO_DOC=internal NO_CHANGELOG=internal
-
- Sep 06, 2022
-
-
Sergey Vorontsov authored
Add the redos_7.3.yml workflow to build Tarantool packages (x86_64) for the RedOS 7.3 system. Packages are created by https://github.com/packpack/packpack. NO_DOC=ci NO_TEST=ci
-