- May 21, 2019
-
-
Vladislav Shpilevoy authored
Users of Lua SWIM module likely will use Lua objects as a payload. Lua objects are serialized into MessagePack automatically, and deserialized back on other instances. But deserialization of 1.2Kb payload on each member:payload() invocation is quite heavy operation. This commit caches decoded payloads to return them again until change. A microbenchmark showed, that cached payload is returned ~100 times faster, than it is decoded each time. Even though a tested payload was quite small and simple: s:set_payload({a = 100, b = 200}) Even this payload is returned 100 times faster, and does not affect GC. Part of #3234
-
Vladislav Shpilevoy authored
Sometimes, especially in tests, it is useful to make something like this: s:add_member({uuid = member:uuid(), uri = member:uri()}) But member:uuid() is cdata struct tt_uuid. This commit allows that. Part of #3234
-
Vladislav Shpilevoy authored
Expose iterators API to be able to iterate over a member table in a 'for' loop like it would just be a Lua table. Part of #3234
-
Vladislav Shpilevoy authored
Expose API to search members by UUID, to read their attributes, to set payload. Part of #3234
-
Vladislav Shpilevoy authored
Expose methods to add, remove, probe members by uri, uuid. Expose broadcast method to probe multiple members by port. Part of #3234
-
Vladislav Shpilevoy authored
SWIM as a library can be useful not only for server internals, but for users as well. This commit exposes Lua bindings to SWIM C API. Here only basic bindings are introduced to create, delete, quit, check a SWIM instance. With sanity tests. Part of #3234
-
Vladislav Shpilevoy authored
Similar methods validate their arguments: add_member, remove_member. Validate here as well for consistency. Part of #3234
-
Vladislav Shpilevoy authored
Firstly, I thought that there is an error - swim_begin_step() does not reschedules round timer, when new_round() fails. But then new_round() appeared never failing. This commit makes it void to eliminate confusion. Probably it is a legacy since the shuffled members array was allocated and freed in new_round(). Part of #3234
-
Vladislav Shpilevoy authored
Appeared, that libev does not allow to change ev_timer values in flight. A timer, reset via ev_timer_set(), should be restarted, because the function changes 'ev_timer.at', which in turn is used internally by timer routines. Part of #3234
-
Vladislav Shpilevoy authored
Lua, which suffers from lack of ability to pass values by pointers into FFI functions, nor has an address operator '&' to take an address of integer or char or anything. Because of that a user need to either use ffi.new(type[1]) or use static buffer, but for such small allocations they are both too expensive and aggravate GC problem. Now buffer module provides preallocated basic types to use in FFI functions. The commit is motivated by one another place where ffi.new('int[1]') appeared, in SWIM module, to obtain payload size as an out parameter of a C function.
-
Vladislav Shpilevoy authored
Static allocator gives memory blocks from cyclic BSS memory block of 3 pages 4096 bytes each. It is much faster than malloc, when a temporary buffer is needed. Moreover, it does not complicate GC job. Despite being faster than malloc, it is still slower, than ffi.new() of size <= 128 known in advance (according to microbenchmarks). ffi.new(size<=128) works either much faster or with the same speed as static_alloc, because internally FFI allocator library caches small blocks and can return them without malloc(). A simple micro benchmark showed, that ffi.new() vs buffer.static_alloc() is ~100 times slower on allocations of > 128 size, on <= 128 when size is not inlined. To better understand what is meant as 'inlined': this ffi.new('char[?]', < <=128 >) works ~100 times faster than this: local size = <= 128 ffi.new('char[?]', size) ffi.new() with inlined size <= 128 works faster than light, and even static allocator can't beat it.
-
Vladimir Davydov authored
Certain kinds of DML requests don't update secondary indexes, e.g. UPDATE that doesn't touch secondary index parts or DELETE for which generation of secondary index statements is deferred. For such a request vy_is_committed(env, space) may return false on recovery even if it has actually been dumped: since such a statement is not dumped for secondary indexes, secondary index's vy_lsm::dump_lsn may be less than such statement's signature, which makes vy_is_committed() assume that the statement hasn't been dumped. Further in the code we have checks that ensure that if we execute a request on recovery, it must not be dumped for the primary index (as the primary index is always dumped after secondary indexes for the sake of recovery), which fires in this case. To fix that, let's refactor the code basing on the following two facts: - Primary index is always updated by a DML request. - Primary index may only be dumped after secondary indexes. Closes #4222
-
Alexander Turenko authored
Yet another fix for building of small library as part of tarantool. Before this commit slab_arena test fails: | [019] Test failed! Result content mismatch: | [019] --- small/slab_arena.result Mon May 20 21:37:46 2019 | [019] +++ small/slab_arena.reject Mon May 20 21:47:01 2019 | [019] @@ -23,3 +23,4 @@ | [019] arena->maxalloc = 2000896 | [019] arena->used = 0 | [019] arena->slab_size = 65536 | [019] +ERROR: Expected dd flag on VMA address 0x7f3ec2080000 See the corresponding commit in the small submdoule for more info.
-
Vladimir Davydov authored
The autoincrement code was written when there were no nested field. Now, it isn't enough to just skip to the autoincrement field - we also need to descend deeper if key_part->path is set. Note, the code expects the nested field to be present and set to NULL. That is, if field path is [1].a.b, the tuple must have all intermediate fields set: {{a = {b = box.NULL}}} (usage of box.NULL is mandatory to create a tuple like that in Lua). Closes #4210
-
Vladimir Davydov authored
Closes #4009 @TarantoolBot document Title: Sequence can now be set for an index part other than the first Initially one could attach a sequence (aka autoincrement) only to the first index part. Now it's possible to attach a sequence to any primary index part. The part still must be integer though. Syntax: ``` box.schema.space.create('test') box.space.test:create_index('primary', { parts = {{1, 'string'}, {2, 'unsigned'}, {3, 'unsigned'}}, sequence = true, sequence_part = 2 }) box.space.test:insert{'a', box.null, 1} -- inserts {'a', 1, 1} ``` Note, `sequence_part` option is 1-base. If `sequence_part` is omitted, 1 is used, which assures backward compatibility with the original behavior. One can also attach a sequence to another index part using `index.alter` (the code below continues the example above): ``` box.space.test.index.primary:alter{sequence_part = 3} box.space.test:insert{'a', 1, box.null, 'x'} -- inserts {'a', 1, 2, 'x'} ```
-
Vladimir Davydov authored
A check was missing in index.alter. This resulted in an attempt to drop the sequence attached to the altered index even if the sequence was not modified. Closes #4214
-
Vladimir Davydov authored
When schema.lua was introduced, there was no such thing as space format and we had to access tuple fields by no. Now we can use human readable names. Let's do it - this should improve code readability. A note about box/alter.test.lua: for some reason it clears format of _space and _index system spaces, which apparently breaks our assumption about field names. Let's zap those pointless test cases.
-
- May 20, 2019
-
-
Alexander Turenko authored
Updated small submodule with the corresponding fix.
-
Vladislav Shpilevoy authored
Background. Coio provides a way to schedule arbitrary tasks execution in worker threads. A task consists of a function to execute, and a custom destructor. To push a task the function coio_task_post(task, timeout) was used. When the function returns 0, a caller can obtain a result and should free the task manually. But the trick is that if timeout was 0, the task was posted in a detached state. A detached task frees its memory automatically despite coio_task_post() result, and does not even yield. Such a task object can't be accessed and so much the more freed manually. coio_getaddrinfo() used coio_task_post() and freed the task when the latter function returned 0. It led to double free when timeout was set 0. The bug was introduced here 800cec73 in an attempt to do not yield in say_logrotate, because it is not fiber-safe. Now there are two functions: coio_task_execute(task, timeout), which never detaches a task completed successfully, and coio_task_post(task), which posts a task in a detached state. Closes #4209
-
Vladislav Shpilevoy authored
According to the standard by Open Group, getaddrinfo() hints argument is optional - it can be NULL. When it is NULL, hints is assumed to have 0 in ai_flags, ai_socktype, and ai_protocol; AF_UNSPEC in ai_family. See The Open Group Base Specifications.
-
Alexander Turenko authored
The result file of the test app-tap/init_script.test.lua was not updated in 549140b3 ('box/memtx: Allow to skip tuple memory from coredump'). Follow up #3509.
-
Alexander V. Tikhonov authored
Made fixes: - Added CMAKE_EXTRA_PARAMS environment to docker's container runs to enable -DENABLE_LTO=ON/OFF cmake option. - Added CC/CXX environment to docker's container runs to set clang for cmake. Also the additional environment variables {CC,CXX}_FOR_BUILD were postponed, because we didn't run cross-compilation at the moment, for more info check: https://docs.travis-ci.com/user/languages/cpp/#choosing-compilers-to-test-against - Changed LTO docker's image to 'debian-buster' due to LTO needed higher versions of packages, check for more information commit: f9e28ce4 ('Add LTO support') - Fixed sources to avoid of failures on builds by GCC with LTO: 1) src/box/memtx_rtree.c: In function ‘mp_decode_rect’: src/box/memtx_rtree.c:86:24: error: ‘c’ may be used uninitialized in this function [-Werror=maybe-uninitialized] rect->coords[i * 2] = c; ^ src/box/memtx_rtree.c:74:10: note: ‘c’ was declared here coord_t c; ^ 2) src/box/sql/func.c: In function ‘quoteFunc’: src/box/sql/func.c:1103:3: error: ‘b’ may be used uninitialized in this function [-Werror=maybe-uninitialized] sql_result_text(context, sql_value_boolean(argv[0]) ? ^ src/box/sql/vdbeapi.c:217:7: note: ‘b’ was declared here bool b; ^ 3) src/box/tuple_update.c: In function ‘update_read_ops’: src/box/tuple_update.c:1022:4: error: ‘field_no’ may be used uninitialized in this function [-Werror=maybe-uninitialized] diag_set(ClientError, ER_NO_SUCH_FIELD_NO, field_no); ^ src/box/tuple_update.c:1014:11: note: ‘field_no’ was declared here int32_t field_no; ^ 4) src/httpc.c: In function ‘httpc_set_verbose’: src/httpc.c:267:2: error: call to ‘_curl_easy_setopt_err_long’ declared with attribute warning: curl_easy_setopt expects a long argument for this option [-Werror] curl_easy_setopt(req->curl_request.easy, CURLOPT_VERBOSE, curl_verbose); ^ 5) src/lua/httpc.c: In function ‘luaT_httpc_request’: src/lua/httpc.c:128:64: error: ‘MEM[(int *)&parser + 20B]’ may be used uninitialized in this function [-Werror=maybe-uninitialized] lua_pushinteger(L, (parser.http_minor > 0) ? parser.http_minor: 0); ^ src/lua/httpc.c:67:21: note: ‘MEM[(int *)&parser + 20B]’ was declared here struct http_parser parser; ^ src/lua/httpc.c:124:64: error: ‘MEM[(int *)&parser + 16B]’ may be used uninitialized in this function [-Werror=maybe-uninitialized] lua_pushinteger(L, (parser.http_major > 0) ? parser.http_major: 0); ^ src/lua/httpc.c:67:21: note: ‘MEM[(int *)&parser + 16B]’ was declared here struct http_parser parser; ^ Close #4215
-
Cyrill Gorcunov authored
In case if there are huge amount of tuples the whole memory goes to coredump file even if we don't need it for problem investigation. In result coredump may blow up to gigabytes in size. Lets allow to exclude this memory from dumping via box.cfg::strip_core boolean parameter. Note that the tuple's arena is used not only for tuples themselves but for memtx->index_extent_pool and memtx->iterator_pool as well, so they are affected too. Fixes #3509 @TarantoolBot document Title: Document box.cfg.strip_core When Tarantool runs under a heavy load the memory allocated for tuples may be very huge in size and to eliminate this memory from being present in `coredump` file the `box.cfg.strip_core` parameter should be set to `true`. The default value is `false`.
-
Vladislav Shpilevoy authored
Negative size led to an assertion. The commit adds a check if size is negative. Closes #4224
-
Alexander Turenko authored
box_process_join() and box_process_subscribe() use coio_write_xrow(), which calls coio_writev_timeout() under hood. If a socket will block at write() the function calls ev_io_start() to wake the fiber up when the socket will be ready to write. This code assumes that the watcher (struct ev_io) is initialized as coio watcher, i.e. coio_create() has been called. The reason why the code works before is that coio_write_xrow() in box_process_{join,subscribe}() writes a small piece of data and so the situation when a socket write buffer has less free space then needed is rare. Fixes #4110.
-
- May 18, 2019
-
-
Georgy Kirichenko authored
-
- May 17, 2019
-
-
Mergen Imeev authored
This patch replaces schema_find_id() with box_space_id_by_name() in SQL. The box_space_id_by_name() is more specialized. In addition, it checks if the user has sufficient rights, unlike schema_find_id(). Closes #3570
-
Mergen Imeev authored
This patch stops the parser if any error occurs. Prior to this patch, it was possible to replace the error with another one, since the parser may continue to work, even if an error occurred. For example: box.execute("insert into not_exist values(1) a") The first error is "Space 'NOT_EXIST' does not exist", but "Syntax error near 'a'" is displayed. After this patch, the first error will be displayed. Closes #3964 Closes #4195
-
Alexander Turenko authored
Fixes #4194.
-
Georgy Kirichenko authored
As we enforced applier row order so we don't need to reacquire schema latch after a ddl statement. Followup for: 056deb2c
-
- May 16, 2019
-
-
Alexander Turenko authored
Support more then 60 parallel jobs (#82, PR #171).
-
- May 15, 2019
-
-
Vladislav Shpilevoy authored
crypto.lua is a public module using OpenSSL directly. But now lib/crypto encapsulates OpenSSL with additional checks and similar but more conforming API. It allows to replace OpenSSL cipher in crypto.lua with lib/crypto methods.
-
Vladislav Shpilevoy authored
OpenSSL API is quite complex and hard to follow, additionally it is very unstable. Encoding/decoding via OpenSSL methods usually consists of multiple calls of a lot of functions. This patch wraps OpenSSL API with one more easy to use and conforming Tarantool code style in scope of crypto library. The traditional OpenSSL API is wrapped as well in a form of crypto_stream object, so OpenSSL API is not cut off. Besides struct crypto_stream the library provides struct crypto_codec which encapsulates all the steps of encryption logic in two short functions: crypto_codec_encrypt/decrypt(iv, in, in_size, out, out_size) A caller can create a needed codec via crypto_codec_new, which now supports all the same algorithms as crypto.lua module. Needed for #3234
-
Vladislav Shpilevoy authored
Tarantool has a strict rule for naming methods of libraries - use the library name as a prefix. For crypto lib methods it should be 'crypto_', not 'tnt_'.
-
Vladislav Shpilevoy authored
Crypto in Tarantool core was implemented and used very poorly uintil now. It was just a one tiny file with one-line wrappers around OpenSSL API. Despite being small and simple, it provided a powerful interface to the Lua land used by Lua 'crypto' public and documented module. Now the time comes when OpenSSL crypto features are wanted on lower level and with richer API, in core library SWIM written in C. This patch moves crypto wrappers into a separate library in src/lib, and drops some methods from the header file because they are never used from C, and are needed for exporting only. Needed for #3234
-
Vladislav Shpilevoy authored
-
Vladislav Shpilevoy authored
msgpack.decode() internally uses 'const char *' variable to decode msgpack, but somewhy expects only 'char *' as input. This commit allows to pass 'const char *' as well.
-
Vladislav Shpilevoy authored
Before the patch msgpack Lua module provided a method encode() able to take a custom buffer to encode into. But it should be of type 'struct ibuf', what made it impossible to use buffer.IBUF_SHARED as a buffer, because its type is 'struct ibuf *'. Strangely, but FFI can't convert these types automatically. This commit allows to use 'struct ibuf *' as well, and moves this functionality into a function in utils.h. Now both msgpack and merger modules can use ibuf directly and by pointer.
-
Vladislav Shpilevoy authored
swim_quit() notifies all the members that this instance has left the cluster. Strangely, except self. It is not a real bug, but showing 'left' status in self struct swim_member would be more correct than 'alive', obviously. It is possible, that self struct swim_member was referenced by a user - this is how 'self' can be available after SWIM instance deletion. Part of #3234
-
Vladislav Shpilevoy authored
SWIM internally tries to avoid unnecessary close+socket+bind calls on reconfiguration if a new URI is the same as an old one. SWIM transport compares <IP, port> couples and if they are equal, does nothing. But if a port is 0, it is not a real port, but a sign to the kernel to find any free port on the IP address. In such a case SWIM transport after bind() retrieves and saves a real port. When the same URI is specified again, the transport compares two addresses: old <IP, auto found port>, new <IP, 0>, sees they are 'different', and rebinds. It is not necessary, obviously, because the new URI covers the old one. This commit avoids rebind, when new IP == old IP, and new port is 0. Part of #3234
-