Skip to content
Snippets Groups Projects
  1. May 21, 2019
    • Vladislav Shpilevoy's avatar
      swim: cache decoded payload in the Lua module · 8ae88a3f
      Vladislav Shpilevoy authored
      Users of Lua SWIM module likely will use Lua objects as a
      payload. Lua objects are serialized into MessagePack
      automatically, and deserialized back on other instances. But
      deserialization of 1.2Kb payload on each member:payload()
      invocation is quite heavy operation. This commit caches decoded
      payloads to return them again until change.
      
      A microbenchmark showed, that cached payload is returned ~100
      times faster, than it is decoded each time. Even though a tested
      payload was quite small and simple:
      
          s:set_payload({a = 100, b = 200})
      
      Even this payload is returned 100 times faster, and does not
      affect GC.
      
      Part of #3234
      8ae88a3f
    • Vladislav Shpilevoy's avatar
      swim: allow to use cdata struct tt_uuid in Lua API · 70e99323
      Vladislav Shpilevoy authored
      Sometimes, especially in tests, it is useful to make something
      like this:
      
          s:add_member({uuid = member:uuid(), uri = member:uri()})
      
      But member:uuid() is cdata struct tt_uuid. This commit allows
      that.
      
      Part of #3234
      70e99323
    • Vladislav Shpilevoy's avatar
      swim: pairs() function to iterate over member table · 248f2425
      Vladislav Shpilevoy authored
      Expose iterators API to be able to iterate over a member table in
      a 'for' loop like it would just be a Lua table.
      
      Part of #3234
      248f2425
    • Vladislav Shpilevoy's avatar
      swim: Lua bindings to access individual members · 4b8c8723
      Vladislav Shpilevoy authored
      Expose API to search members by UUID, to read their attributes,
      to set payload.
      
      Part of #3234
      4b8c8723
    • Vladislav Shpilevoy's avatar
      swim: Lua bindings to manipulate member table · 1203bf90
      Vladislav Shpilevoy authored
      Expose methods to add, remove, probe members by uri, uuid. Expose
      broadcast method to probe multiple members by port.
      
      Part of #3234
      1203bf90
    • Vladislav Shpilevoy's avatar
      swim: introduce Lua interface · 407adf40
      Vladislav Shpilevoy authored
      SWIM as a library can be useful not only for server internals,
      but for users as well. This commit exposes Lua bindings to SWIM
      C API. Here only basic bindings are introduced to create, delete,
      quit, check a SWIM instance. With sanity tests.
      
      Part of #3234
      407adf40
    • Vladislav Shpilevoy's avatar
      swim: validate URI in swim_probe_member() · ae651ab4
      Vladislav Shpilevoy authored
      Similar methods validate their arguments: add_member,
      remove_member. Validate here as well for consistency.
      
      Part of #3234
      ae651ab4
    • Vladislav Shpilevoy's avatar
      swim: make swim_new_round() void · dcf4fbdf
      Vladislav Shpilevoy authored
      Firstly, I thought that there is an error - swim_begin_step()
      does not reschedules round timer, when new_round() fails. But
      then new_round() appeared never failing. This commit makes it
      void to eliminate confusion.
      
      Probably it is a legacy since the shuffled members array was
      allocated and freed in new_round().
      
      Part of #3234
      dcf4fbdf
    • Vladislav Shpilevoy's avatar
      swim: fix an assertion on attempt to chage timeouts · 10ad9706
      Vladislav Shpilevoy authored
      Appeared, that libev does not allow to change ev_timer values in
      flight. A timer, reset via ev_timer_set(), should be restarted,
      because the function changes 'ev_timer.at', which in turn is used
      internally by timer routines.
      
      Part of #3234
      10ad9706
    • Vladislav Shpilevoy's avatar
      buffer: replace all ffi.new(type[1]) with cached union · a2c19aa8
      Vladislav Shpilevoy authored
      Lua, which suffers from lack of ability to pass values by
      pointers into FFI functions, nor has an address operator '&' to
      take an address of integer or char or anything. Because of that
      a user need to either use ffi.new(type[1]) or use static buffer,
      but for such small allocations they are both too expensive and
      aggravate GC problem.
      
      Now buffer module provides preallocated basic types to use in FFI
      functions. The commit is motivated by one another place where
      ffi.new('int[1]') appeared, in SWIM module, to obtain payload
      size as an out parameter of a C function.
      a2c19aa8
    • Vladislav Shpilevoy's avatar
      buffer: port static allocator to Lua · 7fd6c809
      Vladislav Shpilevoy authored
      Static allocator gives memory blocks from cyclic BSS memory
      block of 3 pages 4096 bytes each. It is much faster than
      malloc, when a temporary buffer is needed. Moreover, it does not
      complicate GC job. Despite being faster than malloc, it is still
      slower, than ffi.new() of size <= 128 known in advance (according
      to microbenchmarks).
      
      ffi.new(size<=128) works either much faster or with the same
      speed as static_alloc, because internally FFI allocator library
      caches small blocks and can return them without malloc().
      
      A simple micro benchmark showed, that ffi.new() vs
      buffer.static_alloc() is ~100 times slower on allocations of
      > 128 size,  on <= 128 when size is not inlined.
      
      To better understand what is meant as 'inlined': this
      
          ffi.new('char[?]', < <=128 >)
      
      works ~100 times faster than this:
      
          local size = <= 128
          ffi.new('char[?]', size)
      
      ffi.new() with inlined size <= 128 works faster than light, and
      even static allocator can't beat it.
      7fd6c809
    • Vladimir Davydov's avatar
      vinyl: fix assertion while recovering dumped statement · 9566f14c
      Vladimir Davydov authored
      Certain kinds of DML requests don't update secondary indexes, e.g.
      UPDATE that doesn't touch secondary index parts or DELETE for which
      generation of secondary index statements is deferred. For such a request
      vy_is_committed(env, space) may return false on recovery even if it has
      actually been dumped: since such a statement is not dumped for secondary
      indexes, secondary index's vy_lsm::dump_lsn may be less than such
      statement's signature, which makes vy_is_committed() assume that the
      statement hasn't been dumped. Further in the code we have checks that
      ensure that if we execute a request on recovery, it must not be dumped
      for the primary index (as the primary index is always dumped after
      secondary indexes for the sake of recovery), which fires in this case.
      
      To fix that, let's refactor the code basing on the following two facts:
       - Primary index is always updated by a DML request.
       - Primary index may only be dumped after secondary indexes.
      
      Closes #4222
      9566f14c
    • Alexander Turenko's avatar
      build: fix strip_core with in-source build · 3cf75bc8
      Alexander Turenko authored
      Yet another fix for building of small library as part of tarantool.
      Before this commit slab_arena test fails:
      
       | [019] Test failed! Result content mismatch:
       | [019] --- small/slab_arena.result	Mon May 20 21:37:46 2019
       | [019] +++ small/slab_arena.reject	Mon May 20 21:47:01 2019
       | [019] @@ -23,3 +23,4 @@
       | [019]  arena->maxalloc = 2000896
       | [019]  arena->used = 0
       | [019]  arena->slab_size = 65536
       | [019] +ERROR: Expected dd flag on VMA address 0x7f3ec2080000
      
      See the corresponding commit in the small submdoule for more info.
      Unverified
      3cf75bc8
    • Vladimir Davydov's avatar
      box: fix autoincrement for json path indexes · a6a33e9f
      Vladimir Davydov authored
      The autoincrement code was written when there were no nested field.
      Now, it isn't enough to just skip to the autoincrement field - we also
      need to descend deeper if key_part->path is set.
      
      Note, the code expects the nested field to be present and set to NULL.
      That is, if field path is [1].a.b, the tuple must have all intermediate
      fields set: {{a = {b = box.NULL}}} (usage of box.NULL is mandatory to
      create a tuple like that in Lua).
      
      Closes #4210
      a6a33e9f
    • Vladimir Davydov's avatar
      schema: allow to set sequence for any index part, not just the first · 60111894
      Vladimir Davydov authored
      Closes #4009
      
      @TarantoolBot document
      Title: Sequence can now be set for an index part other than the first
      
      Initially one could attach a sequence (aka autoincrement) only to the
      first index part. Now it's possible to attach a sequence to any primary
      index part. The part still must be integer though.
      
      Syntax:
      
      ```
      box.schema.space.create('test')
      box.space.test:create_index('primary', {
          parts = {{1, 'string'}, {2, 'unsigned'}, {3, 'unsigned'}},
          sequence = true, sequence_part = 2
      })
      box.space.test:insert{'a', box.null, 1} -- inserts {'a', 1, 1}
      ```
      
      Note, `sequence_part` option is 1-base.
      
      If `sequence_part` is omitted, 1 is used, which assures backward
      compatibility with the original behavior.
      
      One can also attach a sequence to another index part using
      `index.alter` (the code below continues the example above):
      
      ```
      box.space.test.index.primary:alter{sequence_part = 3}
      box.space.test:insert{'a', 1, box.null, 'x'} -- inserts {'a', 1, 2, 'x'}
      ```
      60111894
    • Vladimir Davydov's avatar
      schema: fix error while altering index with sequence · 7d778de6
      Vladimir Davydov authored
      A check was missing in index.alter. This resulted in an attempt to drop
      the sequence attached to the altered index even if the sequence was not
      modified.
      
      Closes #4214
      7d778de6
    • Vladimir Davydov's avatar
      schema: use tuple field names in Lua · f47f2004
      Vladimir Davydov authored
      When schema.lua was introduced, there was no such thing as space format
      and we had to access tuple fields by no. Now we can use human readable
      names. Let's do it - this should improve code readability.
      
      A note about box/alter.test.lua: for some reason it clears format of
      _space and _index system spaces, which apparently breaks our assumption
      about field names. Let's zap those pointless test cases.
      f47f2004
  2. May 20, 2019
    • Alexander Turenko's avatar
      build: fix out of source build · 612f05b2
      Alexander Turenko authored
      Updated small submodule with the corresponding fix.
      Unverified
      612f05b2
    • Vladislav Shpilevoy's avatar
      coio: fix getaddrinfo assertion on 0 timeout · b6466ac7
      Vladislav Shpilevoy authored
      Background. Coio provides a way to schedule arbitrary tasks
      execution in worker threads. A task consists of a function to
      execute, and a custom destructor.
      
      To push a task the function coio_task_post(task, timeout) was
      used. When the function returns 0, a caller can obtain a result
      and should free the task manually. But the trick is that if
      timeout was 0, the task was posted in a detached state. A
      detached task frees its memory automatically despite
      coio_task_post() result, and does not even yield. Such a task
      object can't be accessed and so much the more freed manually.
      
      coio_getaddrinfo() used coio_task_post() and freed the task when
      the latter function returned 0. It led to double free when
      timeout was set 0. The bug was introduced here
      800cec73 in an attempt to do not
      yield in say_logrotate, because it is not fiber-safe.
      
      Now there are two functions: coio_task_execute(task, timeout),
      which never detaches a task completed successfully, and
      coio_task_post(task), which posts a task in a detached state.
      
      Closes #4209
      b6466ac7
    • Vladislav Shpilevoy's avatar
      coio: make hints in coio_getaddrinfo optional · 9c4f1c8a
      Vladislav Shpilevoy authored
      According to the standard by Open Group, getaddrinfo() hints
      argument is optional - it can be NULL. When it is NULL, hints
      is assumed to have 0 in ai_flags, ai_socktype, and ai_protocol;
      AF_UNSPEC in ai_family.
      
      See The Open Group Base Specifications.
      9c4f1c8a
    • Alexander Turenko's avatar
      test: fix a test after box.cfg.strip_core addition · 632e3160
      Alexander Turenko authored
      The result file of the test app-tap/init_script.test.lua was not updated
      in 549140b3 ('box/memtx: Allow to skip
      tuple memory from coredump').
      
      Follow up #3509.
      Unverified
      632e3160
    • Alexander V. Tikhonov's avatar
      travis-ci: fix LTO and clang · e55396c8
      Alexander V. Tikhonov authored
      Made fixes:
      
      - Added CMAKE_EXTRA_PARAMS environment to docker's container
        runs to enable -DENABLE_LTO=ON/OFF cmake option.
      
      - Added CC/CXX environment to docker's container runs to set
        clang for cmake. Also the additional environment variables
        {CC,CXX}_FOR_BUILD were postponed, because we didn't
        run cross-compilation at the moment, for more info check:
      
          https://docs.travis-ci.com/user/languages/cpp/#choosing-compilers-to-test-against
      
      - Changed LTO docker's image to 'debian-buster' due to LTO needed
        higher versions of packages, check for more information commit:
      
          f9e28ce4 ('Add LTO support')
      
      - Fixed sources to avoid of failures on builds by GCC with LTO:
      
      1)  src/box/memtx_rtree.c: In function ‘mp_decode_rect’:
          src/box/memtx_rtree.c:86:24: error: ‘c’ may be used uninitialized
            in this function [-Werror=maybe-uninitialized]
              rect->coords[i * 2] = c;
                                  ^
          src/box/memtx_rtree.c:74:10: note: ‘c’ was declared here
            coord_t c;
                    ^
      
      2)  src/box/sql/func.c: In function ‘quoteFunc’:
          src/box/sql/func.c:1103:3: error: ‘b’ may be used uninitialized
            in this function [-Werror=maybe-uninitialized]
             sql_result_text(context, sql_value_boolean(argv[0]) ?
             ^
          src/box/sql/vdbeapi.c:217:7: note: ‘b’ was declared here
            bool b;
                 ^
      
      3)  src/box/tuple_update.c: In function ‘update_read_ops’:
          src/box/tuple_update.c:1022:4: error: ‘field_no’ may be used
            uninitialized in this function [-Werror=maybe-uninitialized]
              diag_set(ClientError, ER_NO_SUCH_FIELD_NO, field_no);
              ^
          src/box/tuple_update.c:1014:11: note: ‘field_no’ was declared here
             int32_t field_no;
                     ^
      
      4)  src/httpc.c: In function ‘httpc_set_verbose’:
          src/httpc.c:267:2: error: call to ‘_curl_easy_setopt_err_long’
            declared with attribute warning: curl_easy_setopt expects a long
            argument for this option [-Werror]
            curl_easy_setopt(req->curl_request.easy, CURLOPT_VERBOSE, curl_verbose);
            ^
      
      5)  src/lua/httpc.c: In function ‘luaT_httpc_request’:
          src/lua/httpc.c:128:64: error: ‘MEM[(int *)&parser + 20B]’ may be used
            uninitialized in this function [-Werror=maybe-uninitialized]
            lua_pushinteger(L, (parser.http_minor > 0) ? parser.http_minor: 0);
                                                                          ^
          src/lua/httpc.c:67:21: note: ‘MEM[(int *)&parser + 20B]’ was declared here
            struct http_parser parser;
                               ^
          src/lua/httpc.c:124:64: error: ‘MEM[(int *)&parser + 16B]’ may be used
            uninitialized in this function [-Werror=maybe-uninitialized]
            lua_pushinteger(L, (parser.http_major > 0) ? parser.http_major: 0);
                                                                          ^
          src/lua/httpc.c:67:21: note: ‘MEM[(int *)&parser + 16B]’ was declared here
            struct http_parser parser;
                               ^
      
      Close #4215
      Unverified
      e55396c8
    • Cyrill Gorcunov's avatar
      box/memtx: Allow to skip tuple memory from coredump · 549140b3
      Cyrill Gorcunov authored
      In case if there are huge amount of tuples the whole
      memory goes to coredump file even if we don't need it
      for problem investigation. In result coredump may
      blow up to gigabytes in size.
      
      Lets allow to exclude this memory from dumping via
      box.cfg::strip_core boolean parameter.
      
      Note that the tuple's arena is used not only for tuples
      themselves but for memtx->index_extent_pool and
      memtx->iterator_pool as well, so they are affected
      too.
      
      Fixes #3509
      
      @TarantoolBot document
      Title: Document box.cfg.strip_core
      
      When Tarantool runs under a heavy load the memory allocated
      for tuples may be very huge in size and to eliminate this
      memory from being present in `coredump` file the `box.cfg.strip_core`
      parameter should be set to `true`.
      
      The default value is `false`.
      549140b3
    • Vladislav Shpilevoy's avatar
      msgpack: validate msgpack.decode() cdata size argument · 10873f16
      Vladislav Shpilevoy authored
      Negative size led to an assertion. The commit adds a check if
      size is negative.
      
      Closes #4224
      10873f16
    • Alexander Turenko's avatar
      iproto: init coio watcher before join/subscribe · 539aee3d
      Alexander Turenko authored
      box_process_join() and box_process_subscribe() use coio_write_xrow(),
      which calls coio_writev_timeout() under hood. If a socket will block at
      write() the function calls ev_io_start() to wake the fiber up when the
      socket will be ready to write. This code assumes that the watcher
      (struct ev_io) is initialized as coio watcher, i.e. coio_create() has
      been called.
      
      The reason why the code works before is that coio_write_xrow() in
      box_process_{join,subscribe}() writes a small piece of data and so the
      situation when a socket write buffer has less free space then needed is
      rare.
      
      Fixes #4110.
      539aee3d
  3. May 18, 2019
  4. May 17, 2019
    • Mergen Imeev's avatar
      sql: replace schema_find_id() by box_space_id_by_name() · afc5f92d
      Mergen Imeev authored
      This patch replaces schema_find_id() with box_space_id_by_name()
      in SQL. The box_space_id_by_name() is more specialized. In
      addition, it checks if the user has sufficient rights, unlike
      schema_find_id().
      
      Closes #3570
      afc5f92d
    • Mergen Imeev's avatar
      sql: stop parser on error · d039fc05
      Mergen Imeev authored
      This patch stops the parser if any error occurs. Prior to this
      patch, it was possible to replace the error with another one,
      since the parser may continue to work, even if an error occurred.
      
      For example:
      box.execute("insert into not_exist values(1) a")
      
      The first error is "Space 'NOT_EXIST' does not exist", but "Syntax
      error near 'a'" is displayed.
      
      After this patch, the first error will be displayed.
      
      Closes #3964
      Closes #4195
      d039fc05
    • Alexander Turenko's avatar
      travis-ci: add fedora 30 · e6869dd2
      Alexander Turenko authored
      Fixes #4194.
      e6869dd2
    • Georgy Kirichenko's avatar
      Remove schema latch relock · c783c8b7
      Georgy Kirichenko authored
      As we enforced applier row order so we don't need to reacquire schema
      latch after a ddl statement.
      
      Followup for: 056deb2c
      c783c8b7
  5. May 16, 2019
  6. May 15, 2019
    • Vladislav Shpilevoy's avatar
      crypto: use crypto library in crypto.lua · 544d648c
      Vladislav Shpilevoy authored
      crypto.lua is a public module using OpenSSL directly. But now
      lib/crypto encapsulates OpenSSL with additional checks and
      similar but more conforming API. It allows to replace OpenSSL
      cipher in crypto.lua with lib/crypto methods.
      544d648c
    • Vladislav Shpilevoy's avatar
      crypto: implement crypto libary · 4467fdac
      Vladislav Shpilevoy authored
      OpenSSL API is quite complex and hard to follow, additionally it
      is very unstable. Encoding/decoding via OpenSSL methods usually
      consists of multiple calls of a lot of functions. This patch
      wraps OpenSSL API with one more easy to use and conforming
      Tarantool code style in scope of crypto library.
      
      The traditional OpenSSL API is wrapped as well in a form of
      crypto_stream object, so OpenSSL API is not cut off.
      
      Besides struct crypto_stream the library provides struct
      crypto_codec which encapsulates all the steps of encryption logic
      in two short functions:
      
          crypto_codec_encrypt/decrypt(iv, in, in_size, out, out_size)
      
      A caller can create a needed codec via crypto_codec_new, which
      now supports all the same algorithms as crypto.lua module.
      
      Needed for #3234
      4467fdac
    • Vladislav Shpilevoy's avatar
      crypto: make exported methods conform code style · 0c718345
      Vladislav Shpilevoy authored
      Tarantool has a strict rule for naming methods of libraries - use
      the library name as a prefix. For crypto lib methods it should be
      'crypto_', not 'tnt_'.
      0c718345
    • Vladislav Shpilevoy's avatar
      crypto: move crypto business into a separate library · 1b9d3c6b
      Vladislav Shpilevoy authored
      Crypto in Tarantool core was implemented and used very poorly
      uintil now. It was just a one tiny file with one-line wrappers
      around OpenSSL API. Despite being small and simple, it provided a
      powerful interface to the Lua land used by Lua 'crypto' public
      and documented module.
      
      Now the time comes when OpenSSL crypto features are wanted on
      lower level and with richer API, in core library SWIM written
      in C. This patch moves crypto wrappers into a separate library
      in src/lib, and drops some methods from the header file because
      they are never used from C, and are needed for exporting only.
      
      Needed for #3234
      1b9d3c6b
    • Vladislav Shpilevoy's avatar
      Drop an unused function and class · 0891b159
      Vladislav Shpilevoy authored
      0891b159
    • Vladislav Shpilevoy's avatar
      msgpack: allow to pass 'const char *' into decode() · 453fff2b
      Vladislav Shpilevoy authored
      msgpack.decode() internally uses 'const char *' variable to
      decode msgpack, but somewhy expects only 'char *' as input.
      This commit allows to pass 'const char *' as well.
      453fff2b
    • Vladislav Shpilevoy's avatar
      msgpack: allow to pass 'struct ibuf *' into encode() · 7d98a3cf
      Vladislav Shpilevoy authored
      Before the patch msgpack Lua module provided a method encode()
      able to take a custom buffer to encode into. But it should be of
      type 'struct ibuf', what made it impossible to use
      buffer.IBUF_SHARED as a buffer, because its type is
      'struct ibuf *'. Strangely, but FFI can't convert these types
      automatically.
      
      This commit allows to use 'struct ibuf *' as well, and moves this
      functionality into a function in utils.h. Now both msgpack and
      merger modules can use ibuf directly and by pointer.
      7d98a3cf
    • Vladislav Shpilevoy's avatar
      swim: set 'left' status in self on swim_quit() · 8cb0206b
      Vladislav Shpilevoy authored
      swim_quit() notifies all the members that this instance has left
      the cluster. Strangely, except self. It is not a real bug, but
      showing 'left' status in self struct swim_member would be more
      correct than 'alive', obviously.
      
      It is possible, that self struct swim_member was referenced by a
      user - this is how 'self' can be available after SWIM instance
      deletion.
      
      Part of #3234
      8cb0206b
    • Vladislav Shpilevoy's avatar
      swim: do not rebind when new 'port' is 0 · 2149496b
      Vladislav Shpilevoy authored
      SWIM internally tries to avoid unnecessary close+socket+bind
      calls on reconfiguration if a new URI is the same as an old one.
      SWIM transport compares <IP, port> couples and if they are
      equal, does nothing.
      
      But if a port is 0, it is not a real port, but a sign to the
      kernel to find any free port on the IP address. In such a case
      SWIM transport after bind() retrieves and saves a real port.
      
      When the same URI is specified again, the transport compares two
      addresses: old <IP, auto found port>, new <IP, 0>, sees they are
      'different', and rebinds. It is not necessary, obviously, because
      the new URI covers the old one.
      
      This commit avoids rebind, when new IP == old IP, and new port is
      0.
      
      Part of #3234
      2149496b
Loading