Skip to content
Snippets Groups Projects
  1. Apr 09, 2019
    • Vladislav Shpilevoy's avatar
      swim: introduce dissemination component · ecef10c3
      Vladislav Shpilevoy authored
      Dissemination components broadcasts events about member status
      updates. When any member attribute is updated (incarnation,
      status, UUID, address), the member stands into an event queue.
      Members from the queue are encoded into each round step message
      with a higher priority and before anti-entropy section.
      
      It means, then even if a cluster consists of hundreds of members
      and one of them was updated on one of instances, this update will
      be disseminated regardless of whether this memeber is encoded
      into anti-entropy section or not. It drastically speeds events
      dissemination up, according to the SWIM paper, and is noticed in
      the tests.
      
      Part of #3234
      ecef10c3
    • Vladislav Shpilevoy's avatar
      test: set packet drop rate instead of flag in swim tests · 86af0bd3
      Vladislav Shpilevoy authored
      Before dissemination component it was enough in the tests to
      either drop all packets to/from a certain member, or do not drop
      at all. But after dissemination it will be time to test more
      granulated packet loss table: not 0/100, but 5/10/20/50/.../100
      packet loss rate.
      
      Part of #3234
      86af0bd3
    • Vladislav Shpilevoy's avatar
      test: speed up swim big cluster failure detection · f30d9ed2
      Vladislav Shpilevoy authored
      The test checks that if a member has failed in a big cluster, it
      is eventually deleted from all instances. But it takes too much
      real time despite usage of virtual time.
      
      This is because member total deletion takes
      O(N + ack_timeout * 5) time. N so as to wait until every member
      pinged the failed one at least once, + 3 * ack_timeout to learn
      that it is dead, and + 2 * ack_timeout to drop it. Of course, it
      is an upper border, and usually it is faster but not much. For
      example, on the cluster of size 50 it takes easily 55 virtual
      seconds.
      
      On the contrary, to just learn that a member is dead on every
      instance takes O(log(N)) according to the SWIM paper. On the
      same test with 50 instances cluster it takes ~15 virtual seconds
      to disseminate 'dead' status of the failed member on every
      instance. And even without dissemination component, with
      anti-entropy only.
      
      Leaping ahead, for the subsequent patches it is tested that with
      the dissemination component it takes already ~6 virtual seconds.
      
      In the summary, without losing test coverage it is much faster to
      turn off SWIM GC and wait until the failed member looks dead on
      all instances.
      
      Part of #3234
      f30d9ed2
    • Vladislav Shpilevoy's avatar
      swim: make members array decoder be a separate function · 190e201a
      Vladislav Shpilevoy authored
      At this moment SWIM protocol stores array of members only in one
      place: inside the anti-entropy component. Its decoding is a
      simple loop taking the member definitions one by one and
      upserting them into the member table.
      
      But the dissemination also has something kinda like members
      array: an array of events. The trick is that an event is
      basically the same as a member +/- a couple of optional fields.
      Events are also decoded into the member definition structure. It
      means that anti-entropy decoder can be easily reused.
      
      Part of #3234
      190e201a
    • Vladislav Shpilevoy's avatar
      swim: encapsulate member bin info into a 'passport' · afbc8504
      Vladislav Shpilevoy authored
      Each member stored in components dissemination and anti-entropy
      should carry a unique identifier, a status, and an address. Those
      attributes are UUID, IP, Port, enum swim_member_status,
      incarnation.
      
      Now they are sent only in scope of anti-entropy, but forthcoming
      dissemination component also would like to use these attributes
      for each event.
      
      This commit makes the vital attributes and their code more
      reusable by encapsulation of them into a binary passport
      structure.
      
      Part of #3234
      afbc8504
  2. Apr 08, 2019
    • Konstantin Osipov's avatar
      vinyl: rename vy_set and vy_set_with_colmask · 7b56f1fe
      Konstantin Osipov authored
      Rename vy_set() and vy_set_with_colmask() to vy_tx_set() and
      vy_tx_set_with_colmask()
      
      These methods really belong to vy_tx module, so move them there.
      7b56f1fe
    • Serge Petrenko's avatar
      lua: add type of operation to space trigger parameters · 5ab0763b
      Serge Petrenko authored
      Add the type of operation which is being executed to before_replace and
      on_replace triggers.
      
      Closes #4099
      
      @TarantoolBot document
      Title: new parameter for space before_replace and on_replace triggers
      Now before_replace and on_replace triggers accept an additional
      parameter: the type of operation that is being executed.
      (INSERT/REPLACE/DELETE/UPDATE/UPSERT)
      For example, a trigger function may now look like this:
      ```
      function before_replace_trig(old, new, space_name, op_type)
          if op_type == 'INSERT' then
      	return old
          else
      	return new
          end
      end
      ```
      And will restrict all INSERTs, but allow REPLACEs, UPSERTs, DELETEs and
      UPDATEs.
      5ab0763b
    • Roman Tokarev's avatar
      Add idle to downstream status in box.info · a4a7744c
      Roman Tokarev authored
      a4a7744c
  3. Apr 07, 2019
    • Alexander Turenko's avatar
      test: update test-run · 879ec075
      Alexander Turenko authored
      Add more logging into wait_fullmesh() and return immediately with false
      when 'stopped' status is observed.
      
      The purpose of the change is to provide more information in case of a
      master-master replication bootstrap failure.
      879ec075
    • Vladimir Davydov's avatar
      vinyl: incorporate tuple comparison hints into vinyl data structures · dafd3926
      Vladimir Davydov authored
      Apart from speeding up statement comparisons and hence index lookups,
      this is also a prerequisite for multikey indexes, which will reuse tuple
      comparison hints as offsets in indexed arrays.
      
      Albeit huge, this patch is pretty straightforward - all it does is
      replace struct tuple with struct vy_entry (which is tuple + hint pair)
      practically everywhere in the code. Now statements are stored and
      compared without hints only in a few places, primarily at the very top
      level. Hints are also computed at the top level so it should be pretty
      easy to replace them with multikey offsets when the time comes.
      dafd3926
    • Vladimir Davydov's avatar
      vinyl: prepare for storing hints in vinyl data structures · a075fb97
      Vladimir Davydov authored
      This patch adds a helper struct vy_entry, which unites a statement with
      a hint. We will use this struct to store hinted statements in vinyl data
      structures, such as cache or memory tree.
      
      Note, it's defined in a separate file to minimize dependencies.
      a075fb97
    • Vladimir Davydov's avatar
      vinyl: add wrapper around vy_tx_set · 43e79618
      Vladimir Davydov authored
      This patch adds vy_set and vy_set_with_colmask functions. For now they
      simply forward all arguments to vy_tx_set, but once comparison hints are
      introduced, they will also compute a hint for the inserted statement.
      Later, with the appearance of multikey indexes, they will also extract
      multikey offsets.
      43e79618
    • Vladimir Davydov's avatar
      vinyl: zap vy_mem_iterator_curr_stmt helper · 6affa359
      Vladimir Davydov authored
      It's a trivial one-line function, which can be folded without hurting
      readability, i.e. it only obfuscates the code. Let's kill it.
      6affa359
    • Vladimir Davydov's avatar
      vinyl: rename tree_mem_key to vy_mem_tree_key · 6e793970
      Vladimir Davydov authored
      For aesthetic purposes. No functional changes.
      6e793970
    • Vladimir Davydov's avatar
      vinyl: rename vy_cache_entry to vy_cache_node · 4465a62e
      Vladimir Davydov authored
      In the next patch I'm planning to introduce the concept of vy_entry,
      which will encapsulate a statement stored in a container. Let's rename
      vy_cache_entry to vy_cache_node so as not to mix the two concepts.
      4465a62e
    • Vladimir Davydov's avatar
      Move hint_t definition to tuple_compare.h · 3475eead
      Vladimir Davydov authored
      So as not to include heavy key_def.h when we only need hint_t.
      3475eead
    • Kirill Shcherbatov's avatar
      lib: update msgpuck library · 51855796
      Kirill Shcherbatov authored
      The msgpack dependency has been updated because the new version
      introduces the new method mp_stack_top for the mp_stack class
      which we will use to store a pointer for a multikey frame to
      initialize a field_map in case of multikey index.
      
      As the library API has changed, the code has been modified
      correspondingly.
      
      @locker: add missing frame update in vy_stmt_new_surrogate_delete.
      
      Needed for #1012
      51855796
    • Serge Petrenko's avatar
      xrow: improve corrupted header logging on an error · 9d9b4188
      Serge Petrenko authored
      Improve row printing to log. Since say only has 16k buffer, there is no
      point in printing the whole packet, which can have arbitrary length, in one
      go.
      So, print the header row by row, 16 bytes in a row, and format output to
      match `xxd` output:
      ```
      [001] 2019-04-05 18:22:46.679 [11859] iproto V> Got a corrupted row:
      [001] 2019-04-05 18:22:46.679 [11859] iproto V> 00000000: A3 02 D6 5A E4 D9 E7 68 A1 53 8D 53 60 5F 20 3F
      [001] 2019-04-05 18:22:46.679 [11859] iproto V> 00000010: D8 E2 D6 E2 A3 02 D6 5A E4 D9 E7 68 A1 53 8D 53
      ```
      Now we can get rid of malloc, and use a preallocated tt_static_buf
      instead.
      Also, replace a big macro with a small macro and a helper function.
      
      Followup to f645119f
      9d9b4188
    • Vladimir Davydov's avatar
      Revert "Revert "Add more tests for DDL outside autocommit mode."" · 9bf66953
      Vladimir Davydov authored
      This reverts commit 8be593ce.
      
      Now, as the use-after-free bug in space_truncate() implementation has
      been fixed, we can enable this test again.
      
      Follow-up #4093
      9bf66953
    • Vladimir Davydov's avatar
      box: fix use-after-free in space_truncate · b76542c4
      Vladimir Davydov authored
      space_truncate allocates a statement on the stack which is grossly
      incorrect as the stack may be purged once the function returns while
      box_process_rw expects the statement to be valid until the end of
      the transaction. By happy accident, it worked fine until commit
      1f7b0d65 ("Require for single statement not autocommit in case of
      ddl"), which made it possible to run this function from a transaction
      and hence increased the probability of hitting the use-after-free bug.
      The fix is trivial: allocate a truncation statement on the region.
      
      Fixes commit 353bcdc5 ("Rework space truncation").
      
      Closes #4093
      b76542c4
    • Alexander Turenko's avatar
      test: enable cleaning of a test environment · 7474c14e
      Alexander Turenko authored
      This commit enables pretest_clean test-run option on 'core = tarantool'
      test suites with Lua tests and 'core = app' test suites. Consider #4094
      for an example of a problem that is eliminated by this option.
      
      For 'core = tarantool': this option drops non-system spaces, drops data
      in system spaces and global variables to the initial state, unloads
      packages except build-in ones.
      
      For 'core = app': this option deletes xlog and snap files before run a
      test.
      
      test-run doesn't remove global variables that are listed in the
      'protected_globals' global variable. Use it for, say, functions that are
      defined in an instance file and called from tests.
      
      Consider test-run/README.md for the information how exactly the option
      works.
      
      Removed unused cfg_filter() function from test/engine/box.lua.
      
      Fixes #4094.
      7474c14e
  4. Apr 05, 2019
    • Alexander Turenko's avatar
      Revert "Add more tests for DDL outside autocommit mode." · 8be593ce
      Alexander Turenko authored
      This reverts commit 14a87bb7.
      
      The test cases generate corrupted xlog files (see #4093) and don't allow
      other tests to proceed successfully, so we need to temporary disable
      these cases. They should be enabled back in the scope of #4093.
      8be593ce
    • Alexander Turenko's avatar
      test: update test-run · 4ee8910b
      Alexander Turenko authored
      * Added default timeout for wait_cond() (60 sec).
      * Updated pyyaml version in requirements.txt.
      * Fixed reporting of non-default server fail at start.
      * Stop 'proxy' when a new non-default instance fails.
      * Added user-defined protected globals for pretest_clean.
      4ee8910b
    • Nikita Pettik's avatar
      sql: fix extra type calculation before bytecode generation · 2f6f3bbd
      Nikita Pettik authored
      In SQL type of constant literal (e.g. 1, 2.5, 'abc') is assigned right
      after parsing and saving into struct Expr. Occasionally, type is
      re-assigned before emitting opcodes to store literal into VDBE memory.
      What is more, for floating point number type is changed to "integer".
      This patch fixes this obvious misbehaviour.
      2f6f3bbd
    • Vladimir Davydov's avatar
      Drop const qualifier of struct tuple · 077671fe
      Vladimir Davydov authored
      Using the const qualifier for complex structures like tuple is bad.
      We already have to cast it to drop the const qualifier now and then,
      e.g. to increment/decrement the reference counter.
      
      We are planning to wrap struct tuple in a helper struct (aka entry) to
      store it in vinyl containers along with a comparison hint (cache, memory
      tree, etc). We will be passing this struct by value so we won't be able
      to retain const qualifier, because in contrast to a const pointer, one
      must initialize a const struct upon definition.
      
      That said, it's time to drop const qualifier of struct tuple everywhere,
      like we have already done in case of struct key_def and tuple_format.
      077671fe
    • Mergen Imeev's avatar
      box: remove _sql_stat1 and _sql_stat4 system tables · ec93b4a5
      Mergen Imeev authored
      These tables won't be used anymore and should be deleted.
      
      Note, this patch breaks backward compatibility between 2.1.1 and
      2.1.2, but that's okay as 2.1.1 was beta and we didn't recommend
      anyone to use it.
      
      Part of #2843
      Follow up #4069
      ec93b4a5
    • Mergen Imeev's avatar
      sql: allocate memory for index_id in VDBE · fd6e4b94
      Mergen Imeev authored
      Currently, the memory for index_id is not allocated in VDBE code
      in the sql_code_drop_table() and sql_drop_index() functions. This
      may lead to SEGMENTATION FAULT.
      
      Needed for #2843
      fd6e4b94
    • Kirill Shcherbatov's avatar
      sql: fix perf degradation on name normalization · 4f1f6ab4
      Kirill Shcherbatov authored
      Because sql_normalize_name used to be called twice - to estimate
      the size of the name buffer and to process data querying the
      UCaseMap object each time performance in SQL felt by 15%.
      
      This patch should eliminate some of the negative effects of using
      ICU for name normalization.
      
      Thanks @avtikhon for a bechmark
      
      Follow up e7558062
      4f1f6ab4
  5. Apr 04, 2019
    • Mergen Imeev's avatar
      sql: set consistent names · bcd5dd84
      Mergen Imeev authored
      This patch changes the name of the function used in the port_sql
      methods and the name of the member of the result of the execution.
      
      Follow up #3505
      bcd5dd84
    • Vladislav Shpilevoy's avatar
      test: fix swim test on a bit cluster failure detection · 4e6bbc45
      Vladislav Shpilevoy authored
      It takes random time since SWIM uses random members selection
      for each round step. Appeared, that in one of tests choosen time
      was too small sometimes. Now it is fixed by just increasing
      timeout, but one of forthcoming patches reworks that test in
      order to make it faster.
      
      Follow up for f510dc6f
      4e6bbc45
    • Serge Petrenko's avatar
      xrow: print corrupted rows on decoding error. · f645119f
      Serge Petrenko authored
      Add row hex printing to log on verbose level. This would be useful
      during investigation of errors related to invalid msgpack packet
      arrival.
      
      Here's some example output:
      ```
      2019-04-01 16:04:51.629 [31304] iproto V> Corrupted row is: 3D A5 41 55 99 18 A8 08 C2 40 2B BA 50 12 F6 C6 0B 27 66 1C
      2019-04-01 16:04:51.629 [31304] iproto xrow.c:133 E> ER_INVALID_MSGPACK: Invalid MsgPack - packet header
      ```
      
      Related to #4040
      f645119f
    • Vladislav Shpilevoy's avatar
      swim: introduce failure detection component · f510dc6f
      Vladislav Shpilevoy authored
      Failure detection components allows to find which members are
      already dead. It works via pings attached to each round message.
      A member, received ping, should respond with an ack. If it is not
      done within a timeout, then the ping is considered unacknowledged.
      When a number of pings grows to a threshold, the member is
      declared DEAD. After more unacked pings it is dropped from the
      member table. Unless the GC was turned off.
      
      Part of #3234
      f510dc6f
  6. Apr 03, 2019
    • Alexander Turenko's avatar
      lua: add luaT_tuple_new() · 64a6464d
      Alexander Turenko authored
      The function allows to create a tuple with specific tuple format in C
      code using a Lua table, another tuple, or objects on a Lua stack.
      
      Needed for #3276, #3398, #4025
      64a6464d
    • Kirill Shcherbatov's avatar
      box: move offset_slot init to tuple_format_add_field · 7835e8a1
      Kirill Shcherbatov authored
      Due to the fact that the allocation of offset_slot in the case of
      multikey indexes will become more complicated and will be
      necessary for intermediate nodes of the tuple_field tree, we must
      move this logic to the tuple_format_add_field that performs
      an intermediate nodes allocation for a JSON path.
      
      Needed for #1257
      7835e8a1
    • Kirill Shcherbatov's avatar
      lib: introduce a new json_path_multikey_offset helper · bc0b8873
      Kirill Shcherbatov authored
      Introduced a new procedure json_path_multikey_offset. This helper
      scans the JSON path string and returns the offset of the first character
      [*] (the array index placeholder).
      
      We need this function in the scope of the multikey index patchset to
      extract the number of keys to be inserted into the index
      using JSON subpath that has json_path_multikey_offset() length.
      
      Needed for #1257
      bc0b8873
    • Kirill Shcherbatov's avatar
      box: cleanup key_def virtual extract_key setter · 18f96090
      Kirill Shcherbatov authored
      This patch is inspired by commit 829c811c ("tuple_compare: cleanup
      key_def virtual func setter") and attempts to simplify setting
      appropriate tuple_extract_key pointer for plain and json indexes in
      key_def_set_extract_func routine. Being split to plain and json blocks
      this code becomes easier to understand and extend.
      
      In further patches we need to introduce is_multikey branch and without
      this refactoring required amendments turn the key_def_set_extract_func
      code into a mess.
      
      Needed for #1257
      18f96090
    • Alexander Turenko's avatar
      test: use pretest_clean on sql-tap test suite · 96cdc5ba
      Alexander Turenko authored
      A problem appears when tests are run in the following order on one
      test-run worker:
      
      - sql-tap/select6.test.lua
      - sql-tap/gh-4077-iproto-execute-no-bind.test.lua
      
      The latter one fails with the following error:
      
      > Space 'T' already exists
      
      This test does not perform require('sqltester'), where *.{xlog,snap}
      files from a previous test are removed, and so the test can observe a
      non-clean state of a database.
      
      Now test-run has pretest_clean option that do exactly what sqltester had
      been doing: when the option enabled test-run removes *.{xlog,snap} files
      before run a 'core = app' test. The only difference that now these files
      will be removed for every test in sql-tap test suite, not only for ones
      that use sqltester.
      
      Follows up #4077.
      96cdc5ba
    • Alexander Turenko's avatar
      test: temporary disable sql-tap/collation_unicode · 4dfa31f7
      Alexander Turenko authored
      The test was introduced in a99d7a0c
      ('Feature request for a new collation'). It succeeds when tarantool is
      linked with a fresh libicu version, but fails with older ones. At least:
      
      * dev-libs/icu-63.1-r1 (Gentoo) -- passes;
      * libicu.x86_64 50.1.2-15.el7 (CentOS 7.4) -- fails.
      
      We need to handle different behaviours of libicu versions somehow, maybe
      add only stable subset of collations. This will also fix the test and
      then it will be enabled back.
      
      Related to #4007.
      4dfa31f7
  7. Apr 02, 2019
    • Vladislav Shpilevoy's avatar
      test: differentiate blocked and closed swim fake fds · 5b4e6957
      Vladislav Shpilevoy authored
      SWIM's fake file descriptors were implemented to test SWIM with
      virtual time and fully controlled network with immediate packet
      delivery. One of their features - API to block a file descriptor
      and test various failures about it.
      
      But blocked fake fd looks the same as closed fd, and it can
      confuse new test's author. Now if an fd is not unblocked at the
      end of a test, it leads to a crash. This commit fixes that via
      adding explicit difference between blocked and closed fd.
      
      Part of #3234
      5b4e6957
    • Vladislav Shpilevoy's avatar
      swim: expose enum swim_member_status to public API · e854e7df
      Vladislav Shpilevoy authored
      At least for testing it is necessary to be able to get status of
      a member. Now it is always 'alive', but forthcoming
      failure-detection component would change it.
      
      Part of #3234
      e854e7df
Loading