Skip to content
Snippets Groups Projects
  1. Mar 19, 2019
    • Alexander Turenko's avatar
      travis: add Fedora 28 and 29 · 67dcbf28
      Alexander Turenko authored
      Fixes #4027.
      67dcbf28
    • Alexander Turenko's avatar
      lua: remove digest.sha() and digest.sha_hex() · 390693a1
      Alexander Turenko authored
      SHA-0 is considered weak for years and was removed in OpenSSL since
      1.1.0.
      
      These Lua functions did not work since 15ed10e4, but give 'Digest
      method "sha" is not supported' error for any input. Removed them to
      don't confuse a user with a Tab completion.
      
      Follow up #1722.
      Fixes #4028.
      390693a1
    • Ivan Koptelov's avatar
      sql: forbid diff. sorting orders in ORDER BY + LIMIT · aa4a2de6
      Ivan Koptelov authored
      Without multi-directional iterators (to be introduced in #3309)
      ORDER BY + LIMIT with different specified ordered leads to wrong
      result. So for now (till #3309 is resolved) different sort
      orders are forbidden in ORDER BY + LIMIT.
      
      Close #4038
      
      @TarantoolBot document
      Title: different sorting orders in ORDER BY are forbidden now
      The error will be raised if different sorting orders are
      encountered.
      aa4a2de6
    • Mergen Imeev's avatar
      sql: set SQL parser errors via diag_set() · 3abd5926
      Mergen Imeev authored
      After this patch all SQL parser errors will be set via diag_set().
      They were saved in field zErrMsg of struct Parse before this
      patch.
      
      Part of #3965
      3abd5926
  2. Mar 18, 2019
    • Alexander Turenko's avatar
      test: update test-run · be7d940f
      Alexander Turenko authored
      Fixed _func system space clean when pretest_clean is set to True in
      suite.ini.
      be7d940f
    • Vladimir Davydov's avatar
      xrow: fix request_str crash on long requests · 9e054f18
      Vladimir Davydov authored
      If tt_static_buf is too small to store the request string, 'pos' will
      become greater than 'end', leading to snprintf(pos, end - pos) crash, as
      it doesn't allow the buffer size to be negative. Use SNPRINT instead.
      9e054f18
    • Cyrill Gorcunov's avatar
      lib/core/fiber: Fix compilation if !HAVE_MADV_DONTNEED · a6374289
      Cyrill Gorcunov authored
      Fixes commit 553dc562 ("lib/core/fiber: Relax stack memory usage on
      recycle").
      a6374289
    • Vladimir Davydov's avatar
      lib/core/fiber: Log madvise and mprotect errors · f0047404
      Vladimir Davydov authored
      So that we notice in case anything goes wrong.
      f0047404
    • Cyrill Gorcunov's avatar
      lib/core/fiber: Relax stack memory usage on recycle · 553dc562
      Cyrill Gorcunov authored
      We want to detect a situation where task in fiber is too eager for
      stack memory and relax rss usage in such case. For this sake upon
      stack creation we put 8 marks near 64K bound (such params allows us
      to fill ~1/4 of a page, which seem reasonable but we might change
      this params with time).
      
      Once stack get recycled we investigate the marks and if they were
      overwritten we drop all pages behind to relax memory usage (if OS
      supports madvise syscall).
      
      Another important moment is that we're marking the whole stack
      as not present thus if fiber never stepped over 64K limit the
      marks will be in tact and it means the fibers are light ones
      there won't be much #pf in future.
      
      Later we plan to implement an intelligent fiber scheduling
      considering how many memory fibers consume in average.
      
      @locker:
       - fix watermark page alignment for grow-up stack
       - improve MADV_DONTNEED check
       - clean up code and elaborate comments
       - add test case to unit/fiber
       - fix unit/guard test
      
      Follow-up #3418
      553dc562
    • Cyrill Gorcunov's avatar
      lib/core/fiber: Increase default stack size · 82f4b4a3
      Cyrill Gorcunov authored
      The default 64K stack size used for years become too small
      for modern distors (Fedora 29 and etc) where third party libraries
      (such as ncurses) started to use 64K for own buffers and we get
      SIGSGV early without reaching interactive console phase.
      
      Thus we increase default size up to 512K which should fit
      for common case. Later we will make this value configurable
      to address arbitrary stack sizes without a need to rebuild
      the whole code.
      
      Closes #3418
      82f4b4a3
  3. Mar 15, 2019
    • Vladimir Davydov's avatar
      test: don't leave open read views after vinyl/errinj_tx · 0c7a3e39
      Vladimir Davydov authored
      See commit 65061cb4 ("test: don't leave open read views after
      vinyl/iterator") for more details.
      
      Follow-up #3862
      0c7a3e39
    • Vladimir Davydov's avatar
      test: fix vinyl/errinj failure · 61756cce
      Vladimir Davydov authored
      The patch fixes the following test failure:
      
       | --- vinyl/errinj.result Fri Mar 15 19:19:19 2019
       | +++ vinyl/errinj.reject Fri Mar 15 19:21:37 2019
       | @@ -854,7 +854,8 @@
       |  ...
       |  ret
       |  ---
       | -- - [3, 3]
       | +- - [2, 2]
       | +  - [3, 3]
       |  ...
       |  s:drop()
       |  ---
      
      The error occurrs, because the test doesn't make sure that a read from
      the scondary index actually occurs and the fiber does stall on a read
      from the primary index. Use index stats to assure that.
      
      Closes #3863
      61756cce
    • Alexander V. Tikhonov's avatar
      travis-ci: fix OS X Mojave 10.14 build · eb964471
      Alexander V. Tikhonov authored
      Fixed Mojave Mac build with setting MACOSX_DEPLOYMENT_TARGET environment
      variable for LuaJIT's Makefile. This variable specifies the minimum
      version of OS X on which the target binaries are to be deployed.
      
      The reason why we need to set MACOSX_DEPLOYMENT_TARGET to at least 10.6
      is that 10.4 SDK (which is set by default in LuaJIT's Makefile) is not
      longer included in Mac OS X Mojave 10.14.
      
      See also https://github.com/LuaJIT/LuaJIT/issues/484
      
      We already set -Wl,-macosx_version_min,10.6 (-macosx_version_min and
      MACOSX_DEPLOYMENT_TARGET are synonymous), but it affects only a linker.
      We possibly should remove -macosx_version_min, because it superseded by
      MACOSX_DEPLOYMENT_TARGET, but it should be done with verification that
      the original problem really fixed by the new way to set a deployment
      target. This is not part of this patch.
      
      Removed virtualenv usage in CI for OS X (so pip just installs packages
      into a system), because OS X Mojave 10.14 does not offer virtualenv by
      default.
      
      Fixed #3797
      eb964471
    • Vladimir Davydov's avatar
      test: fix vinyl/deferred_delete failure · 5433435d
      Vladimir Davydov authored
      The patch fixes the following test failure:
      
       | --- vinyl/deferred_delete.result        Fri Mar 15 18:17:05 2019
       | +++ vinyl/deferred_delete.reject        Fri Mar 15 18:18:18 2019
       | @@ -577,7 +577,7 @@
       |  ...
       |  sk:stat().rows -- ditto
       |  ---
       | -- 5
       | +- 25
       |  ...
       |  s:drop()
       |  ---
      
      The failure was introduced by commit 6dd0d2fb ("vinyl: do not apply
      run_count_per_level to the last level"). Due to the commit compaction of
      the secondary index may happen before compaction of the primary index,
      in which case deferred DELETE statements won't make it to the secondary
      index in time against the test expectation. Fix this by making the first
      run big enough to prevent major compaction from kicking in.
      
      Follow-up #3657
      Closes #4047
      5433435d
    • Nikita Pettik's avatar
      sql: make LIKE accept only TEXT arguments · d902cd11
      Nikita Pettik authored
      According to ANSI[1] specification, LIKE arguments must be of
      string-like type (VARCHAR, CHAR, TEXT etc). If one of arguments is NULL,
      then the result of LIKE function is NULL as well. This patch makes LIKE
      follow these rules.
      
      ANSI 2013 Part 2: Foundation; Chapter 8.5 <like predicate>
      
      Closes #3954
      d902cd11
    • Nikita Pettik's avatar
      sql: remove sql_like_count global counter · 24741360
      Nikita Pettik authored
      We don't rely on this debug facility anymore, so let's remove it.
      24741360
    • Nikita Pettik's avatar
      sql: remove SQL_LIKE_DOESNT_MATCH_BLOBS · fd1888d4
      Nikita Pettik authored
      We are going to always throw an error if value of BLOB type gets to LIKE
      arguments, so code under is macro is not needed anymore.
      
      Part of #3954
      fd1888d4
    • Vladimir Davydov's avatar
      vinyl: reset dump watermark after updating memory limit · b15773fa
      Vladimir Davydov authored
      The watermark is updated every second anyway, however not updating it
      when the limit is reset results in vinyl/quota test failure:
      
       | --- vinyl/quota.result  Thu Mar 14 16:03:54 2019
       | +++ vinyl/quota.reject  Fri Mar 15 16:32:44 2019
       | @@ -146,7 +146,7 @@
       |  for i = 1, count do s:replace{i, pad} end -- does not trigger dump
       |  ---
       |  ...
       | -box.stat.vinyl().memory.level0 > count * pad:len()
       | +box.stat.vinyl().memory.level0 > count * pad:len() or box.stat.vinyl()
       |  ---
       |  - true
       |  ...
      
      Closes #3864
      b15773fa
    • Mergen Imeev's avatar
      sql: rework syntax errors · b7eed190
      Mergen Imeev authored
      This patch reworks SQL syntax errors. After this patch, these
      error will be set as Tarantool errors.
      
      Part of #3965
      b7eed190
    • Kirill Yukhin's avatar
      Update luajit submodule · be0506db
      Kirill Yukhin authored
      Fixed GC issue.
      be0506db
    • Vladimir Davydov's avatar
      test: don't leave open read views after vinyl/iterator · 65061cb4
      Vladimir Davydov authored
      An open iterator may disrupt the following test run, because it may
      prevent dump/compaction from purging stale rows. In particular,
      iterators left by vinyl/iterator result in the following test failure:
      
       | --- vinyl/deferred_delete.result        Mon Feb 11 19:14:01 2019
       | +++ vinyl/deferred_delete.reject        Fri Mar 15 16:21:11 2019
       | @@ -155,7 +155,7 @@
       |  ...
       |  pk:stat().rows -- 5 new REPLACEs
       |  ---
       | -- 5
       | +- 10
       |  ...
       |  i1:stat().rows -- 10 old REPLACE + 5 new REPLACEs + 10 deferred DELETEs
       |  ---
      
      Fix this by calling the Lua garbage collector to delete all dangling
      iterators in the end of vinyl/iterator test.
      
      Closes #3862
      65061cb4
  4. Mar 14, 2019
    • Serge Petrenko's avatar
      evio: fix timeout calculations · 2c0d418b
      Serge Petrenko authored
      The function evio_timeout_update() failed to update the starting time
      point, which lead to timeouts happening much faster than they should if
      there were consecutive calls to the function.
      This lead, for example, to applier timing out while reading a several
      megabyte-size row in 0.2 seconds even if replication_timeout was set to
      15 seconds.
      
      Closes #4042
      2c0d418b
  5. Mar 13, 2019
    • Alexander Turenko's avatar
      test: update test-run · 9d50d571
      Alexander Turenko authored
      Run a unit test from a var directory. It is needed to, say, allow a test
      to write a log file to a gitignored directory.
      
      The only behaviour change observed on tarantool's tests is that
      unit/swim.test and unit/swim_proto.test write a log.txt file to a test
      var directory ./test/xxx_unit instead of ./test.
      9d50d571
    • Sergei Voronezhskii's avatar
      test: enable parallel run for python test suites · 4ea7d729
      Sergei Voronezhskii authored
      Fixed issues:
      
      - box-py/iproto.test.py
        1) Fixed receive_response() to wait for whole response.
        2) Clean up _cluster space.
      
      - replication-py/cluster.test.py
        1) Clean up _cluster space.
      
      - replication-py/multi.test.py
        1) Removed vclock checking because it fails if previous test make some
           DML and vclock was incremented. Looks like it was used for debug
           and is not part of this test case.
        2) Fixed typo in 'Synchronize' block.
      
      The following test sequences did fail due to unexpected IDs in _cluster
      space:
      
      - [box-py/iproto.test.py, null]
      - [box-py/bootstrap.test.py, null]
      
      - [replication-py/cluster.test.py, null]
      - [replication-py/multi.test.py, null]
      
      Part of #3232
      4ea7d729
    • Sergei Voronezhskii's avatar
      test: enable parallel run for long test suites · e01c58fa
      Sergei Voronezhskii authored
      Also enabled it for luajit-tap.
      
      Part of #3232
      e01c58fa
    • Vladimir Davydov's avatar
      tuple_format: zap min_tuple_size · 75ed5b13
      Vladimir Davydov authored
      It had been used only in Vinyl's vy_stmt_new_surrogate_from_key, which
      was deleted by the previous patches, so we can drop it as well.
      75ed5b13
    • Vladimir Davydov's avatar
      vinyl: don't use vy_stmt_new_surrogate_delete if not necessary · 73ad8c3b
      Vladimir Davydov authored
      There are three places where we use this expensive functions while we
      could get along with a cheaper one:
      
       - Deferred DELETE space on_replace trigger. Here we can use simple
         vy_stmt_new_delete, because the trigger is already passed a surrogate
         DELETE statement.
      
       - Secondary index build on_replace trigger. Here we can extract the
         secondary key, set its type to DELETE and insert it into the index.
         We don't need all the other indexed fields.
      
       - Secondary index build recovery procedure. Similarly to the previous
         case, we can use extracted here rather than building a surrogate
         DELETE statement.
      73ad8c3b
    • Vladimir Davydov's avatar
      vinyl: zap vy_stmt_new_surrogate_from_key · 7c782047
      Vladimir Davydov authored
      This heavy function isn't needed anymore, as we can now insert key
      statements into the memory level.
      7c782047
    • Vladimir Davydov's avatar
      vinyl: do not fill secondary tuples with nulls when decoded · 5f7524b6
      Vladimir Davydov authored
      In contrast to a primary index, which stores full tuples, secondary
      indexes only store extended (secondary + primary) keys on disk. To make
      them look like tuples, we fill missing fields with nulls (aka tuple
      surrogate). This isn't going to work nicely with multikey indexes
      though: how would you make a surrogate array from a key? We could
      special-case multikey index handling, but that would look cumbersome.
      So this patch removes nulls from secondary tuples restored from disk
      altogether. To achieve that, it's enough to use key_format for them -
      then the comparators will detect that it's actually a key, not a tuple
      and use the appropriate primitive.
      5f7524b6
    • Vladimir Davydov's avatar
      vinyl: zap vy_write_iterator->format · 902d212b
      Vladimir Davydov authored
      It's actually only needed to initialize disk streams so let's pass it
      to vy_write_iterator_new_slice() instead.
      902d212b
    • Vladimir Davydov's avatar
      vinyl: clean up write iterator source destruction · 2f17c929
      Vladimir Davydov authored
      By convention we have two methods in each write iterator stream
      implementation (including the write iterator itself as it implements
      the interface too): 'stop' and 'close'. The 'stop' method is called
      in a worker thread. It reverses the effect of 'start'. We need it
      unreference all tuples referenced during the iteration (we must do
      it in the worker thread, where the tuples were referenced in the first
      place so as not to unreference tuple formats, see vy_tuple_delete).
      The 'close' method is called from the tx thread to unreference tuple
      formats if necessary and release memory.
      
      For the write iterator itself we follow this convention. However,
      for individual sources, for vy_slice_stream source to be more exact,
      we do not - the write iterator calls both 'stop' and 'close' from
      its own 'stop method. Let's cleanup this mess and make the write
      iterator follow the convention. We'll need it in the next patch.
      2f17c929
    • Vladimir Davydov's avatar
      vinyl: do not pass format to vy_apply_upsert · 54b8add5
      Vladimir Davydov authored
      Use the format of the given statement instead. Passing format is
      a legacy from the time when we have a separate format for UPSERTs.
      Nowadays it only obfuscates the code.
      54b8add5
    • Vladimir Davydov's avatar
      vinyl: add helpers to add/check statement with bloom · 8189b054
      Vladimir Davydov authored
      A Vinyl statement may be either a key or a tuple. We must use different
      functions for the two kinds when working with a bloom filter. Let's
      introduce helpers incorporating that logic.
      
      Notes:
       - Currently, we never add keys to bloom filters, but after the next
         patch we will, so this patch adds tuple_bloom_builder_add_key helper.
       - According to the function protocol, tuple_bloom_builder_add may fail
         with out-of-memory, but we never checked that. Fix that while we are
         at it.
      8189b054
    • Vladimir Davydov's avatar
      bloom: factor out helper to add tuple hash to bloom builder · e9728589
      Vladimir Davydov authored
      No functional changes, just move a piece of code, so as not to mix it in
      the next patch.
      e9728589
    • Vladimir Davydov's avatar
      bloom: do not use tuple_common_key_parts when constructing tuple bloom · afd30b95
      Vladimir Davydov authored
      Tuple bloom filter is an array of bloom filters, each of which reflects
      lookups by all possible partial keys. To optimize the overall bloom
      filter size, we need to know how many unique elements there are for each
      partial key. To achieve that, we require the caller to pass the number
      of key parts that have been hashed for the given tuple. Here's how it
      looks in Vinyl:
      
              uint32_t hashed_parts = writer->last_stmt == NULL ? 0 :
                      tuple_common_key_parts(stmt, writer->last_stmt,
                                             writer->key_def);
              tuple_bloom_builder_add(writer->bloom, stmt,
                                      writer->key_def, hashed_parts);
      
      Actually, there's no need in such a requirement as instead we can
      calculate the hash value for the given tuple, compare it with the hash
      of the tuple added last time, and add the new hash only if the two
      values differ. This should be accurate enough while allowing us to get
      rid of the cumbersome tuple_common_key_parts helper. Note, such a check
      will only work if tuples are added in the order defined by the key
      definition, but that already holds - anyway, one wouldn't be able to
      use tuple_common_key_parts either if it wasn't true.
      
      While we are at it, refresh the obsolete comment to tuple_bloom_builder.
      afd30b95
    • Vladimir Davydov's avatar
      vinyl: don't use IPROTO_SELECT type for key statements · bb4db772
      Vladimir Davydov authored
      To differentiate between key and tuple statements in comparators, we set
      IPROTO_SELECT type for key statements. As a result, we can't use key
      statements in the run iterator directly although secondary index runs do
      store statements in key format. Instead we create surrogate tuples
      filling missing fields with NULLs. This won't play nicely with multikey
      indexes so we need to teach iterators to deal with statements in key
      format. The first step in this direction is dropping IPROTO_SELECT in
      favor of identifying key statements by format.
      bb4db772
    • Vladimir Davydov's avatar
      vinyl: rename key stmt construction routine · ce1066ed
      Vladimir Davydov authored
      Currently, it's called vy_stmt_new_select, but soon a key statement will
      be allowed to have any type, not just IPROTO_SELECT. So let's rename it
      to vy_key_new.
      ce1066ed
    • Vladimir Davydov's avatar
      vinyl: introduce statement environment · d8fbb5e3
      Vladimir Davydov authored
      Store tuple_format_vtab, max_tuple_size, and key_format there.
      This will allow us to determine a statement type (key or tuple)
      by checking its format against key_format.
      d8fbb5e3
    • Vladimir Davydov's avatar
      vinyl: remove optimized comparators · f11b82a6
      Vladimir Davydov authored
      A vinyl statement (vy_stmt struct) may represent either a tuple or a
      key. We differentiate between the two kinds by statement type - we use
      SELECT for keys and other types for tuples. This was done that way so
      that we could pass both tuples and keys to a read iterator as a search
      key. To avoid branching in comparators when the types of compared
      statements are known in advance, we provide several comparators, each of
      which expects certain statement types, e.g. a tuple and a key. Actually,
      such a micro optimization looks like an overkill, because a typical
      comparator is called by function pointer and has a lot of comparisons
      in the code, see tuple_compare_slowpath for instance. Eliminating one
      branch will hardly make the code perform better. At the same time, it
      makes the code more difficult to write. Besides, once we remove nils
      from statements read from disk (aka surrogate tuples), which will
      ease implementation of multikey indexes, the number of places where
      types of compared statements are known will diminish drastically.
      That said, let's remove optimized comparators and always use
      vy_stmt_compare, which checks types of compared statements and calls
      the appropriate comparator.
      f11b82a6
    • Vladimir Davydov's avatar
      replication: update replica gc state on subscribe · b5b4809c
      Vladimir Davydov authored
      We advance replica->gc state only when an xlog file is fully recovered,
      see recovery_close_log and relay_on_close_log_f. It may turn out that an
      xlog file is fully recovered, but isn't closed properly by relay (i.e.
      recovery_close_log isn't called), because the replica closes connection
      for some reason (e.g. timeout). If this happens, the old xlog file
      won't be removed when the replica reconnects, because we don't advance
      replica->gc state on reconnect, so the useless xlog file won't be
      removed until the next xlog file is relayed. This results in occasional
      replication/gc.test.lua failures. Fix this by updating replica->gc on
      reconnect with the current replica vclock.
      
      Closes #4034
      b5b4809c
Loading