Skip to content
Snippets Groups Projects
  1. Aug 22, 2019
    • Serge Petrenko's avatar
      decimal: add conversions to (u)int64_t · 8d2e7839
      Serge Petrenko authored
      Update decNumber library, add methods to convert decimals to uint64_t
      and int64_t, add unit tests.
      Also replace decimal_round() function with decimal_round_with_mode() to
      allow setting rounding mode.
      We need to round with mode DEC_ROUND_DOWN in to_int64 conversions in
      order to be consistent with double to int conversions.
      It will be needed to compute hints for decimal fields.
      
      Prerequisite #4333
      8d2e7839
    • Serge Petrenko's avatar
      decimal: allow to encode/decode decimals as MsgPack · 485439e3
      Serge Petrenko authored
      This patch adds the methods necessary to encode and decode decimals to
      MsgPack. MsgPack EXT type (MP_EXT) together with a new extension type
      MP_DECIMAL is used as a record header.
      
      The decimal MsgPack representation looks like this:
      +--------+-------------------+------------+===============+
      | MP_EXT | length (optional) | MP_DECIMAL | PackedDecimal |
      +--------+-------------------+------------+===============+
      The whole record may be encoded and decoded with
      mp_encode_decimal() and mp_decode_decimal(). This is equivalent to
      performing mp_encode_extl()/mp_decode_extl() on the first 3 fields and
      decimal_pack/unpack() on the PackedDecimal field.
      
      It is also possible to decode and encode decimals to msgpack from lua,
      which means you can insert decimals into spaces, but only into unindexed
      fields for now.
      
      Follow up #692
      Part of #4333
      485439e3
  2. Aug 21, 2019
    • Serge Petrenko's avatar
      lua/pickle: fix a typo · e2d9f664
      Serge Petrenko authored
      Prior to this patch format checking was broken for 'i' (integer) and 'N'
      (big-endian integer). pickle.pack() rejected negative integers with
      these formats. Fix this
      e2d9f664
    • Serge Petrenko's avatar
      decimal: fix encoding numbers with positive exponent. · fb3a2dfa
      Serge Petrenko authored
      When a number having a positive exponent is encoded, the internal
      decPackedFromNumber function returns a negative scale, which differs
      from the scale, returned by decimal_scale(). This leads to errors in
      decoding. Account for negative scale in decimal_pack() and
      decimal_unpack().
      
      Follow-up #692
      fb3a2dfa
    • Serge Petrenko's avatar
      lua: fix decimal comparison with nil · 3c6c1cc9
      Serge Petrenko authored
      Previously decimal comparison with nil failed with following error:
      `expected decimal, number or string as 2 argument`.
      Fix this. Throw a more verbose error in case of '>', '<', '>=', '<='
      and fix equality check.
      
      Follow-up #692
      3c6c1cc9
    • Mergen Imeev's avatar
      build: link libcurl statically from a submodule · 7e51aebb
      Mergen Imeev authored
      Hold libcurl-7.65.3. This version is not affected by the following
      issues:
      
      * #4180 ('httpc: redirects are broken with libcurl-7.30 and older');
      * #4389 ('libcurl memory leak');
      * #4397 ('HTTPS seem to be unstable').
      
      After this patch libcurl will be statically linked when
      ENABLE_BUNDLED_LIBCURL option is set. This option is set by default.
      
      Closes #4318
      
      @TarantoolBot document
      Title: Tarantool dependency list was changed
      
      * Added build dependencies: autoconf, automake, libtool, zlib-devel
        (zlib1g-dev on Debian).
      * Added runtime dependencies: zlib (zlib1g on Debian).
      * Removed build dependencies: libcurl-devel (libcurl4-openssl-dev on
        Debian).
      * Removed runtime dependencies: curl.
      
      The reason is that now we use compiled-in libcurl: so we don't depend on
      a system libcurl, but inherit its dependencies.
      7e51aebb
    • Alexander Turenko's avatar
      lua: workaround pwd.getpwall() issue on Fedora 29 · f5d8331e
      Alexander Turenko authored
      This is a workaround for systemd-nss issue:
      https://github.com/systemd/systemd/issues/9585
      
      The following error is observed on app-tap/pwd.test.lua on Fedora 29
      (glibc-2.28-26.fc29, systemd-239-12.git8bca462.fc29) when tarantool is
      linked with libcurl w/o GSS-API support:
      
       | builtin/pwd.lua:169: getpwall failed [errno 2]: No such file or directory
      
      Such tarantool build lacks of libselinux.so.1 transitive dependency
      (tarantool -> libcurl.so.4 -> libgssapi_krb5.so.2 -> libkrb5support.so.0
      -> libselinux.so.1) and strace shows the following calls when
      pwd.getpwall() is invoked first time:
      
       | openat(AT_FDCWD, "/lib64/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 7A
       | <...>
       | access("/etc/selinux/config", F_OK)     = -1 ENOENT (No such file or directory)
      
      It looks like a part of libselinux initialization code and is invoked
      during execution of a last ffi.C.getpwent() call that returns `nil` as a
      result and left errno set to ENOENT. Our pwd module set errno to zero
      before getpwent() call and expects that it will be preserved if no
      unrecoverable errors occur. It seems that this expectation is not meet
      due to the systemd-nss issue linked above.
      
      Second and next getpwall() calls will succeed, so the commit adds an
      extra getpwall() during pwd module load. This workaround is disabled on
      FreeBSD due to another issue: #4428 ('getpwall() hangs on FreeBSD 12').
      
      See also the previous related commit:
      efccac69 ('lua: fix error handling in
      getpwall and getgrall').
      
      Follows up #3766.
      Part of #4318.
      f5d8331e
    • Alexander V. Tikhonov's avatar
      gitlab-ci: add static build · f7509186
      Alexander V. Tikhonov authored
      Added static build using Dockerfile on Centos 7 for release
      commit criteria only. Added the cleanup for cmake generating
      CMakeCache.txt files and CMakeFiles directories to avoid of
      cmake localy created setup failing inside the docker after
      the whole tarantool path was copied into it. Added testing
      into the static build, running only when RUN_TESTS environment
      variable set to non empty value, used in gitlab-ci job to run
      the testing after the build.
      
      Closes #3668
      f7509186
  3. Aug 20, 2019
    • Vladimir Davydov's avatar
      vinyl: don't exempt dropped indexes from dump and compaction · d7387ec9
      Vladimir Davydov authored
      We remove an LSM tree from the scheduler queues as soon as it is
      dropped, even though the tree may hang around for a while after
      that, e.g. because it is pinned by an iterator. As a result, once
      an index is dropped, it won't be dumped anymore - its memory level
      will simply disappear without a trace. This is okay for now, but
      to implement snapshot iterators we must make sure that an index
      will stay valid as long as there's an iterator that references it.
      
      That said, let's delay removal of an index from the scheduler queues
      until it is about to be destroyed.
      d7387ec9
    • Vladimir Davydov's avatar
      space: get rid of apply_initial_join_row method · 1314b95b
      Vladimir Davydov authored
      There's no reason to use a special method instead of the generic
      space_execute_dml for applying rows received from a master during the
      initial join stage. Moreover, using the special method results in not
      running space.before_replace trigger, which makes it impossible to, for
      example, update space engine on a replica, see the on_schema_init test
      of the replication test suite.
      
      So this patch removes the special method altogether and makes the code
      that used it switch to space_execute_dml.
      
      Closes #4417
      1314b95b
    • Vladimir Davydov's avatar
      memtx: enter small delayed free mode from snapshot iterator · 6136f84f
      Vladimir Davydov authored
      We must enable SMALL_DELAYED_FREE_MODE to safely use a memtx snapshot
      iterator. Currently, we do that in checkpoint related callbacks, but if
      we want to reuse snapshot iterators for other purposes, e.g. feeding
      a read view to a newly joined replica, we better hide this code behind
      snapshot iterator constructors.
      6136f84f
    • Vladimir Davydov's avatar
      memtx: use ref counting to pin indexes for snapshot · 82331221
      Vladimir Davydov authored
      Currently, to prevent an index from going away while it is being
      written to a snapshot, we postpone memtx_gc_task's free() invocation
      until checkpointing is complete, see commit 94de0a08 ("Don't take
      schema lock for checkpointing"). This works fine, but makes it rather
      difficult to reuse snapshot iterators for other purposes, e.g. feeding
      a consistent read view to a newly joined replica.
      
      Let's instead use index reference counting for pinning indexes for
      checkpointing. A reference is taken in a snapshot iterator constructor
      and released when the snapshot iterator is destroyed.
      82331221
    • Vladimir Davydov's avatar
      vinyl: get rid of vy_env::join_lsn · 96a7ae06
      Vladimir Davydov authored
      This fake LSN counter, which is used for assigning LSNs to Vinyl
      statements during the initial join stage, was introduced a long time
      ago, when LSNs were used as identifiers for lsregion allocations and
      hence were supposed to grow strictly monotonically with each new
      transaction. Later on, they were reused for assigning unique LSNs to
      identify indexes in vylog.
      
      These days, we don't need initial join LSNs to be unique, as we switched
      to generations for lsregion allocations while in vylog we now use LSNs
      only as an incarnation counter, not as a unique identifier. That said,
      let's zap vy_env::join_lsn and simply assign 0 to all statements
      received during the initial join stage.
      
      To achieve that, we just need to relax an assertion in vy_tx_commit()
      and remove the assumption that an LSN can't be zero in the write
      iterator implementation.
      96a7ae06
    • Vladimir Davydov's avatar
      vinyl: don't pin index for iterator lifetime · 02da82ea
      Vladimir Davydov authored
      vinyl_iterator keeps a reference to the LSM tree it was created for
      until it is destroyed, which may take indefinitely long in case the
      iterator is used in Lua. Actually, we don't need to keep a reference to
      the index for the whole iterator lifetime, because iterator_next()
      wrapper guarantees that iterator->next won't be called for a dropped
      index. What we need to do is keep a reference while we are yielding on
      disk read, similarly to vinyl_index_get().
      
      Currently, pinning an index for indefinitely long is harmless, because
      an LSM tree is exempted from dump/compaction as soon as it is dropped so
      we just pin some memory, that's all. However, following patches are
      going to enable dump/compaction for dropped but pinned indexes in order
      to implement snapshot iterator so we better relax the dependency of an
      iterator on an index know.
      
      While we are at it, let's remove env and lsm members of vinyl_iterator
      struct: lsm can be accessed via vy_read_iterator embedded in the struct
      while env is only needed to access iterator_pool so we better store a
      pointer to the pool in vinyl_iterator instead.
      02da82ea
    • Kirill Shcherbatov's avatar
      sql: remove SQL_FUNC_SLOCHNG flag · c33f0804
      Kirill Shcherbatov authored
      The SQL_FUNC_SLOCHNG flag was useful for datetime function
      that are currently not supported. So it could be removed.
      
      Needed for #2200, #4113, #2233
      c33f0804
    • Kirill Shcherbatov's avatar
      sql: rename OP_Function to OP_BuiltinFunction · 2437b059
      Kirill Shcherbatov authored
      Renamed OP_Function opcode to OP_BuiltinFunction to introduce a
      new OP_Function operation in a new meaning: a new OP_Function
      would call Tarantool's function with new port-based API while
      legacy OP_BuiltinFunction is an efficient implementation of
      SQL Builtins functions.
      
      Needed for #2200, #4113, #2233
      2437b059
    • Kirill Shcherbatov's avatar
      sql: rework SQL_FUNC_COUNT flag semantics · 1d69f568
      Kirill Shcherbatov authored
      Tarantool's SQL engine generates a different VDBE bytecode
      for ..COUNT(*).. and ..COUNT(fieldname).. operations:
      the first one produces a lightweight OP_Count operation that uses
      native mechanism to report the count of record in index while
      the second one pessimistically opens a space read iterator and
      uses Count aggregate function.
      
      A helper routine is_simple_count decides whether such
      optimisation is correct. It used to use SQL_FUNC_COUNT flag to
      mark a dummy (non-functional) function entry with 0 arguments.
      This patch changes SQL_FUNC_COUNT semantics: now it is a marker
      of any COUNT function, while is_simple_count relies on count
      of arguments to distinguish aggregate and non-aggregate
      functions.
      
      Needed for #2200, #4113, #2233
      1d69f568
    • Kirill Shcherbatov's avatar
      sql: wrap all trim functions in dispatcher · 77e4c842
      Kirill Shcherbatov authored
      A new dispatcher function trim_func calls corresponding trim_
      function implementation in relation with number of argc - a count
      of arguments.
      
      This is an important step to get rid of function's name
      overloading required for replace FuncDef cache with Tarantool's
      function cache.
      
      Needed for #2200, #4113, #2233
      77e4c842
    • Kirill Shcherbatov's avatar
      sql: GREATEST, LEAST instead of MIN/MAX overload · a46b5200
      Kirill Shcherbatov authored
      This patch does two things: renames existing scalar min/max
      functions and reserves names for them in NoSQL cache.
      
      Moreover it is an important step to get rid of function's name
      overloading required for replace FuncDef cache with Tarantool's
      function cache.
      
      Closes #4405
      Needed for #2200, #4113, #2233
      
      @TarantoolBot document
      Title: Scalar functions MIN/MAX are renamed to LEAST/GREATEST
      
      The MIN/MAX functions are typically used only as aggregate
      functions in other RDBMS(MSSQL, Postgress, MySQL, Oracle) while
      Tarantool's SQLite legacy code use them also in meaning
      GREATEST/LEAST scalar function. Now it fixed.
      a46b5200
    • Kirill Shcherbatov's avatar
      sql: remove SQL_PreferBuiltin flag · 79f3bf4b
      Kirill Shcherbatov authored
      The SQL_PreferBuiltin flag is redundant (because builtin names
      are forbidden for UDFs) so we may to remove it.
      
      Needed for #4113, #2200, #2233
      79f3bf4b
    • Kirill Shcherbatov's avatar
      sql: improve vdbe_field_ref fetcher · 72279c1c
      Kirill Shcherbatov authored
      Vdbe field ref is a dynamic index over tuple fields storing
      offsets to each field and filling the offset array on demand.
      It is highly used in SQL, because it strongly relies on fast and
      repetitive access to any field, not only indexed.
      
      There is an optimisation for the case when a requested field
      fieldno is indexed, and the tuple itself stores offset to the
      field in its own small field map, used by indexes. vdbe_field_ref
      then uses that map to retrieve offset value without decoding
      anything. But when SQL requests any field > fieldno, then
      vdbe_field_ref decodes the tuple from the beginning in the worst
      case. Even having previously accessed fieldno. But it could
      start decoding from the latter.
      
      An updated vdbe_field_ref fetcher class uses a bitmask of
      initialized slots to use pre-calculated offsets when possible.
      This speed-ups SQL in some corner case and doesn't damage
      performance in general scenarios.
      
      Closes #4267
      72279c1c
  4. Aug 19, 2019
  5. Aug 16, 2019
    • Konstantin Osipov's avatar
      gc: randomie the next checkpoint time also after a manual box.snapshot(). · 6277f48a
      Konstantin Osipov authored
      Before this patch, snapshot interval was set randomly within
      checkpoint_interval period. However, after box.snapshot(), the next
      snapshot was scheduled exactly checkpoint_interval from the current time.
      Many orchestration scripts snapshot entire cluster right after deployment,
      to take a backup. This kills randomness, since all instances begin to
      count the next checkpoint time from the current time.
      
      Randomize the next checkpoint time after a manual snapshot as well.
      
      Fixes gh-4432
      6277f48a
    • Alexander Turenko's avatar
      test: update test-run · 05fb6faa
      Alexander Turenko authored
      pretest_clean: preserve GREATEST and LEAST built-in functions.
      
      Needed for #4405.
      05fb6faa
    • Roman Khabibov's avatar
      sql: remove mask from struct Keyword · 13ed1126
      Roman Khabibov authored
      Originally, mask in struct Keyword served to reduce set of reserved
      keywords for build-dependent features. For instance, it was allowed to
      disable triggers as a compilation option, and in this case TRIGGER
      wouldn't be reserved word. Nowadays, our build always comprises all
      features, so there's no need in this option anymore. Hence, we can
      remove mask alongside with related to it macros.
      
      Closes #4155
      13ed1126
    • Nikita Pettik's avatar
      Hotfix for b7d595ac · 0894bec2
      Nikita Pettik authored
      It was forgotten to update result file of sql/bind.test.lua
      in previous patch. Let's fix that and refresh sql/bind.result with
      up-to-date results.
      0894bec2
  6. Aug 15, 2019
  7. Aug 14, 2019
    • Vladimir Davydov's avatar
      wal: make wal_sync fail on write error · 2d5e56ff
      Vladimir Davydov authored
      wal_sync() simply flushes the tx<->wal request queue, it doesn't
      guarantee that all pending writes are successfully committed to disk.
      This works for now, but in order to implement replica join off the
      current read view, we need to make sure that all pending writes have
      been persisted and won't be rolled back before we can use memtx
      snapshot iterators. So this patch adds a return code to wal_sync():
      since now on it returns -1 if rollback is in progress and hence
      some in-memory changes are going to be rolled back. We will use
      this method after opening memtx snapshot iterators used for feeding
      a consistent read view a newly joined replica so as to ensure that
      changes frozen by the iterators have made it to the disk.
      2d5e56ff
    • Alexander V. Tikhonov's avatar
      test: app/socket flaky fails at 1118 line · 952d8d1d
      Alexander V. Tikhonov authored
      Found that on high loaded hosts the test flaky fails at:
      
      [004] --- app/socket.result	Mon Jul 15 07:18:57 2019
      [004] +++ app/socket.reject	Tue Jul 16 16:37:35 2019
      [004] @@ -1118,7 +1118,7 @@
      [004]  ...
      [004]  ch:get(1)
      [004]  ---
      [004] -- true
      [004] +- null
      [004]  ...
      [004]  s:error()
      [004]  ---
      
      Found that the test in previous was used for testing the
      the channel get() function timeout and the error occurred
      on it, but later the checking error changed to:
      "builtin/socket.lua: attempt to use closed socket" and the
      test became not correct. Because for now it passes when the
      socket read function runs before the socket closing, but in
      this way read call doesn't wait. In the other way on high
      loaded hosts the close call may occure before read call and
      in this way read call halts and socket get call returns
      'null'. As seen both ways are not correct to check the error.
      Decided to remove this subtest.
      
      Check commit ba7a4fee ("Add tests for socket:close closes #360")
      
      Fixes #4354
      952d8d1d
    • Vladimir Davydov's avatar
      xrow: factor out helper for setting REPLACE request body · e687cacd
      Vladimir Davydov authored
      We will reuse it to relay a snapshot to a newly joined replica.
      e687cacd
    • Vladimir Davydov's avatar
      memtx: allow snapshot iterator to fail · aef84078
      Vladimir Davydov authored
      Memtx iterators never fail, that's why the snapshot iterator interface
      doesn't support failures. However, once we introduce snapshot iterator
      support for vinyl, we will need a way to handle errors in the API.
      aef84078
    • Vladimir Davydov's avatar
      memtx: don't store pointers to index internals in iterator · a22f2219
      Vladimir Davydov authored
      It's pointless as we can always access the index via iterator->index.
      a22f2219
    • Vladimir Davydov's avatar
      vinyl: move reference counting from vy_lsm to index · 7e11dd4f
      Vladimir Davydov authored
      Now, as vy_lsm and index are basically the same object, we can implement
      reference counting right in struct index. This will allow us to prevent
      an index from destruction when a space object it belongs to is freed
      anywhere in the code, not just in vinyl.
      7e11dd4f
    • Vladimir Davydov's avatar
      vinyl: embed index in vy_lsm · 576611eb
      Vladimir Davydov authored
      There's no point in having vinyl_engine and vinyl_index wrapper structs
      to bind vy_env and vy_lsm to struct engine and index. Instead we can
      simply embed engine and index in vy_env and vy_lsm. This will simplify
      further development, e.g. this will allow us to move reference counting
      from vy_lsm up to struct index so that it can be used in the generic
      code.
      576611eb
    • Vladimir Davydov's avatar
      vinyl: embed engine in vy_env · a771c2e5
      Vladimir Davydov authored
      There's no point in having vinyl_engine and vinyl_index wrapper structs
      to bind vy_env and vy_lsm to struct engine and index. Instead we can
      simply embed engine and index in vy_env and vy_lsm. This will simplify
      further development, e.g. this will allow us to move reference counting
      from vy_lsm up to struct index so that it can be used in the generic
      code.
      a771c2e5
    • Serge Petrenko's avatar
      decimal: add modulo operator · 8ea7106a
      Serge Petrenko authored
      Part of #4403
      
      @TarantoolBol document
      Title: Document decimal modulo operator
      
      There is now a modulo operator for decimal numbers:
      ```
      a = decimal.new(172.51)
      a % 1
      ---
      - '0.51'
      ...
      a % 0.3
      ---
      - '0.01'
      ...
      a % 13.27
      ---
      - '0.00'
      ...
      a % 173
      ---
      - '172.51'
      ...
      a % 72
      ---
      - '28.51'
      ...
      720 % a
      ---
      - '29.96'
      ...
      ```
      8ea7106a
Loading