Skip to content
Snippets Groups Projects
  1. Aug 02, 2018
  2. Aug 01, 2018
    • Vladimir Davydov's avatar
      txn: add helper to detect transaction boundaries · 13acfe47
      Vladimir Davydov authored
      Add txn_is_first_statement() function, which returns true if this is the
      first statement of the transaction. The function is supposed to be used
      from on_replace trigger to detect transaction boundaries.
      
      Needed for #2129
      13acfe47
    • Vladimir Davydov's avatar
      vinyl: rename vy_task::status to is_failed · 21eed04c
      Vladimir Davydov authored
      vy_task::status stores the return code of the ->execute method. There
      are only two codes in use: 0 - success and -1 - failure. So let's chage
      this to a boolean flag.
      21eed04c
    • Vladimir Davydov's avatar
      vinyl: zap vy_scheduler::is_worker_pool_running · d77b4dc9
      Vladimir Davydov authored
      This flag is set iff worker_pool != NULL hence it is pointless.
      d77b4dc9
    • Vladimir Davydov's avatar
      vinyl: use cbus for communication between scheduler and worker threads · f4625e64
      Vladimir Davydov authored
      We need cbus for forwarding deferred DELETE statements generated in a
      worker thread during primary index compaction to the tx thread where
      they can be inserted into secondary indexes. Since pthread mutex/cond
      and cbus are incompatible by their nature, let's rework communication
      channel between the tx and worker threads using cbus.
      
      Needed for #2129
      f4625e64
    • Vladimir Davydov's avatar
      vinyl: rename some members of vy_scheduler and vy_task struct · 46f50aad
      Vladimir Davydov authored
      I'm planning to add some new members and remove some old members from
      those structs. For this to play nicely, let's do some renames:
      
        vy_scheduler::workers_available => idle_worker_count
        vy_scheduler::input_queue       => pending_tasks
        vy_scheduler::output_queue      => processed_tasks
        vy_task::link                   => in_pending, in_processed
      46f50aad
    • Vladimir Davydov's avatar
      vinyl: store pointer to scheduler in struct vy_task · 1331d232
      Vladimir Davydov authored
      Currently, we don't really need it, but once we switch communication
      channel between the scheduler and workers from pthread mutex/cond to
      cbus (needed for #2129), tasks won't be completed on behalf of the
      scheduler fiber and hence we will need a back pointer from vy_task to
      vy_scheduler.
      
      Needed for #2129
      1331d232
    • Vladimir Davydov's avatar
      vinyl: do not free pending tasks on shutdown · 15c28b75
      Vladimir Davydov authored
      This is a prerequisite for switching scheduler-worker communication from
      pthread mutex/cond to cbus, which in turn is needed to generate and send
      deferred DELETEs from workers back to tx (#2129).
      
      After this patch, pending tasks will be leaked on shutdown. This is OK,
      as we leak a lot of objects on shutdown anyway. The proper way of fixing
      this leak would be to rework shutdown without atexit() so that we can
      use cbus till the very end.
      
      Needed for #2129
      15c28b75
    • Vladimir Davydov's avatar
      vinyl: store full tuples in secondary index cache · 0c5e6cc8
      Vladimir Davydov authored
      Currently, both vy_read_iterator_next() and vy_point_lookup() add the
      returned tuple to the tuple cache. As a result, we store partial tuples
      in a secondary index tuple cache although we could store full tuples
      (we have to retrieve them anyway when reading a secondary index). This
      means wasting memory. Besides, when the #2129 gets implemented, there
      will be tuples in a secondary index that have to be skipped as they have
      been overwritten in the primary index. Caching them would be inefficient
      and error prone. So let's call vy_cache_add() from the upper level and
      add only full tuples to the cache.
      
      Closes #3478
      Needed for #2129
      0c5e6cc8
  3. Jul 31, 2018
    • Vladimir Davydov's avatar
      vinyl: refactor unique check · 85608344
      Vladimir Davydov authored
      For the sake of further patches, let's do some refactoring:
       - Rename vy_check_is_unique to vy_check_is_unique_primary and use it
         only for checking the unique constraint of primary indexes. Also,
         make it return immediately if the primary index doesn't need
         uniqueness check, like vy_check_is_unique_secondary does.
       - Open-code uniqueness check in vy_check_is_unique_secondary instead of
         using vy_check_is_unique.
       - Reduce indentation level of vy_check_is_unique_secondary by inverting
         the if statement.
      85608344
    • Vladimir Davydov's avatar
      vinyl: fold vy_delete_impl · f88a0bd1
      Vladimir Davydov authored
      vy_delete_impl helper is only used once in vy_delete and it is rather
      small so inlining it definitely won't hurt. On the contrary, it will
      consolidate DELETE logic in one place, making the code easier to follow.
      f88a0bd1
    • Vladimir Davydov's avatar
      vinyl: fold vy_replace_one and vy_replace_impl · 1dfeb601
      Vladimir Davydov authored
      There's no point in separating REPLACE path between the cases when
      the space has secondary indexes and when it only has the primary
      index, because they are quite similar. Let's fold vy_replace_one
      and vy_replace_impl into vy_replace to remove code duplication.
      1dfeb601
    • Vladimir Davydov's avatar
      vinyl: always get full tuple from pk after reading from secondary index · 5ceca76c
      Vladimir Davydov authored
      Currently, we don't always need a full tuple. Sometimes (e.g. for
      checking uniqueness constraint), a partial tuple read from a secondary
      index is enough. So we have vy_lsm_get() which reads a partial tuple
      from an index. However, once the optimization described in #2129 is
      implemented, it might happen that a tuple read from a secondary index
      was overwritten or deleted in the primary index, but DELETE statement
      hasn't been propagated to the secondary index yet, i.e. we will have to
      read the primary index anyway, even if we don't need a full tuple.
      
      That said, let us:
      
       - Make vy_lsm_get() always fetch a full tuple, even for secondary
         indexes, and rename it to vy_get().
      
       - Rewrite vy_lsm_full_by_key() as a wrapper around vy_get() and rename
         it to vy_get_by_raw_key().
      
       - Introduce vy_get_by_secondary_tuple() which gets a full tuple given a
         tuple read from a secondary index. For now, it's basically a call to
         vy_point_lookup(), but it'll become a bit more complex once #2129 is
         implemented.
      
       - Prepare vy_get() for the fact that a tuple read from a secondary
         index may be absent in the primary index, in which case it should
         try the next matching one.
      
      Needed for #2129
      5ceca76c
    • Vladimir Davydov's avatar
      vinyl: simplify vy_squash_process · 128503ea
      Vladimir Davydov authored
      Since vy_point_lookup() now guarantees that it returns the newest
      tuple version, we can remove the code that squashes UPSERTs from
      vy_squash_process().
      128503ea
    • Vladimir Davydov's avatar
      vinyl: make point lookup always return the latest tuple version · 6d85c35c
      Vladimir Davydov authored
      Currently, vy_point_lookup(), in contrast to vy_read_iterator, doesn't
      rescan the memory level after reading disk, so if the caller doesn't
      track the key before calling this function, the caller won't be sent to
      a read view in case the key gets updated during yield and hence will
      be returned a stale tuple. This is OK now, because we always track the
      key before calling vy_point_lookup(), either in the primary or in a
      secondary index. However, for #2129 we need it to always return the
      latest tuple version, no matter if the key is tracked or not.
      
      The point is in the scope of #2129 we won't write DELETE statements to
      secondary indexes corresponding to a tuple replaced in the primary
      index. Instead after reading a tuple from a secondary index we will
      check whether it matches the tuple corresponding to it in the primary
      index: if it is not, it means that the tuple read from the secondary
      index was overwritten and should be skipped. E.g. suppose we have the
      primary index over the first field and a secondary index over the second
      field and the following statements in the space:
      
        REPLACE{1, 10}
        REPLACE{1, 20}
      
      Then reading {10} from the secondary index will return REPLACE{1, 10}, but
      lookup of {1} in the primary index will return REPLACE{1, 20} which
      doesn't match REPLACE{1, 10} read from the secondary index hence the
      latter was overwritten and should be skipped.
      
      The problem is in the example above we don't want to track key {1} in
      the primary index before lookup, because we don't actually read its
      value. So for the check to work correctly, we need the point lookup to
      guarantee that the returned tuple is always the newest one. It's fairly
      easy to do - we just need to rescan the memory level after yielding on
      disk if its version changed.
      
      Needed for #2129
      6d85c35c
    • Konstantin Nazarov's avatar
      Add tarantoolctl rocks pack/unpack subcommands · 0746fdb4
      Konstantin Nazarov authored
      The subcommands are used to create binary rock distributions.
      In context of #3525
      0746fdb4
    • Kirill Yukhin's avatar
      sql: remove pragma schema_version · f2debd8b
      Kirill Yukhin authored
      This pragma is dead and produces nothing else but segfault.
      Along w/ this pragma, remove now dead opcodes which set/read
      schema_version and all related routines.
      Also, improve opcode generation script.
      
      Part of #3541
      f2debd8b
    • Kirill Shcherbatov's avatar
      sql: prevent executing cross-engine sql · 7324a1cf
      Kirill Shcherbatov authored
      Some sql requests are complex and could contain R/W with
      multiple spaces. As we have no ability to make such changes
      transactionaly, we have to disallow such requests.
      Since now iterators in SQL start transaction so we cant prevent
      such vicious and dangerous things.
      
      Closes #3551
      7324a1cf
    • Kirill Yukhin's avatar
      Merge branch '1.10' into 2.0 · 13df2b1f
      Kirill Yukhin authored
      13df2b1f
    • Mergen Imeev's avatar
      sql: UPPER and LOWER support COLLATE · 0ecdb0f5
      Mergen Imeev authored
      SQL functions UPPER and LOWER now works
      with COLLATE as they should according to
      ANSI Standart.
      
      Closes #3052.
      0ecdb0f5
  4. Jul 30, 2018
    • Vladimir Davydov's avatar
      vinyl: implement rebootstrap support · 06658416
      Vladimir Davydov authored
      If vy_log_bootstrap() finds a vylog file in the vinyl directory, it
      assumes it has to be rebootstrapped and calls vy_log_rebootstrap().
      The latter scans the old vylog file to find the max vinyl object id,
      from which it will start numbering objects created during rebootstrap to
      avoid conflicts with old objects, then it writes VY_LOG_REBOOTSTRAP
      record to the old vylog to denote the beginning of a rebootstrap
      section. After that initial join proceeds as usual, writing information
      about new objects to the old vylog file after VY_LOG_REBOOTSTRAP marker.
      Upon successful rebootstrap completion, checkpoint, which is always
      called right after bootstrap, rotates the old vylog and marks all
      objects created before the VY_LOG_REBOOTSTRAP marker as dropped in the
      new vylog. The old objects will be purged by the garbage collector as
      usual.
      
      In case rebootstrap fails and checkpoint never happens, local recovery
      writes VY_LOG_ABORT_REBOOTSTRAP record to the vylog. This marker
      indicates that the rebootstrap attempt failed and all objects created
      during rebootstrap should be discarded. They will be purged by the
      garbage collector on checkpoint. Thus even if rebootstrap fails, it is
      possible to recover the database to the state that existed right before
      a failed rebootstrap attempt.
      
      Closes #461
      06658416
    • Vladimir Davydov's avatar
      vinyl: simplify vylog recovery from backup · 8e710090
      Vladimir Davydov authored
      Since we don't create snapshot files for vylog, but instead append
      records written after checkpoint to the same file, we have to use the
      previous vylog file for backup (see vy_log_backup_path()). So when
      recovering from a backup we need to rotate the last vylog to keep vylog
      and checkpoint signatures in sync. Currently, we do it on recovery
      completion and we use vy_log_create() instead of vy_log_rotate() for it.
      This is done so that we can reuse the context that was used for recovery
      instead of rereading vylog for rotation. Actually, there's no point in
      this micro-optimization, because we rotate vylog only when recovering
      from a backup. Let's remove it and use vy_log_rotate() for this.
      
      Needed for #461
      8e710090
    • Vladimir Davydov's avatar
      replication: print master uuid when (re)bootstrapping · 71cec841
      Vladimir Davydov authored
      Currently only the remote address is printed. Let's also print the UUID,
      because replicas are identified by UUID everywhere in tarantool, not by
      the address. An example of the output is below:
      
        I> can't follow eb81a67e-99ee-40bb-8601-99b03fa20124 at [::1]:58083: required {1: 8} available {1: 12}
        C> replica is too old, initiating rebootstrap
        I> bootstrapping replica from eb81a67e-99ee-40bb-8601-99b03fa20124 at [::1]:58083
      
        I> can't follow eb81a67e-99ee-40bb-8601-99b03fa20124 at [::1]:58083: required {1: 17, 2: 1} available {1: 20}
        I> can't rebootstrap from eb81a67e-99ee-40bb-8601-99b03fa20124 at [::1]:58083: replica has local rows: local {1: 17, 2: 1} remote {1: 23}
        I> recovery start
      
      Suggested by @kostja.
      
      Follow-up ea69a0cd ("replication: rebootstrap instance on startup
      if it fell behind").
      71cec841
    • Vladimir Davydov's avatar
      vinyl: zap tx_manager_vlsn · 5a772639
      Vladimir Davydov authored
      This function is not used anywhere since commit a1e005d8
      ("vinyl: write_iterator merges vlsns subsequnces")
      5a772639
  5. Jul 27, 2018
    • Kirill Yukhin's avatar
      sql: remove preupdate hook · 5d03ba58
      Kirill Yukhin authored
      SQLITE_ENABLE_PREUPDATE_HOOK is dead macro.
      Remove it.
      
      Part of #2356
      5d03ba58
    • Kirill Shcherbatov's avatar
      sql: introduce TRUNCATE TABLE operation · dbe38b0d
      Kirill Shcherbatov authored
      To implement new TRUNCATE operation, we have introduced a
      new P2 argument for OP_Clear opcode that calles box_truncate
      instead of tarantoolSqlite3ClearTable.
      This operation should work faster than DELETE FROM; but have
      a few restricts.
      
      Closes #2201.
      
      @TarantoolBot document
      Title: New TRUNCATE operation
      TRUNCATE is DDL operation.
      Removes all rows from a table or specified partitions of a table,
      without logging the individual row deletions.
      TRUNCATE TABLE is similar to the DELETE statement with no WHERE
      clause; however, TRUNCATE TABLE is faster and uses fewer system
      resources.
      It couldn't be used with system tables or with tables having FKs.
      It also couldn't be called in transaction.
      The triggers on table will have ignored.
      Example:
      TRUNCATE TABLE t1;
      dbe38b0d
  6. Jul 26, 2018
Loading