Skip to content
Snippets Groups Projects
  1. Apr 09, 2018
  2. Apr 07, 2018
    • Vladimir Davydov's avatar
      Merge branch '1.10' into 2.0 · a4b8b64d
      Vladimir Davydov authored
      a4b8b64d
    • Vladimir Davydov's avatar
      Merge branch '1.9' into 1.10 · e7543883
      Vladimir Davydov authored
      e7543883
    • Vladimir Davydov's avatar
      vinyl: fix crash if index is dropped while read task is in progress · 2a7cf7f5
      Vladimir Davydov authored
      If a fiber waiting for a read task to complete is cancelled, it will
      leave the read iterator immediately, leaving the read task pending.
      If the index is dropped before the read task is complete, the task
      will attempt to dereference a deleted run upon completion:
      
          0  0x560b4007dbbc in print_backtrace+9
          1  0x560b3ff80a1d in _ZL12sig_fatal_cbiP9siginfo_tPv+1e7
          2  0x7f52b09190c0 in __restore_rt+0
          3  0x7f52af6ea30a in bzero+5a
          4  0x560b3ffc7a99 in mempool_free+2a
          5  0x560b3ffcaeb7 in vy_page_read_cb_free+47
          6  0x560b400806a2 in cbus_call_done+3f
          7  0x560b400805ea in cmsg_deliver+30
          8  0x560b40080e4b in cbus_process+51
          9  0x560b4003046b in _ZL10tx_prio_cbP7ev_loopP10ev_watcheri+2b
          10 0x560b4023d86e in ev_invoke_pending+ca
          11 0x560b4023e772 in ev_run+5a0
          12 0x560b3ff822dc in main+5ed
          13 0x7f52af6862b1 in __libc_start_main+f1
          14 0x560b3ff801da in _start+2a
          15 (nil) in +2a
      
      Fix this by elevating the run reference counter per each read task.
      
      Note, currently we use vy_run::refs not only as a reference counter, but
      also as a counter of slices created for the run - see how we compare it
      to vy_run::compacted_slice_count in vy_task_compact_complete(). This
      isn't going to work anymore, obviously. Now we need to count slices
      created per each run in a separate counter, vy_run::slice_count. Anyway,
      it was a rather dubious hack to abuse reference counter for counting
      slices and it's good to finally get rid of it.
      2a7cf7f5
    • Vladimir Davydov's avatar
      vinyl: use ERRINJ_DOUBLE for ERRINJ_VY_READ_PAGE_TIMEOUT · 8dc9895f
      Vladimir Davydov authored
      We use ERRINJ_DOUBLE for all other timeout injections. This makes them
      more flexible as we can inject an arbitrary timeout in tests, not just
      enable some hard-coded timeout. Besides, it makes tests easier to
      follow. So let's use ERRINJ_DOUBLE for ERRINJ_VY_READ_PAGE_TIMEOUT too.
      8dc9895f
    • Vladimir Davydov's avatar
      alter: do not crash if sequence is created for space with no indexes · 95aefec3
      Vladimir Davydov authored
      If a space has no indexes, index_find() will return NULL, which will be
      happily dereferenced by on_replace_dd_sequence(). Looks like this bug
      goes back to the time when we made index_find() exception-free and
      introduced index_find_xc() wrapper. Fix it and add a test case.
      95aefec3
  3. Apr 06, 2018
    • Vladimir Davydov's avatar
      alter: zap space_vtab::commit_alter · 9ec3b1a4
      Vladimir Davydov authored
      space_vtab::commit_alter is implemented only by memtx, which uses it
      to set bsize for a new space. However, it isn't necessary to use
      commit_alter for this - instead we can set bsize in prepare_alter and
      reset it in drop_primary_key, along with memtx_space::replace. Let's
      do it and zap space_vtab::commit_alter altogether, because the callback
      is confusing: judging by its name it should be called after WAL write,
      but it is called before.
      9ec3b1a4
    • Vladimir Davydov's avatar
      memtx: do not use space_vtab::commit_alter for freeing tuples · 3bb5a4b1
      Vladimir Davydov authored
      When the last index of a memtx space is dropped, we need to delete all
      tuples stored in the space. We do it in space_vtab::commit_alter, but
      this is wrong, because this function is not rolled back and so we may
      get use-after-free error if we fail to write a DDL operation to WAL.
      To avoid that, let's delete tuples in index_vtab::commit_drop, which is
      called after WAL write.
      
      There's a nuance here: index_vtab::commit_drop is called if an index is
      rebuilt (because essentially it is drop + create) so we must elevate the
      reference counter of every tuple added to the new index during rebuild
      and, respectively, drop all the references in index_vtab::abort_create,
      which is called if index creation is aborted for some reason. This also
      means that now we iterate over all tuples twice when a primary key is
      rebuilt - first to build the new index, then to unreference all tuples
      stored in the old index. This is OK as we can make the last step
      asynchronous, which will also speed up the more common case of space
      drop.
      
      Closes #3289
      3bb5a4b1
    • Vladislav Shpilevoy's avatar
      Merge branch '1.10' into 2.0 · da15f705
      Vladislav Shpilevoy authored
      da15f705
    • Vladislav Shpilevoy's avatar
      netbox: don't cancel pending requests on schema change · 62ba7ba7
      Vladislav Shpilevoy authored
      When a schema version change is detected, there is no reason to
      cancel and retry already sent requests. They can be already
      executed on a server, and their retrying leads to multiple
      execution.
      
      A request must be retried only if a server responded with
      WRONG_SCHEMA_VERSION error exactly to this request.
      
      Closes #3325
      62ba7ba7
    • Vladislav Shpilevoy's avatar
      netbox: replace first_value(...) with (...) · 8576a68a
      Vladislav Shpilevoy authored
      In Lua to get the only first value in a sequence it is
      enough to place it into ().
      
      Follow up #3323, #3322
      8576a68a
    • Konstantin Osipov's avatar
      Merge branch '1.10' into 2.0 · 6facef72
      Konstantin Osipov authored
      6facef72
    • Konstantin Osipov's avatar
      rfc: add rfc template · 51411a7a
      Konstantin Osipov authored
      Introducing a formal, trackable process for server enhancement.
      
      Before working on a complex feature, please write an RFC document,
      describing what and how you see changed, and get it approved.
      
      All historical RFCs are kept in doc/rfc.
      51411a7a
  4. Apr 05, 2018
    • Vladimir Davydov's avatar
      vinyl: do not use space_vtab::commit_alter for preparing new indexes · 48a47600
      Vladimir Davydov authored
      Currently, space_vtab::commit_alter is called before WAL write so we can
      use it for preparing new indexes in vinyl. However, this is going to
      change soon, because actually space_vtab::commit_alter should be called
      after WAL write, like index_vtab::commit_drop or commit_create. Calling
      it before WAL write may result in use-after-free in memtx (see #3289).
      Besides, using this function for iterating over space indexes just feels
      wrong, as we have index methods invoked by AlterSpaceOp descendants for
      this.
      
      So let's move all the operations performed by vinyl_space_commit_alter
      somewhere else. Fortunately, it's easy to do without damaging code
      readability or efficiency:
      
       - Update of vy_lsm::pk can be done by vinyl_space_swap_index and
         vinyl_build_secondary_key.
      
       - vy_lsm->check_is_unique can be computed by vinyl_engine_create_space
         and then set by vinyl_space_swap_index.
      
       - Call to key_def_update_optionality is implied by key_def_swap, which
         is already called by vinyl_space_swap_index, hence it can be removed.
      
      Needed for #3289
      48a47600
    • Vladimir Davydov's avatar
      memtx: don't call begin_buid and end_build for new pk after recovery · 30db003a
      Vladimir Davydov authored
      Basically, index_begin_build() followed by index_end_build() is a no-op.
      There's absolutely no point in calling it for primary indexes after
      initial recovery has completed.
      30db003a
    • Vladislav Shpilevoy's avatar
      netbox: reuse _request() to do SQL execute() · 9401c4f9
      Vladislav Shpilevoy authored
      _request() is a wrapper for perform_request, that detects schema
      changes, and waits until it is reloaded. Lets use _request()
      instead of direct perform_request() for execute(). The reason why
      the _request() was not used earlier was my attempt to avoid
      multiple return values in _request(), that leads to minor fixes
      in non-execute methods like index.select or eval/call_16, which
      return the _request() directly.
      
      But diplicating schema reloading logic for execute() is worse.
      
      Closes #3323
      Closes #3322
      9401c4f9
    • Vladislav Shpilevoy's avatar
      netbox: forbid conn:timeout():execute(...) · 3874ab58
      Vladislav Shpilevoy authored
      This API is deprecated in 1.7.4, so it must not be
      supported in new methods like execute().
      3874ab58
    • Vladimir Davydov's avatar
      vinyl: use disk_format in vy_run_rebuild_index · 828a33bf
      Vladimir Davydov authored
      We read tuples from disk hence we should use disk_format, not
      mem_format. Fix it. While we are at it, let's also update the
      outdated comment to vy_run_rebuild_index.
      828a33bf
    • Vladimir Davydov's avatar
      memtx: rtree: remove pointless index_vtab::begin_build implementation · 50fcc202
      Vladimir Davydov authored
      The rtree is empty when this function is called (in fact, it is called
      right after creating the index), there's no need to purge it.
      50fcc202
    • Vladimir Davydov's avatar
      vinyl: allow to modify key definition if it does not require rebuild · a0299db8
      Vladimir Davydov authored
      To allow extending key definition for non-empty vinyl spaces, this patch
      performs the following steps:
      
       - Revert commit c31dd19a ("vinyl: forbid vinyl index key definition
         alter") that forbade any key def alter. It isn't needed anymore.
      
       - Update key_def and cmp_def in vinyl_space_swap_index(). We simply
         swap the definitions between the old and new indexes in memory.
         Since all vinyl objects reference either vy_lsm::key_def or
         vy_lsm::cmp_def or both, and the change is compatible (does not
         change the order for existing tuples), this should work just fine.
      
       - Update key definition in vylog on ALTER. For this, we introduce a new
         vylog record type, VY_LOG_MODIFY_LSM, which updates key definition.
         To be able to replay it on recovery in case we failed to flush it
         before restart, we also store the LSN of the WAL record that
         triggered the ALTER.
      
      It also adds the following test cases:
      
       - Modify key definition of primary and secondary indexes of a non-empty
         space (engine/ddl).
      
       - Modify key definition before snapshot and relay it to a newly joined
         replica (engine/replica_join).
      
       - Make sure key definition is updated in vylog on ALTER (vinyl/layout).
      a0299db8
    • Vladimir Davydov's avatar
      vinyl: do not write VY_LOG_DUMP_LSM record to snapshot · 48e97cd3
      Vladimir Davydov authored
      There's no point in writing this record to snapshot, because we can
      store LSN of the last index dump right in VY_LOG_CREATE_LSM record.
      48e97cd3
    • Vladimir Davydov's avatar
      vinyl: rename vy_log_record::commit_lsn to create_lsn · aadd7901
      Vladimir Davydov authored
      So as to draw the line between LSN of index creation and LSN of last
      index modification, which is introduced by later in the series.
      aadd7901
    • Nikita Pettik's avatar
      sql: rework 'DROP INDEX' and 'DROP TABLE' handling · 41616a28
      Nikita Pettik authored
      As a part of SQL data dictionary integration, 'DROP INDEX' and
      'DROP TABLE' statements proccessing has been refactored in order
      to avoid using SQL specific internal structures.
      However, triggers still aren't transfered to server, so to drop them
      it is required to lookup SQL table and emit apporpriate opcodes.
      Also, added comments and fixed codestyle for functions processing
      'DROP' routine.
      
      Part of #3222.
      41616a28
    • Nikita Pettik's avatar
      sql: remove obsolete SQLite routine · 4e4f8529
      Nikita Pettik authored
      Some of legacy functions seem to be useless, since they serve as
      wrappers around others; the rest rely on capabilities which are no
      longer relevant. This patch provides slight refactoring of such
      functions or removes them at all.
      4e4f8529
    • Vladimir Davydov's avatar
      vinyl: use source tuple format when copying field map · d647f27a
      Vladimir Davydov authored
      There are two functions in vy_stmt.c that blindly copy tuple field map,
      vy_stmt_dup() and vy_stmt_replace_from_upsert(). Both these functions
      take a tuple format to use for the new statement and require this format
      to be the same as the source tuple format in terms of fields definition,
      otherwise they'll just crash. The only reason why we did that is that
      back when these functions were written we used a separate format for
      UPSERT statements so we needed this extra argument for creating a
      REPLACE from UPSERT. Now it's not needed, and we can use the source
      tuple format instead. Moreover, passing the current tuple format to any
      of those functions is even harmful, because tuple format can be extended
      by ALTER, in which case these functions will crash if called on a
      statement created before ALTER. That being said, let's drop the tuple
      format argument.
      d647f27a
    • Vladimir Davydov's avatar
      vinyl: remove pointless is_nullable initialization for disk_format · b2469121
      Vladimir Davydov authored
      space->format and cmp_def must be compatible, i.e. space->format has
      is_nullable flag set for a field iff it is set for all key parts
      indexing this field. Therefore there's no point to set is_nullable for
      disk_format as it must have been initialized by tuple_format_create().
      Remove the pointless loop.
      
      Also, while we are at it, fix the minor memory leak - disk_format is
      referenced twice for the primary key.
      b2469121
    • Vladimir Davydov's avatar
      vinyl: zap vy_mem_update_formats · 0ce7381f
      Vladimir Davydov authored
      A piece of code left from the inglorious past, which doesn't even have
      a forward declaration, let alone used anywhere. Remove it.
      0ce7381f
    • Vladimir Davydov's avatar
      vinyl: zap vy_lsm_validate_formats · fac96568
      Vladimir Davydov authored
      We allocate index formats in the only place, vy_lsm_new, so there's no
      point in this debug-only check anymore.
      fac96568
    • Vladimir Davydov's avatar
      vinyl: do not reallocate tuple formats on alter · d75c24a4
      Vladimir Davydov authored
      We create new formats for all indexes of the new space in
      vinyl_space_commit_alter() while we don't actually need to
      do this, because the new formats have already been created
      by vy_lsm_new() - all we need to do is reuse them somehow.
      
      This patch does the trick: it implements the swap_index()
      space virtual method for vinyl so that it swaps tuple formats
      between the old and new spaces.
      d75c24a4
    • Vladimir Davydov's avatar
      space: make space_swap_index method virtual · 1cb16edb
      Vladimir Davydov authored
      This function is called by MoveIndex and ModifyIndex ALTER operations,
      i.e. when the index definition is not changed at all or is extended.
      Making this method virtual will allow to avoid reallocation of vinyl
      formats in vinyl_space_commit_alter().
      1cb16edb
    • Vladimir Davydov's avatar
      alter: do not rebuild secondary indexes on compatible pk changes · 7e214255
      Vladimir Davydov authored
      If the new cmp_def of a secondary index is compatible with the old one
      after the primary key parts have changed, we don't need to rebuild it,
      we just need to update its definition.
      7e214255
    • Nikita Pettik's avatar
      sql: remove obsolete pragmas · 86aaf3e8
      Nikita Pettik authored
      Some pragmas turn out to be useless and can't be applied to Tarantool's SQL.
      Moreover, couple of them lead to assertion faults.
      List or removed pragmas:
      - synchronous
      - empty_result_callbacks
      - WAL_autocheckpoint
      - WAL_checkpoint
      - secure_delete
      - database_list
      - data_version
      - compile_options
      - application_id
      - user_version
      - pragmas connected with HAS_CODEC macro
      - pragmas connected with heap memory managment
      86aaf3e8
    • Vladimir Davydov's avatar
      alter: require rebuild of all secondary vinyl indexes if pk changes · d6d3e2c0
      Vladimir Davydov authored
      If the primary key is modified, we schedule rebuild of all non-unique
      (including nullable) secondary TREE indexes. This is valid for memtx,
      but is not quite right for vinyl. For vinyl we have to rebuild all
      secondary indexes, because they are all non-clustered (i.e. point to
      tuples via primary key parts). This doesn't result in any bugs for now,
      because rebuild of vinyl indexes is not supported, but hopefully this is
      going to change soon. So let's introduce a new virtual index method,
      index_vtab::depends_on_pk, which returns true iff the index needs to be
      updated if the primary key changes, and define this new method for vinyl
      and memtx TREE indexes.
      d6d3e2c0
    • vanyail's avatar
      sql: ban of REINDEX syntax · 73445865
      vanyail authored
      This change removes REINDEX from keywords and also disables tests
      which were using this syntax.
      
      This patch affects only parser, REINDEX implementation still exists,
      because it is planned to be fixed and enabled in the scope of #3195.
      
      Note that REINDEX keyword is still in '%fallback' directive in
      'parse.y'. Reserved keywords included there to avoid situation when
      a keyword has no usages within 'parse.y' file.
      
      Fixes #2174
      73445865
    • Vladimir Davydov's avatar
      index: add commit_modify virtual method · 28c31d69
      Vladimir Davydov authored
      The new method is called after successful update of index definition.
      It is passed the signature of the WAL record that committed the
      operation. It will be used by Vinyl to update key definition in vylog.
      28c31d69
    • Bulat Niatshin's avatar
      sql: fix non-working 'INDEXED BY' clause · e4537700
      Bulat Niatshin authored
      Fix non-working 'INDEXED BY' clause for SELECT statement,
      which caused syntax-error during execution.
      
      Closes #2996
      e4537700
    • Vladislav Shpilevoy's avatar
      sql: remove useless branching in insertOrReplace · cc5e57d9
      Vladislav Shpilevoy authored
      Type of an operation for struct request can be passed directly
      with no "proxying" by special codes.
      cc5e57d9
    • Vladislav Shpilevoy's avatar
      sql: simplify lua SQL executor · 04ab1b01
      Vladislav Shpilevoy authored
      1) Code is completely unreadable and complicated by strange
         optimizations;
      2) It still supposes, that there can be multiple statements in a
         single string;
      3) It pushes first letter of affinity together with column names
         - why? It is not used anywhere.
      
      Lets remove prepared statements list, affinity first letters
      pushing.
      04ab1b01
    • Vladislav Shpilevoy's avatar
      sql: light error codes refactoring · 4dcffa84
      Vladislav Shpilevoy authored
      Remove unused error codes, use enum instead of
      defines. Later the prefix SQLITE_ must be removed -
      see #3315.
      4dcffa84
    • Vladislav Shpilevoy's avatar
      sql: remove unused operands from OP_IdxInsert/Replace · 5668add4
      Vladislav Shpilevoy authored
      And rework sqlite3CompleteInsertion. Before the patch it
      * accepts actually unused pTab;
      * array of records to insert in different indexes, but inserts
        only one;
      * has sqlite code style.
      
      Code style fix is obvious. PTab is used only to check, that it is
      not a view and to check, that a first index is primary:
      1) caller functions already guarantee the pTab is not a view;
      2) regardless of a first index nature: primary or not - it is not
         used here. It is useless check. With the same success we can
         check this in each function, that uses struct Table.
      
      Array of records to insert makes no sense, since insertion is
      being done in a primary index only. It is enough to pass a
      register with a primary index tuple.
      5668add4
Loading