Skip to content
Snippets Groups Projects
  1. Jul 18, 2023
    • Dmitriy Nesterov's avatar
      test/fuzz: add grammar-based LuaJIT fuzzer · 87f4c930
      Dmitriy Nesterov authored
      Patch adds a LuaJIT fuzzer based on libprotobuf-mutator and LibFuzzer.
      Grammar is described via messages in protobuf format, serializer is
      applied to convert .proto format to string.
      
      For displaying generated code on the screen during fuzzing set
      the environment variable 'LPM_DUMP_NATIVE_INPUT'.
      
      For displaying error messages from lua functions set
      the environment variable 'LUA_FUZZER_VERBOSE'.
      
      Note: UndefinedBehaviourSanitizer is unsupported by LuaJIT (see #8473),
      so fuzzing test is disabled when CMake option ENABLE_UB_SANITIZER is
      passed.
      
      Closes #4823
      
      NO_DOC=<fuzzing testing of LuaJIT>
      NO_TEST=<fuzzing testing of LuaJIT>
      
      (cherry picked from commit a287c853)
      87f4c930
    • Dmitriy Nesterov's avatar
      cmake: add dependencies for LuaJIT and SQL fuzzers · 70469594
      Dmitriy Nesterov authored
      Added Google's 'libprotobuf-mutator' and 'protobuf' libraries
      for developing grammar-based LuaJIT and SQL fuzzers based on
      LibFuzzer.
      
      It is needed to build protobuf module from source because
      by default, the system-installed version of protobuf is used
      by libprotobuf-mutator, and this version can be too old.
      
      Part of #4823
      
      NO_CHANGELOG=<dependencies>
      NO_DOC=<dependencies>
      NO_TEST=<dependencies>
      
      (cherry picked from commit b11072a6)
      70469594
    • Dmitriy Nesterov's avatar
      test/fuzz: add options for better fuzzing · 85496d4a
      Dmitriy Nesterov authored
      Added options for fuzzing and for getting more information
      on debugging.
      
      NO_CHANGELOG=<fuzzing options>
      NO_DOC=<fuzzing options>
      NO_TEST=<fuzzing options>
      
      (cherry picked from commit 69f21e25)
      85496d4a
  2. Jul 14, 2023
    • Vladimir Davydov's avatar
      box: allow to truncate temp and local spaces in ro mode · 2fc54ba1
      Vladimir Davydov authored
      To achieve that, we bypass the read-only check for the _truncate system
      space in box_process1() and perform it in the on_replace system trigger
      instead, when we know which space is truncated.
      
      Note, we have to move the check for insertion of a new record into the
      _truncate system space before the read-only check in the on_replace
      trigger callback; this is needed for initial recovery with a non-empty
      _truncate space to work. While we are at it, let's use recovery_state to
      make the check explicit.
      
      Closes #5616
      
      @TarantoolBot document
      Title: Mention that temp and local spaces can be truncated in ro mode
      
      DML operations on temporary and local spaces can be performed even if
      the instance is in the read-only mode, but DDL operations (such as
      `alter`) are forbidden in this case[^1]. Technically, `truncate` is
      a DDL operation so initially it was forbidden as well. However, it
      should be safe to perform this operation on a temporary or local space
      because logically it only modifies the data stored in the space (like
      DML) and it isn't replicated (see tarantool/tarantool#4263). So starting
      from Tarantool 2.11.1 we allow users to truncate temporary spaces in the
      read-only mode.
      
      [^1]: https://www.tarantool.io/en/doc/latest/concepts/replication/repl_architecture/#replication-local
      
      (cherry picked from commit 054526ac)
      2fc54ba1
    • Vladimir Davydov's avatar
      vinyl: fix use-after-free in vy_read_iterator_next · 2a6a9b75
      Vladimir Davydov authored
      A read source iterator stores statements in a vy_history object using
      vy_history_append_stmt(). If a statement can be referenced, it's
      reference counter is incremented. If it can't, i.e. it belongs to a
      memory source, it's stored in a vy_history object without referencing.
      
      This works fine because memory sources are append-only. A problem arises
      only when we get to scanning disk sources. Since we yield while reading
      disk, a dump task may complete concurrently dropping the memory sources
      and possibly invalidating statements stored in the iterator history.
      Although we drop the history accumulated so far and restart the
      iteration from scratch in this case, there's still an issue that can
      result in a use-after-free bug in vy_read_iterator_next().
      
      The problem is that we access the current candidate for the next
      statement while evaluating a disk source after a disk read. If 'next'
      refers to a referenced statement, it's fine, but if it refers to a
      statement from a memory source, it may cause use-after-free because
      the memory source may be dropped during a disk read.
      
      To fix this issue, let's make vy_history_append_stmt() copy statements
      coming from memory sources. This should be fine performance-wise because
      we copied memory statements eventually in vy_history_apply() anyway,
      before returning them to the user.
      
      Note that we also have to update vy_read_iterator_restore_mem() because
      it implicitly relied on the fact that 'next' coming from a memory source
      can't be freed by vy_mem_iterator_restore(), which cleans up the memory
      source history. Now, it isn't true anymore so we have to temporarily
      take a reference to 'next' explicitly.
      
      Closes #8852
      
      NO_DOC=bug fix
      NO_TEST=tested by ASAN
      
      (cherry picked from commit 0e5a3cc2)
      2a6a9b75
  3. Jul 13, 2023
    • Igor Munkin's avatar
      cmake: introduce FIBER_STACK_SIZE option · aae1daab
      Igor Munkin authored
      In scope of the commit 82f4b4a3 ("lib/core/fiber: Increase default
      stack size") the default value of fiber stack size is increased up to
      512 Kb (you can find the reasons in the aforementioned commit message
      and in https://github.com/tarantool/tarantool/issues/3418 description).
      
      Some of the tests in test/PUC-Rio-Lua-5.1-test suite in LuaJIT repo
      (e.g. some cases with deep recursion in errors.lua or pm.lua) have
      already been tweaked according to the limitations mentioned in
      https://github.com/tarantool/tarantool/issues/5782, but the crashes
      still occurs while running LuaJIT tests with ASan support enabled.
      
      To make the testing routine more convenient, FIBER_STACK_SIZE option is
      introduced to Tarantool CMake machinery. One can provide the size either
      by raw digits (i.e. in bytes) or using Kb/Mb suffixes for convenience.
      
      A couple of important nits:
      * If the given value is not a multiple of 4Kb, CMake machinery adjusts
        it up to the nearest one greater than this value.
      * If the adjusted value is less than 512Kb, configuration fails with the
        corresponding CMake fatal error.
      
      Follows up #3418
      Relates to #5782
      
      @TarantoolBot document
      Title: introduce FIBER_STACK_SIZE configuration option
      
      To make managing of the default fiber stack size more convenient, the
      corresponding CMake option is added.
      
      **NB**: The stack size can't be less than 512Kb and if the given value
      is not a multiple of 4Kb, CMake machinery adjusts it up to the nearest
      one greater than this value.
      
      (cherry picked from commit ff57f990)
      Unverified
      aae1daab
    • Gleb Kashkin's avatar
      compat: fix err msg in compat.<option>:is_new() · d33e0635
      Gleb Kashkin authored
      In the original commit 5f6d367c ("compat: add is_new and
      is_old to options") `compat.<option_name>:is_new()` and `:is_old()`
      were introduced, but by mistake they contained different usage
      messages. This patch updates `:is_new()` usage msg to more
      informative one from `:is_old()`.
      
      Follows up #8807
      
      NO_CHANGELOG=changelog from 5f6d367c is valid
      NO_DOC=doc from 5f6d367c is valid
      
      (cherry picked from commit f590ec22)
      Unverified
      d33e0635
  4. Jul 12, 2023
    • Gleb Kashkin's avatar
      compat: add is_new and is_old to options · 380956f4
      Gleb Kashkin authored
      It used to be somewhat complicated to check the effective value
      of a compat option, because `<option_name>.current` could contain
      'default' state.
      This patch introduces helper functions that take care of that.
      
      The following alternatives were considered:
      * `compat.<option_name>.effective` - it is excessive in the presence
        if `current` and `default`, and is visible in serialization
      * `compat.<option_name>.get()` - while it is a function, it does only
        half of the work required, user still has to compare result to 'new'
      
      Closes #8807
      
      @TarantoolBot document
      Title: Add `:is_new/old()` helpers to tarantool.compat options
      
      `compat.<option_name>.current` can be 'new', 'old' or 'default',
      thus when it is default there must be an additional check if
      `compat.<option_name>.default` is 'new'. It is handier to have a
      helper to deal with that instead of complicated `if`:
      * check if effective value is 'new' before the patch:
        ```lua
        if compat.<option_name>.current == 'new' or
                (compat.<option_name>.current == 'default' and
                 compat.<option_name>.default == 'new') then
            ...
        end
        ```
      * after the patch:
        ```lua
        if compat.<option_name>:is_new() then
            ...
        end
        ```
      
      Please update [tutorial on using compat], maybe add an example to
      [Listing options details].
      
      [tutorial on using compat]: https://www.tarantool.io/en/doc/latest/reference/reference_lua/compat/compat_tutorial/
      [Listing options details]: https://www.tarantool.io/en/doc/latest/reference/reference_lua/compat/compat_tutorial/#listing-options-details
      
      (cherry picked from commit 5f6d367c)
      Unverified
      380956f4
  5. Jul 06, 2023
    • Magomed Kostoev's avatar
      box: compare and hash msgpack value of double key field as double · 45044d20
      Magomed Kostoev authored
      1. Make double-formatted fields accept integer and float values.
      2. Make indexes compare the values as double if the field key type
         is FIELD_TYPE_DOUBLE.
      3. Make hashers cast double key field to double before hashing, so
         we are able to insert and select any int, uint, float or double
         if their value casted to double is equal (for double keys).
      
      Notes about tuple_compare.cc:
      
      Since now `mp_compare_double` casts any value placed in field to
      double it was renamed to `mp_compare_as_double` to not semantically
      conflict with existing `mp_compare_double_*` functions.
      
      Notes about tuple_hash.cc:
      
      The hashee cast result is encoded in MP_DOUBLE and hashed for
      backward compatibility reasons.
      
      Since now the field hashing function (tuple_hash_field) requires
      field type to hash the field correctly, a new parameter has been
      introduced.
      
      By the way added assertions to the generic `field_hash` to prevent
      invalid hashing for new precompiled hashers and made
      `key_hash_slowpath` static cause it's only used in this file.
      
      Closes #7483
      Closes #5933
      Unblocks tarantool/crud#298
      
      @TarantoolBot document
      Title: It's not required to ffi-cast integral floating point to
      double anymore.
      
      The page describing tarantool data model states that:
      
      > In Lua, fields of the double type can only contain non-integer
      > numeric values...
      
      If the patch is merged this isn't the case anymore, so this
      statement and the code snippet below it should be updated.
      
      Link to the document: [Data storage](https://www.tarantool.io/en/doc/latest/concepts/data_model/value_store/#field-type-details).
      
      Affected segments:
      
      > double. The double field type exists mainly to be equivalent
      > to Tarantool/SQL’s DOUBLE data type. In msgpuck.h (Tarantool’s
      > interface to MsgPack), the storage type is MP_DOUBLE and the
      > size of the encoded value is always 9 bytes. In Lua, fields of
      > the double type can only contain non-integer numeric values and
      > cdata values with double floating-point numbers. Examples: 1.234,
      > -44, 1.447e+44.
      >
      > To avoid using the wrong kind of values inadvertently, use
      > ffi.cast() when searching or changing double fields. For example,
      > instead of space_object:insert{value} use ffi = require('ffi')
      > ... space_object:insert({ffi.cast('double',value)}). Example:
      >
      > ```
      > s = box.schema.space.create('s', {format = {{'d', 'double'}}})
      > s:create_index('ii')
      > s:insert({1.1})
      > ffi = require('ffi')
      > s:insert({ffi.cast('double', 1)})
      > s:insert({ffi.cast('double', tonumber('123'))})
      > s:select(1.1)
      > s:select({ffi.cast('double', 1)})
      > ```
      
      (cherry picked from commit 51af059c)
      45044d20
    • Magomed Kostoev's avatar
      Bump msgpuck submodule · 206ed899
      Magomed Kostoev authored
      This update pulls the following commits:
      * Add mp_read_double_lossy without direct convertibility checks
      * Fix mp_read_double_lossy tests freebsd build
      
      These commits introduce a function required to compare and hash
      msgpack values of double key fields as double.
      
      Need for #7483, #5933
      
      NO_DOC=see the next commit
      NO_CHANGELOG=see the next commit
      
      (cherry picked from commit be47f8fb)
      206ed899
    • Sergey Bronnikov's avatar
      test: fix tarantool process teardown · 56024743
      Sergey Bronnikov authored
      Test uses a popen module that starts tarantool process in background
      mode. Tarantool process started in background mode forks a new process
      and closes a parent, after that popen loses a PID of the started process
      and `ph:kill()` and `ph:terminate()` doesn't work anymore. It leads to
      non-terminated tarantool processes after running the test.
      
      Patch fixes that by running `kill` using os.execute with a PID of
      tarantool process written to a pid file.
      
      Follows up #6128
      
      NO_CHANGELOG=fix test
      NO_DOC=fix test
      
      (cherry picked from commit 88686227)
      Unverified
      56024743
    • Ilya Grishnov's avatar
      box: fix shared lang between connected clients · 93812b3b
      Ilya Grishnov authored
      Fixed the implementation of the box console.
      Before this fix, result of `\set language` is shared between clients
      via `console.connect`, despite the fact that clients have different
      `box.session.id`. Now the parameter of the selected language is stored
      by each client in his own `box.session.storage`.
      
      Fixes #8817
      
      NO_DOC=bugfix
      
      (cherry picked from commit e4fda4b7)
      Unverified
      93812b3b
  6. Jul 04, 2023
    • Igor Munkin's avatar
      luajit: bump new version · ae11d314
      Igor Munkin authored
      * test: fix flaky <unit-jit-parse.test.lua>
      * Fix use-def analysis for vararg functions.
      * Fix use-def analysis for BC_VARG.
      * Fix TNEW load forwarding with instable types.
      * Fix memory probing allocator to check for valid end address, too.
      * Another fix for lua_yield() from C hook.
      * Fix lua_yield() from C hook.
      * Fix saved bytecode encapsulated in ELF objects.
      * x64: Fix 64 bit shift code generation.
      * Fix canonicalization of +-0.0 keys for IR_NEWREF.
      * test: add utility for parsing `jit.dump`
      * test: split utils.lua into several modules
      * test: rewrite lj-49-bad-lightuserdata test in C
      * test: rewrite misclib-sysprof-capi test in C
      * test: rewrite misclib-getmetrics-capi test in C
      * test: introduce utils.h helper for C tests
      * test: introduce module for C tests
      * test: fix setting of {DY}LD_LIBRARY_PATH variables
      * build: fix build with LUAJIT_USE_GDBJIT enabled
      * ci: update the branch name for Tarantool 2.11
      
      Closes #8718
      Part of #7900
      Part of #8516
      
      NO_DOC=LuaJIT submodule bump
      NO_TEST=LuaJIT submodule bump
      ae11d314
    • Sergey Bronnikov's avatar
      test: fix flakiness in gh_6128_background_mode_test · 9d5cd29d
      Sergey Bronnikov authored
      Previous attempt to fix flakiness in commit 6a2c73f8 ("test: fix
      flakiness in gh_6128_background_mode_test") used a constant buffer size
      in check_err_msg function. Tarantool 2.10 has a bit bigger log before a
      desired message that other versions of Tarantool and it leads to a this
      resulted in a truncated message ("entering the even" instead of
      "entering the event loop"). Patch replaces check_err_msg()
      implementation to grep_log used in luatest, it reads the whole log.
      
      Also patch renames check_err_msg to check_msg, because "entering the
      event loop" is not an error message.
      
      Follows up #6128
      
      NO_CHANGELOG=fix test
      NO_DOC=fix test
      
      (cherry picked from commit 1c8e7124)
      Unverified
      9d5cd29d
    • Magomed Kostoev's avatar
      box: check permissions on constraint functions on creation · 2cde4c66
      Magomed Kostoev authored
      Function execution permissions should only be checked on constraint
      creation.
      
      So when the function is used to check a tuple access rights don't
      have to be checked on each call for the current user.
      
      Closes #7873
      
      NO_DOC=bugfix
      
      (cherry picked from commit 6b8f2c5f)
      2cde4c66
    • Magomed Kostoev's avatar
      box: check permissions on functional index functions on creation · 18e54fe7
      Magomed Kostoev authored
      Function execution permissions should only be checked on functional
      index creation and on functional index function set.
      
      So when the function is used by key_list_iterator its rights don't
      have to be checked on each call for the current user.
      
      Part of #7873
      
      NO_DOC=bugfix
      NO_CHANGELOG=see the next commit
      
      (cherry picked from commit ddbdb77a)
      18e54fe7
    • Georgiy Lebedev's avatar
      memtx: fix heap-use-after-free of tuple stories caused by space alter · 1d0bc37b
      Georgiy Lebedev authored
      When a space is altered, we abort all in-progress transactions and delete
      all stories related to that space: the problem is we don't delete the
      stories' read gaps, which are also linked to the stories' transactions,
      which get cleaned up on transaction destruction — this, in turn, results in
      heap-use-after-free. To fix this, clean up stories' read gap in
      `memtx_on_space_delete` — we don't do this in `memtx_tx_story_delete` since
      it expects the story to not have any read gaps (see
      `memtx_tx_story_gc_step`).
      
      Tested this patch manually against Nick Shirokovskiy's experimental
      small-ASAN integration branch.
      
      Closes #8781
      
      NO_DOC=bugfix
      NO_TEST=<already covered by existing tests, but was not detectable by ASAN>
      
      (cherry picked from commit e1ed31bb)
      1d0bc37b
  7. Jun 30, 2023
    • Sergey Bronnikov's avatar
      test: fix flakiness in gh_6128_background_mode_test · 6381fc21
      Sergey Bronnikov authored
      Test runs an external process with tarantool that writes to a log file.
      Then test reads that log file and searches a string with required
      message in it (see function check_err_msg). Test was flaky on macOS and
      I suspect it was happening due to a high log level - timeout was not
      enough to wait message in the log file.
      
      Patch decreases a log level to a default value and replaces io
      functions with the similar alternatives in a fio module. Using
      fio functions allows to not block fibers.
      
      NO_CHANGELOG=test fix
      NO_DOC=test fix
      
      (cherry picked from commit 47380bb7)
      Unverified
      6381fc21
    • Vladimir Davydov's avatar
      lua/xlog: don't ignore unknown header fields · e874c471
      Vladimir Davydov authored
      The xlog reader Lua module uses the xlog_cursor_next_row, which decodes
      the row header with xrow_header_decode. The latter silently ignores any
      unknown fields, which complicates catching bugs when garbage is written
      to a row header by mistake, for example, see #8783.
      
      Let's parse a row header without using xrow_header_decode in the xlog
      reader module, like we parse a row body, and output all unknown/invalid
      keys as is.
      
      To do that, we have to extend the xlog cursor API with the new method
      xlog_cursor_next_row_raw that returns a pointer to the position in the
      tx buffer where the next xrow is stored without advancing it. To avoid
      a memory leak in case the caller fails to parse an xrow returned by this
      function, we also have to move the call to xlog_tx_cursor_destroy from
      xlog_tx_cursor_next_row to xlog_cursor_next_tx.
      
      While we are at it,
       - Don't raise an error if a key type encountered in a row body is
         invalid (not an integer). Instead, silently ignore such keys.
       - Remove the useless body MsgPack validness check because we already
         check it after decoding the header.
       - Add error injection based tests to check all the corner cases.
      
      NO_DOC=bug fix
      
      (cherry picked from commit 8a25d170)
      e874c471
    • Vladimir Davydov's avatar
      txn: reset stream_id row header field · 37a500d2
      Vladimir Davydov authored
      To avoid garbage written to xlog.
      
      Closes #8783
      
      NO_DOC=bug fix
      NO_TEST=next commit
      
      (cherry picked from commit f058cee7)
      37a500d2
  8. Jun 29, 2023
  9. Jun 27, 2023
  10. Jun 23, 2023
    • Georgiy Lebedev's avatar
      box: fix memory leaks on `ER_MULTISTATEMENT_TRANSACTION` in DDL · 48d67e5b
      Georgiy Lebedev authored
      Space index build and space format checking operations don't destroy space
      iterator on `txn_check_singlestatement` failure — fix this.
      
      Closes #8773
      
      NO_DOC=bugfix
      NO_TEST=<leak happens in small, cannot be detected by sanitizer>
      
      (cherry picked from commit 6689f511)
      48d67e5b
    • Georgiy Lebedev's avatar
      core: use `malloc` instead of `region` allocator in procedure name cache · 0802c68d
      Georgiy Lebedev authored
      The procedure name cache uses a region for hash table entry allocation, but
      the cache is a thread local variable, while the `region` uses a `slab`
      allocator the lifetime of which is bounded by `main`, while thread locals
      are destroyed after `main`: use the `malloc` allocator instead.
      
      Benchmark from #7207 shows that this change does not effect performance.
      
      Closes #8777
      
      NO_CHANGELOG=<leak does not affect users>
      NO_DOC=bugfix
      NO_TEST=<detected by ASAN>
      
      (cherry picked from commit 7135910e)
      0802c68d
  11. Jun 22, 2023
    • Sergey Bronnikov's avatar
      test: testing tarantool in background mode · 861f6bf3
      Sergey Bronnikov authored
      Before a commit ec1af129 ("box: do not close xlog file descriptors in
      the atfork handler") there was a bug when Tarantool with enabled
      background mode via environment variable could lead a crash:
      
      NO_WRAP
      ```
      $ TT_PID_FILE=tarantool.pid TT_LOG=tarantool.log TT_BACKGROUND=true TT_LISTEN=3301 tarantool -e 'box.cfg{}'
      $ tail -3 tarantool.log
      2021-11-02 16:05:43.672 [2341202] main init.c:696 E> LuajitError: cannot read stdin: Resource temporarily unavailable
      2021-11-02 16:05:43.672 [2341202] main F> fatal error, exiting the event loop
      2021-11-02 16:05:43.672 [2341202] main F> fatal error, exiting the event loop
      ```
      NO_WRAP
      
      With commit ec1af129 ("box: do not close xlog file descriptors in
      the atfork handler") described bug could not be reproduced.
      
      Proposed patch adds a test that starts Tarantool in background mode
      enabled via box.cfg option and via environment variable TT_BACKGROUND to
      make sure this behaviour will not be broken in a future.
      
      Closes #6128
      
      NO_DOC=test
      
      (cherry picked from commit f676fb7c)
      Unverified
      861f6bf3
    • Ilya Verbin's avatar
      build: disable backtrace feature on AArch64 Linux · e999c315
      Ilya Verbin authored
      There are sporadic segfaults in libunwind during backtrace_collect().
      Reproducible with the latest version, and there are open issues with
      similar stacks:
      
      https://github.com/libunwind/libunwind/issues/150
      https://github.com/libunwind/libunwind/issues/260
      https://github.com/libunwind/libunwind/issues/473
      
      Let's disable ENABLE_BACKTRACE for AArch64 Linux until these issues are
      resolved in libunwind.
      
      Closes #8572
      Part of #8791
      
      NO_DOC=bugfix
      
      (cherry picked from commit 418e749c)
      e999c315
    • Ilya Verbin's avatar
      build: remove backtrace feature compiler dependency from rpm spec · ecd4af16
      Ilya Verbin authored
      The ability to support backtraces is checked in cmake/compiler.cmake,
      it makes no sense to duplicate the check in rpm/tarantool.spec. Also do
      not enable backtraces unconditionally in apk/APKBUILD and static-build.
      
      Part of #6998
      
      NO_DOC=build
      NO_TEST=build
      NO_CHANGELOG=build
      
      (cherry picked from commit f7c4a34a)
      ecd4af16
    • Aleksandr Lyapunov's avatar
      memtx: abort readers of rollbacked prepared · aca30926
      Aleksandr Lyapunov authored
      There's case when a transaction is rolled back from prepared
      state. This happens when WAL fails, synchronized replication
      confirmation failure or perhaps in other similar cases. By design
      other RW transactions and transactions with READ_COMMITTED
      isolation level are allowed to read prepared state. All these
      transactions must be aborted in case of rollback of prepared
      transaction since they definitely have read no more possible
      database state.
      
      This patch implements this abortion.
      
      Closed #8654
      
      NO_DOC=bugfix
      
      (cherry picked from commit 54986902)
      aca30926
    • Aleksandr Lyapunov's avatar
      memtx: fix and refactor memtx_tx_history_rollback_stmt · b7edae28
      Aleksandr Lyapunov authored
      Rollback is rather complicated part if MVCC implementation that
      is meant to handle two kinds of rollback:
      * rollback from in-progress state, if box.rollback() is called.
      * rollback from prepared state, when WAL fails.
      Unfortunately the last one was not properly tested and surely
      has at least one flaw. When an inserting transaction becomes
      prepared its stories could be linked as deleted (via del_stmt
      pointer) by other in-progress transactions in order to maintain
      correct visibility. The problem is that in case of rollback of
      such prepared transaction those links remained. Broken links
      breaks the chain structure, and some older changes (that were
      linked as deleted before that hapless preparation) can become
      visible to other transaction.
      
      Refactor, simplify a bit that part of code and fix the issue
      described above; cover with tests.
      
      Closes #8648
      
      NO_DOC=bugfix
      
      (cherry picked from commit 85569d9c)
      b7edae28
    • Aleksandr Lyapunov's avatar
      memtx: simplify and refactor memtx_tx_history_prepare_stmt · 99f36d1d
      Aleksandr Lyapunov authored
      Almost completely rewrite, simplify and comment this part of code.
      
      Part of #8648
      Part of #8654
      
      NO_DOC=refactoring
      NO_TEST=refactoring
      NO_CHANGELOG=refactoring
      
      (cherry picked from commit e9015074)
      99f36d1d
    • Aleksandr Lyapunov's avatar
      memtx: simplify and refactor memtx_tx_history_add_stmt · 418b0e07
      Aleksandr Lyapunov authored
      Almost completely rewrite, simplify and comment this part of code.
      
      Part of #8648
      Part of #8654
      
      NO_DOC=refactoring
      NO_TEST=refactoring
      NO_CHANGELOG=refactoring
      
      (cherry picked from commit 63da3bed)
      418b0e07
    • Aleksandr Lyapunov's avatar
      memtx: replace is_pure_insert flag with is_own_change flag · ce050b59
      Aleksandr Lyapunov authored
      The latter flag is a bit wider: it reveals not only inserting
      statements after deleting by the same transaction, but also
      replacing and deleting statements after all kinds of previois
      changes but the same transaction. This extended behavior will
      be used in further commits.
      
      Part of #8648
      Part of #8654
      
      NO_DOC=refactoring
      NO_TEST=refactoring
      NO_CHANGELOG=refactoring
      
      (cherry picked from commit 3cfa6756)
      ce050b59
    • Aleksandr Lyapunov's avatar
      memtx: remove does_require_old_tuple member · a2ed6eb9
      Aleksandr Lyapunov authored
      It was an ugly solution when MVCC engine requires outside engine
      to set this flag which is not convenient.
      
      Remove it and use mode arguments to set up proper read trackers.
      
      Part of #8648
      Part of #8654
      
      NO_DOC=refactoring
      NO_TEST=refactoring
      NO_CHANGELOG=refactoring
      
      (cherry picked from commit 74149734)
      a2ed6eb9
    • Aleksandr Lyapunov's avatar
      memtx: remove dead code · 79a2f764
      Aleksandr Lyapunov authored
      The function memtx_tx_story_delete is expected to delete fully
      unlinked stories and thus should not try to unlink something by
      itself. So remove unlink and add asserts instead.
      
      Part of #8648
      Part of #8654
      
      NO_DOC=refactoring
      NO_TEST=refactoring
      NO_CHANGELOG=refactoring
      
      (cherry picked from commit 07067407)
      79a2f764
    • Aleksandr Lyapunov's avatar
      memtx: join and unify mvcc gap trackers · 80e492fa
      Aleksandr Lyapunov authored
      Now there are three kinds of very close trackers:
      * The transaction have read some tuple that is not committed and
        thus not visible. This kind is now stored as gap_item with
        is_nearby = false.
      * The transaction made a select or range scan, reading a key or
        range between two adjacent tuples of the index. This kind is
        stored as gap_iteam with is_nearby = true.
      * A transaction completed a full scan of unordered index. This
        kind is stored as full_scan_item.
      
      All these trackers serve for the same thing: to record a fact
      that a transaction read something but didn't see anything.
      
      There are some problems with the current solution:
      * gap_item with is_nearby = false has several unused members that
        just consume space.
      * bool is_nearby flag for type descriptin is an ugly solution.
      * full_scan_item is separated from logically close items.
      
      This commit joins all these trackers under one base (that is
      struct gap_item_base) and solves problems above.
      
      Part of #8648
      Part of #8654
      
      NO_DOC=refactoring
      NO_TEST=refactoring
      NO_CHANGELOG=refactoring
      
      (cherry picked from commit f8d97a2e)
      80e492fa
    • Aleksandr Lyapunov's avatar
      memtx: logically divide read tracking and gap tracking · 47fb7277
      Aleksandr Lyapunov authored
      Now read trackers are used both for cases when a transaction has
      read an existing value and it has read nothing (read by key but
      there was no visible tuple in this place). For latter case an
      additional index_mask was used to identify from which index the
      read was done. Along with that there was per-index interval gap
      trackers.
      
      This patch divides area of responsibility between read trackers
      and gap tracker in the following way:
      * Reads of existing visible values are stored in read trackers.
      * Reads of non-existing or non-visible values are store in gap
        trackers.
      
      This new approach allows to provide new invariants: gap trackers
      are stored only at top of chain, and read trackers are stored
      only at topmost committed story in chain.
      
      Part of #8648
      Part of #8654
      
      NO_DOC=refactoring
      NO_TEST=refactoring
      NO_CHANGELOG=refactoring
      
      (cherry picked from commit 7b8b78be)
      47fb7277
    • Aleksandr Lyapunov's avatar
      memtx: rename nearby_gaps -> read_gaps · b55d362c
      Aleksandr Lyapunov authored
      In further commit this list will be used for tracking all read
      gaps, not only 'nearby'. Since this rename has rather huge
      diff, let's make it in separate commit.
      
      No logical changes.
      
      Part of #8648
      Part of #8654
      
      NO_DOC=refactoring
      NO_TEST=refactoring
      NO_CHANGELOG=refactoring
      
      (cherry picked from commit d3feb691)
      b55d362c
    • Aleksandr Lyapunov's avatar
      memtx: check for ephemeral spaces in a uniform way · 39acee0c
      Aleksandr Lyapunov authored
      In our SQL implementation temporary spaces are used. They come to
      MVCC engine in two variants - NULL or ephemeral. In both cases
      MVCC engine must not do anything, there are several checks for
      that in different parts of code.
      
      Normalize these checks and make them similar to each other.
      
      Part of #8648
      Part of #8654
      
      NO_DOC=refactoring
      NO_TEST=refactoring
      NO_CHANGELOG=refactoring
      
      (cherry picked from commit 86a8155c)
      39acee0c
    • Aleksandr Lyapunov's avatar
      memtx: drop memtx_tx_track_read_story_slow · 61bad333
      Aleksandr Lyapunov authored
      The only place where this static function is used is more general
      static function - memtx_tx_track_read_story. This is a bit
      confusing - usually slow variant stand for public inline 'fast'
      method.
      
      So merge both functions in one.
      
      Part of #8648
      Part of #8654
      
      NO_DOC=refactoring
      NO_TEST=refactoring
      NO_CHANGELOG=refactoring
      
      (cherry picked from commit 32e41b7a)
      61bad333
    • Aleksandr Lyapunov's avatar
      memtx: fix lost gap and full scan items · f58ecaeb
      Aleksandr Lyapunov authored
      By a mistake in 8a565144 a shortcut was added to procedure
      that handles gap write: it was considered that if the writing
      transaction is the same as reading - there is no actual conflict
      that must be stored further. That was a wrong decision: if such
      a transaction yields and another transaction comes and commits
      a value with the same key - the first one must go to conflicted
      state since it has read no more possible state.
      
      Another similar mistake was made in e6f5090c, where writing
      after full scan of the same transaction was not tracked as read.
      Obviously that was wrong: if some other transaction overwrites
      the key and commits - this transaction must go to read view since
      it did not see anything by this key which is not so anymore.
      
      Fix it, reverting the first commit and an modifying the second and
      add a test.
      
      Closes #8326
      
      NO_DOC=bugfix
      
      (cherry picked from commit b41c4546)
      f58ecaeb
Loading