Skip to content
Snippets Groups Projects
  1. Aug 20, 2024
  2. Aug 16, 2024
    • Nikita Zheleztsov's avatar
      engine: introduce stubs for checkpoint FETCH_SNAPSHOT · 23c7899e
      Nikita Zheleztsov authored
      This commit introduces engine stubs that enable a new method
      of fetching snapshots for anonymous replicas. Instead of using
      the traditional read-view join approach, this update allows
      file snapshot fetching. Note that file snapshot fetching
      is only available in Tarantool EE.
      
      Checkpoint fetching is done via IPROTO_IS_CHECKPOINT_JOIN,
      IPROTO_CHECKPOINT_VCLOCK and IPROTO_CHECKPOINT_LSN fields.
      
      If IPROTO_CHECKPOINT_JOIN is set to true, join will be done from
      files: .snap for memtx, .run for vinyl, if false - from read view.
      
      Checkpoint join allows to continue from the place, where client
      stopped in case of snapshot fetching error. This allows to avoid
      rebootstrap of an anonymous client. This can be done by specifying
      CHECKPOINT_VCLOCK, which says from which file server should continue
      join, client gets vclock at the beginning of the join. Specifying
      CHECKPOINT_LSN allows to continue from some position in checkpoint.
      Server sends all data >= CHECKPOINT_LSN.
      
      If CHECKPOINT_VCLOCK is not specified, fetching is done from the latest
      available checkpoint. If CHECKPOINT_LSN is not specified - start from
      the beginning of the snap. So, specifying only IS_CHECKPOINT_JOIN
      triggers fetching the latest checkpoint from files.
      
      Needed for tarantool/tarantool-ee#741
      
      NO_DOC=ee
      NO_TEST=ee
      NO_CHANGELOG=ee
      
      (cherry picked from commit 2fca5c13)
      23c7899e
    • Nikita Zheleztsov's avatar
      engine: send vclock with 0th component during join · 9434531b
      Nikita Zheleztsov authored
      This commit makes engine to send vclock without ignoring 0th component
      during join, which is needed for checkpoint FETCH SNAPSHOT.
      
      Currently engine join functions are invoked only from
      relay_initial_join, which is done during JOIN or FETCH SNAPSHOT.
      They respond with vclock of the read view we're going to send.
      
      In the following commit checkpoint FETCH SNAPSHOT will be introduced,
      which responds with vclock of the checkpoint, we're going to send.
      Such vclock may include 0th component and it's crucial to send it to
      a client, as in case of connection failure, client will send us the
      same vclock and we'll have to use its signature to figure out, which
      checkpoint client wants.
      
      So, we have to send and receive 0th component of the vclock during
      FETCH_SNAPSHOT. This commit also introduces decoding vclocks without
      ignoring 0th component, as they'll be used in the following commit too.
      
      Needed for tarantool/tarantool-ee#741
      
      NO_DOC=internal
      NO_TEST=ee
      NO_CHANGELOG=internal
      
      (cherry picked from commit 56058393)
      9434531b
    • Nikita Zheleztsov's avatar
      xrow: rename xrow_encode_vclock · 4de3d0d6
      Nikita Zheleztsov authored
      This commit renames xrow_encode_vlock to xrow_encode_vclock_ignore0
      since the next commit will introduce encoding vclock without ignoring
      0th component, which is needed during sending the response to fetch
      snapshot request.
      
      This commit also removes internal field inside the replication_request
      structure, as the following commit will use 'vclock' for
      encoding/decoding vclock without ignoring component.
      
      Needed for tarantool/tarantool-ee#741
      
      NO_DOC=refactoring
      NO_TEST=refactoring
      NO_CHANGELOG=refactoring
      
      (cherry picked from commit 313bd730)
      4de3d0d6
    • Nikita Zheleztsov's avatar
      relay: refactor relay_initial_join · 854d09ff
      Nikita Zheleztsov authored
      From now on during initial join memtx engine prepares vclock, raft and
      limbo states, it also sends them during memtx_engine_join.
      
      It's done in order to simplify the code of initial join, as in the
      consequent commit checkpoint initial join will be introduced and we want
      relay code to handle it the same as read-view join without confusing
      conditions.
      
      Needed for tarantool/tarantool-ee#741
      
      NO_DOC=refactoring
      NO_TEST=refactoring
      NO_CHANGELOG=refactoring
      
      (cherry picked from commit 72cc2b3e)
      854d09ff
    • Nikita Zheleztsov's avatar
      engine: move raft and limbo states after system data in checkpoint · c107ba11
      Nikita Zheleztsov authored
      Before this commit raft and limbo states were written at the end of the
      checkpoint, which makes it very costly to access them.
      
      Checkpoint join needs to access limbo and raft state in order to send
      them during JOIN_META stage. We cannot use the latest states, like it's
      done for read-view snapshot fetching: states may be far ahead of the
      data, written to the checkpoint, which we're going to send.
      
      This commit moves raft and limbo states after data from the system
      spaces but before user data. We cannot put them right at the beginning
      of the snapshot, because then we'll have to patch recovery process,
      which currently strongly relies on the fact, that system spaces are
      at the beginning of the snapshot (this was done in order to apply force
      recovery only for user data). If we patch recovery process, then old
      versions, where it's unpatched, won't be able to recover from the
      snapshots done by the newer version, compatibility of snapshots will be
      broken.
      
      The current change is not breaking, old Tarantool versions can restore
      from the snapshot made by the newer one.
      
      Needed for tarantool/tarantool-ee#741
      
      NO_DOC=internal
      NO_CHANGELOG=internal
      
      (cherry picked from commit 3da31b83)
      c107ba11
    • Serge Petrenko's avatar
      replication: send anonymous replica version id · 2c7e82c2
      Serge Petrenko authored
      Starting with commit f1c2127d ("replication: add META stage to JOIN")
      replication master appends a special section, called IPROTO_JOIN_META to
      the initial snapshot sent to the replica. This section contains the
      latest raft term and synchronous transaction queue owner and term.
      
      The section is only sent to nodes, which have a non-zero version_id.
      For some reason, version_id encoding for FETCH_SNAPSHOT (analog of JOIN
      for anonymous replicas) wasn't added in that commit, so anonymous
      replicas do not receive synchronous queue state.
      
      This leads to them raising ER_SPLIT_BRAIN errors later after join, when
      the first synchronous row arrives.
      
      In order to fix this, start encoding version_id in FETCH_SNAPSHOT
      requests.
      
      Closes #9401
      
      NO_DOC=cherry-pick
      
      (cherry picked from commit 58c006c9)
      2c7e82c2
    • Aleksandr Lyapunov's avatar
      box: export more symbols · 03f8270b
      Aleksandr Lyapunov authored
      box_info_lsn
      box_is_ro
      box_ro_reason
      box_wait_ro
      
      Closes #10378
      
      NO_CHANGELOG=minor change
      
      @TarantoolBot document
      Title: document 4 more symbols in public C API
      
      box_info_lsn
      box_is_ro
      box_ro_reason
      box_wait_ro
      
      Their meaning is identical to lua methods:
      
      box.info.lsn
      box.info.ro
      box.info.ro_reason
      box.wait_ro
      
      (cherry picked from commit f1c76b976077e8078c42990546d9dbbdea315425)
      03f8270b
  3. Aug 15, 2024
    • Sergey Kaplun's avatar
      luajit: bump new version · 21f2ba4c
      Sergey Kaplun authored
      * FFI: Treat cdata finalizer table as a GC root.
      * FFI: Turn FFI finalizer table into a proper GC root.
      
      Part of #10199
      
      NO_DOC=LuaJIT submodule bump
      NO_TEST=LuaJIT submodule bump
      21f2ba4c
    • Sergey Bronnikov's avatar
      datetime: support timestamp field in :totable() · 9d7b33bc
      Sergey Bronnikov authored
      `datetime` module has a function `:totable()` that converts
      the information from a datetime object into the table format.
      The field `timestamp` can be accessed in a datetime object:
      
      ```
      tarantool> dt = datetime.new({timestamp=100})
      ---
      ...
      
      tarantool> dt.timestamp
      ---
      - 100
      ...
      ```
      
      However, the timestamp is missed in a table produced by
      `:totable()` method. The patch fixes that.
      
      Fixes #10374
      
      @TarantoolBot document
      Title: Support of timestamp field in :totable()
      
      The field `timestamp` is available in a table produced by
      `:totable`:
      
      ```
      tarantool> datetime.new({timestamp=100}):totable()
      ---
      - timestamp: 100
        tz:
        sec: 40
        min: 1
        yday: 1
        day: 1
        nsec: 0
        isdst: false
        wday: 5
        tzoffset: 0
        month: 1
        year: 1970
        hour: 0
      ...
      
      tarantool>
      ```
      
      (cherry picked from commit f66020b670296642547cbfe0a34ab6c11d69989d)
      9d7b33bc
    • Sergey Bronnikov's avatar
      test: remove testcase with fractional datetime fields · 6b4a6dd1
      Sergey Bronnikov authored
      There is no requirement for using fractional numbers
      in datetime constructors. This issue will be addressed in [1].
      The patch removes testcase with fractional numbers in datetime
      constructor.
      
      1. https://github.com/tarantool/tarantool/issues/10391
      
      See also #10391
      Needed for #10374
      
      NO_CHANGELOG=testing
      NO_DOC=testing
      
      (cherry picked from commit 989c63c75f5f8c7b38dc67f8e5c32ce8bef1da08)
      6b4a6dd1
    • Sergey Bronnikov's avatar
      datetime: test RFC 3339 and ISO 8601 formats · a218d415
      Sergey Bronnikov authored
      The commit 3c403661 ("datetime, lua: date parsing functions")
      introduced a function `datetime.parse()`. The function accepts
      a datetime format: 'iso8601', 'rfc3339' and a custom datetime
      format. However, 'iso8601' as well as 'rfc3339' is a set of
      datetime formats (see [1] and [2]), and `datetime.parse()`
      supports a subset of these formats. There is a documentation
      issue [3], where we should describe what formats are supported
      and what formats are not.
      
      The patch adds a tests for most of supported formats.
      The tests were created using a page about date and time formats
      in ISO 8601 and RFC 3339 standards [2] and these tests are not
      exhaustive. More thorough testing can be done with property-based
      testing and randomization. It will be a subject of the following
      commits.
      
      1. https://datatracker.ietf.org/doc/html/rfc3339
      2. https://ijmacd.github.io/rfc3339-iso8601/
      3. https://github.com/tarantool/doc/issues/3155
      
      
      
      Follows up #6731
      Needed for tarantool/doc#3155
      
      Co-authored-by: default avatarSergey Kaplun <skaplun@tarantool.org>
      
      NO_CHANGELOG=testing
      NO_DOC=tarantool/doc#3155
      
      (cherry picked from commit dcb196366621db4ab39ef97029ee126d836b77fb)
      a218d415
  4. Aug 14, 2024
    • Sergey Bronnikov's avatar
      datetime: support tz field in :totable() · 9121eccc
      Sergey Bronnikov authored
      `datetime` module has a function `:totable()` that converts
      the information from a datetime object into the table format.
      The commit 43e10ed3 ("build, lua: built-in module datetime")
      added `tzoffset` field to the datetime object and to table
      produced by `:totable()`. The commit 9ee45289
      ("datetime: datetime.TZ array") added fields `tz` and `tzindex` to
      the datetime object, but not to the table produced by
      `:totable()`. The patch fixes that. Note, `tzindex` is not added,
      because it is an internal field.
      
      ```
      tarantool> datetime.parse('2004-12-01T00:00 Europe/Moscow'):totable()
      ---
      - tz: Europe/Moscow
        sec: 0
        min: 0
        yday: 336
        day: 1
        nsec: 0
        isdst: false
        wday: 4
        tzoffset: 180
        month: 12
        year: 2004
        hour: 0
      ...
      ```
      
      Fixes #10331
      Follows up #6751
      
      @TarantoolBot document
      Title: Support of tz field in :totable()
      
      In addition to the `tzoffset` in a table produced by `:totable`
      we added `tz` field.
      
      ```
      tarantool> datetime.parse('2004-12-01T00:00 Europe/Moscow'):totable()
      ---
      - tz: Europe/Moscow
        sec: 0
        min: 0
        yday: 336
        day: 1
        nsec: 0
        isdst: false
        wday: 4
        tzoffset: 180
        month: 12
        year: 2004
        hour: 0
      ...
      ```
      
      (cherry picked from commit 90552e55e0921405c43ea086ae418a72c9f000e4)
      9121eccc
  5. Aug 09, 2024
    • Maksim Tiushev's avatar
      serializer: remove duplicate of lua serializer · 1b0cc057
      Maksim Tiushev authored
      This patch fixes a bug found by the ASAN instrumentation of LuaJIT
      allocator [1]. The problem is using a Lua serializer object that has
      been cleaned up by GC.
      
      The crash occurs when executing `tarantool> \set output lua`.
      
      Failing tests:
       - ./test/app-luatest/gh_7031_configure_eos_in_lua_console_test.lua
       - ./test/app-tap/console.test.lua
       - ./test/box/push.test.lua
       - ./app-tap/console_lua.test.lua
       - ./app-luatest/varbinary_test.lua
      
      The `serializer_lua` static member is removed from
      `src/box/lua/serialize_lua.c` along with `serializer_lua_init`,
      because it is not used by any function other than `serializer_lua_init`,
      which is not needed now too.
      
      [1]: Issue #10231
      
      Closes #10177 (this issue is a duplicate of #7404)
      
      NO_TEST=rely on existing tests
      NO_CHANGELOG=codehealth
      NO_DOC=codehealth
      
      (cherry picked from commit 830f27bbaf94dadcacc66a7f543e0e5feec3988d)
      1b0cc057
    • Maksim Tiushev's avatar
      refactor: use Lua C API instead of G(L) · cfac40fb
      Maksim Tiushev authored
      To ensure better encapsulation, maintainability, and portability of
      the code, it is necessary to replace direct access to the fields of
      global structures with calls using the Lua C API.
      
      Closes #10284
      
      NO_DOC=refactoring
      NO_TEST=refactoring
      NO_CHANGELOG=refactoring
      
      (cherry picked from commit f7bb3fc7a1222107753dc420675ecda043b0a5d2)
      cfac40fb
  6. Aug 08, 2024
    • Vladimir Davydov's avatar
      vinyl: do not abort unrelated transactions on DDL · afe09076
      Vladimir Davydov authored
      Since commit 8f4be322 ("txm: disallow yields after DDL operation in
      TX"), any DDL operation aborts **all** active transactions, even those
      that wouldn't be affected by it anyway, see `memtx_engine_prepare()`,
      `memtx_tx_abort_all_for_ddl()`. Actually, there's no need to do that in
      Vinyl because it properly handles concurrent DDL operations, see commit
      d3e12369 ("vinyl: abort affected transactions when space is removed
      from cache"). Let's skip Vinyl transactions from consideration by
      marking the Vinyl engine with a special flag.
      
      Closes #10375
      
      NO_DOC=bug fix
      
      (cherry picked from commit f5f061d051dc6268949bfcb141d211142282578d)
      afe09076
    • Sergey Bronnikov's avatar
      test/fuzz: fix the CMake warning · 3f9ac0b7
      Sergey Bronnikov authored
      CMake 3.29.0 produces a warning on configuration stage:
      
      NO_WRAP
       | CMake Warning (dev) in cmake/ProtobufMutator.cmake:
       |  A logical block opening on the line
       |    /home/sergeyb/sources/MRG/tarantool/cmake/ProtobufMutator.cmake:38 (if)
       |   closes on the line
       |    /home/sergeyb/sources/MRG/tarantool/cmake/ProtobufMutator.cmake:40 (endif)
       |  with mis-matching arguments.
      NO_WRAP
      
      The patch fixes the warning.
      
      NO_CHANGELOG=build
      NO_DOC=build
      NO_TEST=build
      
      (cherry picked from commit 1e9e70f4c55c74c94922cb0beef44ae2b82255a9)
      3f9ac0b7
    • Sergey Bronnikov's avatar
      cmake: fix protobuf library name · 0e368154
      Sergey Bronnikov authored
      Protocol Buffers library has a name `protobufd` when CMAKE_BUILD_TYPE is
      equal to "Debug". In other cases the name is `protobuf`. The patch fixes
      the library name.
      
      Follows up commit b11072a6 ("cmake: add dependencies for LuaJIT and
      SQL fuzzers").
      
      Follows up #4823
      
      NO_CHANGELOG=build
      NO_DOC=build
      NO_TEST=build
      
      (cherry picked from commit c0d8cb54)
      0e368154
    • Sergey Bronnikov's avatar
      cmake: fix building statically by Ninja · c4d70795
      Sergey Bronnikov authored
      An attempt to build Tarantool statically by Ninja (with the
      enabled CMake option BUILD_STATIC_WITH_BUNDLED_LIBS), error below
      is produced:
      
       | ninja: error: build.ninja:1405: bad $-escape
       | (literal $ must be written as $$)
      
      The commit fixes that error. Follows up commit c92a1699
      ("cmake: support build using Ninja").
      
      NO_CHANGELOG=build
      NO_DOC=build
      NO_TEST=build
      
      (cherry picked from commit d48c40e69495b8cc3475ea0b184768054d87b780)
      c4d70795
    • Sergey Bronnikov's avatar
      cmake: fix warning in an ExternalProject_Add() · a1a09fdb
      Sergey Bronnikov authored
      CMake 3.24+ produces a warning about the missed option
      DOWNLOAD_EXTRACT_TIMESTAMP in `ExternalProject_Add()` [1]:
      
       | The DOWNLOAD_EXTRACT_TIMESTAMP option was not given and policy
       | CMP0135 is not set.
      
      Documentation about policy CMP0135 [2] said:
      
      CMake 3.23 and below set the timestamps of the extracted contents
      to the same as the timestamps in the archive. When the URL
      changes, the new archive is downloaded and extracted, but the
      timestamps of the extracted contents might not be newer than the
      previous contents. Anything that depends on the extracted contents
      might not be rebuilt, even though the contents may change.
      
      CMake 3.24 and above prefer to set the timestamps of all
      extracted contents to the time of the extraction. This ensures
      that anything that depends on the extracted contents will be
      rebuilt whenever the URL changes.
      
      1. https://cmake.org/cmake/help/latest/module/ExternalProject.html#url
      2. https://cmake.org/cmake/help/latest/policy/CMP0135.html
      
      NO_CHANGELOG=build
      NO_DOC=build
      NO_TEST=build
      
      (cherry picked from commit 4c0e08d85cd39916818bc4b915ac3acdacce8fe4)
      a1a09fdb
  7. Aug 07, 2024
    • Sergey Bronnikov's avatar
      datetime: fix values of unspecified fields in dt.parse() · 42afb4b7
      Sergey Bronnikov authored
      datetime object can be created using `datetime.new()` and
      `datetime.parse()`. `datetime.new()` sets values of unspecified
      fields to appropriate values of fields in a Unix time
      (00:00:00 UTC on 1 January 1971, Thursday), see commit
      43e10ed3 ("build, lua: built-in module datetime"):
      
      ```
      tarantool> datetime.new()
      ---
      - 1970-01-01T00:00:00Z
      ...
      
      tarantool> datetime.new():totable()
      ---
      - sec: 0
        min: 0
        yday: 1
        day: 1
        nsec: 0
        isdst: false
        wday: 5
        tzoffset: 0
        month: 1
        year: 1970
        hour: 0
      ```
      
      The function `datetime.parse` converts an input string with the
      date and time information into a datetime object. When fields are
      not specified in a input string their values becomes undefined:
      
      ```
      tarantool> dt.parse('01:01:01', {format ='%H:%M:%S'}):totable()
      ---
      - sec: -59
        min: -58
        yday: 366
        day: 31
        nsec: 0
        isdst: false
        wday: 1
        tzoffset: 0
        month: 12
        year: 0
        hour: -22
      ...
      
      tarantool>
      ```
      
      The commit fixes aforementioned behaviour of `datetime.parse()`
      by setting values for fields not specified by user to values of
      appropriate fields of Unix time.
      
      NO_DOC=bugfix
      
      Fixes #8588
      
      (cherry picked from commit 9ac56a1222e916951010091fbdbfc497f4a4971d)
      42afb4b7
    • Sergey Bronnikov's avatar
      datetime: remove unused function datetime_gmtoff · 10798e34
      Sergey Bronnikov authored
      The function `datetime_gmtoff()` has been introduced in commit
      aec6fbac ("datetime: implement date.isdst") for a function
      `datetime_increment_by()`. In commit 6ca07285
      ("datetime: fix interval arithmetic for DST") the function
      `datetime_increment_by()` has been updated and call of the
      function `datetime_gmtoff()` was removed. `datetime_gmtoff` is
      dead code now. The patch removes `datetime_gmtoff()`.
      
      NO_CHANGELOG=codehealth
      NO_DOC=codehealth
      NO_TEST=codehealth
      
      (cherry picked from commit 625afe418151de1015483ed24acaa57e1f85efb3)
      10798e34
    • Sergey Bronnikov's avatar
      datetime: fix typos · 2efb8cda
      Sergey Bronnikov authored
      The patch fixes a number of typos in datetime source code.
      
      NO_CHANGELOG=codehealth
      NO_DOC=codehealth
      NO_TEST=codehealth
      
      (cherry picked from commit c9c5b9f139ccbf5372d1568827fbb50bec7239bb)
      2efb8cda
  8. Aug 01, 2024
    • Georgiy Belyanin's avatar
      lua-yaml: wrap large doubles in quotes · f704c7cd
      Georgiy Belyanin authored
      Since tarantool/luajit@a16313f large exponent double strings are not
      considered convertible to number. It broke encoding lua objects to YAML
      because single quotes weren't considered necessary for decoding.
      
      This commit adds wrapping of every string containing infinite double values
      into a single quotes.
      
      Closes #10164
      
      NO_DOC=bug fix
      
      (cherry picked from commit 7c3f42590240525d2e543305b6c289ddb30054a2)
      f704c7cd
  9. Jul 31, 2024
  10. Jul 24, 2024
    • Andrey Saranchin's avatar
      memtx: allow to abort checkpoint if it wasn't started · 1bb3ee93
      Andrey Saranchin authored
      When checkpoint fails, we abort it in all engines even if it wasn't
      started successfully. If it fails right from the start so that checkpoint
      in memtx wasn't started, assertion in `memtx_engine_abort_checkpoint`
      fails - memtx doesn't expect that checkpoint will be aborted if it
      failed to start. Let's do the same thing as vinyl does - no-op if
      there is no checkpoint in progress.
      
      Closes #10265
      
      NO_CHANGELOG=reproducible only with error injection
      NO_DOC=bugfix
      
      (cherry picked from commit 6b484622259c01a2468b1f248dd6f1bcdc227021)
      1bb3ee93
    • Vladimir Davydov's avatar
      vinyl: skip bad vylog records in force_recovery mode · 551f82a1
      Vladimir Davydov authored
      We've had a number of issues when Tarantool was permanently broken
      (unable to recover after restart) because of a bad vylog record.
      The `force_recovery` mode didn't help so the user would have no other
      choice but to rebootstrap. A funny thing is those bugs were usually
      caused by a race between the garbage collector and dump/compaction when
      a vylog record was written for a dropped index. The worst thing that
      could happen if we ignored such a bad record is an unused run file not
      deleted from disk. Apparently, this is better than a permanent recovery
      failure so let's support the `force_recovery` mode in vylog.
      
      The tricky part here is handling checkpoint after restart. The problem
      is that to create a vylog checkpoint, we load the previous vylog file
      so we have to ignore errors if it was loaded in the `force_recovery`
      mode.
      
      Closes #10292
      
      NO_DOC=bug fix
      
      (cherry picked from commit c68e8a8e029d849d68c6018ed00b5a79cc769222)
      551f82a1
    • Vladimir Davydov's avatar
      vinyl: use xalloc for vylog allocations · 90be4422
      Vladimir Davydov authored
      The vinyl metadata log processor allocates its internal objects either
      from malloc or region, neither of which should fail for small
      allocations. Let's switch to xalloc to simplify the code. A good thing
      about this change is that now we can ignore all errors raised by
      vy_log_record_decode() and vy_recovery_process_record() if the
      force_recovery flag is set (see the next commit).
      
      Needed for #10292
      
      NO_DOC=refactoring
      NO_TEST=refactoring
      NO_CHANGELOG=refactoring
      
      (cherry picked from commit b6f015e98b7c30861dc33032ba7eca47de0cc198)
      90be4422
    • Vladimir Davydov's avatar
      box: use xalloc in key_def_dump_parts · 09ddc82e
      Vladimir Davydov authored
      Temporary allocations from a region don't fail so let's use xalloc to
      simplify the code.
      
      NO_DOC=refactoring
      NO_TEST=refactoring
      NO_CHANGELOG=refactoring
      
      (cherry picked from commit 28d51f8076b390c50d5b18f8e767c1eb540e5dcc)
      09ddc82e
    • Vladimir Davydov's avatar
      errinj: log error injection value · cdc59290
      Vladimir Davydov authored
      Let's log the new value when an error injection is set in orer to ease
      debugging in tests.
      
      NO_DOC=logging
      NO_TEST=logging
      NO_CHANGELOG=logging
      
      (cherry picked from commit 019bacbe)
      cdc59290
    • Ilya Verbin's avatar
      test: do not test errinj.info() output · 00e15340
      Ilya Verbin authored
      There is no much sense in testing it, but it is sensitive to source code
      changes, especially `ERRINJ_*_COUNTDOWN` injections, e.g. see commit
      697123d0 ("box: use maximal space id instead of _schema.max_id").
      
      Needed for tarantool/tarantool-ee#712
      
      NO_DOC=test
      NO_CHANGELOG=test
      
      (cherry picked from commit dc0fd81c)
      00e15340
  11. Jul 23, 2024
    • Vladimir Davydov's avatar
      vinyl: do not log dump if index was dropped · 37eea2b9
      Vladimir Davydov authored
      An index can be dropped while a memory dump is in progress. If the vinyl
      garbage collector happens to delete the index from the vylog by the time
      the memory dump completes, the dump will log an entry for a deleted
      index, resulting in an error next time we try to recover the vylog,
      like:
      
      ```
      ER_INVALID_VYLOG_FILE: Invalid VYLOG file: Run 2 committed after deletion
      ```
      
      or
      
      ```
      ER_INVALID_VYLOG_FILE: Invalid VYLOG file: Deleted range 9 has run slices
      ```
      
      We already fixed a similar issue with compaction in commit 29e2931c
      ("vinyl: fix race between compaction and gc of dropped LSM"). Let's fix
      this one in exactly the same way: discard the new run without logging it
      to the vylog on a memory dump completion if the index was dropped while
      the dump was in progress.
      
      Closes #10277
      
      NO_DOC=bug fix
      
      (cherry picked from commit ae6a02eb)
      37eea2b9
  12. Jul 22, 2024
    • Vladimir Davydov's avatar
      tuple: allocate formats table statically · bfbf5a10
      Vladimir Davydov authored
      The tuple formats table may be accessed with `tuple_format_by_id()` from
      any thread, not just tx. For example, it's accessed by a vinyl writer
      thread when it deletes a tuple. If a thread happens to access the table
      while it's being reallocated by tx, see `tuple_format_register()`,
      the accessing thread may crash with a use-after-free or NULL pointer
      dereference bug, like the one below:
      
      ```
       # 1  0x64bd45c09e22 in crash_signal_cb+162
       # 2  0x76ce74e45320 in __sigaction+80
       # 3  0x64bd45ab070c in vy_run_writer_append_stmt+700
       # 4  0x64bd45ada32a in vy_task_write_run+234
       # 5  0x64bd45ad84fe in vy_task_f+46
       # 6  0x64bd45a4aba0 in fiber_cxx_invoke(int (*)(__va_list_tag*), __va_list_tag*)+16
       # 7  0x64bd45c13e66 in fiber_loop+70
       # 8  0x64bd45e83b9c in coro_init+76
      ```
      
      To avoid that, let's make the tuple formats table statically allocated.
      This shouldn't increase actual memory usage because system memory is
      allocated lazily, on page fault. The max number of tuple formats isn't
      that big (64K) to care about the increase in virtual memory usage.
      
      Closes #10278
      
      NO_DOC=bug fix
      NO_TEST=mt race
      
      (cherry picked from commit a2da1de7)
      bfbf5a10
    • Vladislav Shpilevoy's avatar
      applier: drop apply_final_join_tx · 596d56f7
      Vladislav Shpilevoy authored
      Can use the regular applier_apply_tx(), they do the same. The
      latter is just more protective, but doesn't matter much in this
      case if the code does a few latch locks.
      
      The patch also drops an old test about double-received row panic
      during final join. The logic is that absolutely the same situation
      could happen during subscribe, but it was always filtered out by
      checking replicaset.applier.vclock and skipping duplicate rows.
      
      There doesn't seem to be a reason why final join must be any
      different. It is, after all, same subscribe logic but the received
      rows go into replica's initial snapshot instead of xlogs. Now it
      even uses the same txn processing function applier_apply_tx().
      
      The patch also moves `replication_skip_conflict` option setting
      after bootstrap is finished. In theory, final join could deliver
      a conflicting row and it must not be ignored. The problem is that
      it can't be reproduced anyhow without illegal error injection
      (which would corrupt something in an unrealistic way). But lets
      anyway move it below bootstrap for clarity.
      
      Follow-up #10113
      
      NO_DOC=refactoring
      NO_CHANGELOG=refactoring
      
      (cherry picked from commit da158b9b)
      596d56f7
    • Vladislav Shpilevoy's avatar
      box: make instance_vclock const · a62da4ee
      Vladislav Shpilevoy authored
      No code besides box.cc can now update instance's vclock
      explicitly. That is a protection against hacks like #9916.
      
      Closes #10113
      
      NO_DOC=refactoring
      NO_TEST=refactoring
      NO_CHANGELOG=refactoring
      
      (cherry picked from commit 19b2cc20)
      a62da4ee
    • Vladislav Shpilevoy's avatar
      box: make final join vclock update only in box.cc · 15f4482c
      Vladislav Shpilevoy authored
      The goal is to make sure that no files except box.cc can change
      instance_vclock_storage directly. That leads to all sorts of hacks
      which in turn lead to bugs - #9916 is a good example.
      
      Now applier on final join only sends rows into the journal. The
      journal then is handled by box.cc where vclock is properly
      updated.
      
      Part of #10113
      
      NO_DOC=refactoring
      NO_TEST=refactoring
      NO_CHANGELOG=refactoring
      
      (cherry picked from commit fe338ed4)
      15f4482c
    • Vladislav Shpilevoy's avatar
      journal: extract journal_write_row from limbo · 972d909b
      Vladislav Shpilevoy authored
      The function writes a single xrow into the journal in a blocking
      way. It isn't so simple, so makes sense to keep as a function,
      especially given that it will be used more in the next commit.
      
      Part of #10113
      
      NO_DOC=refactoring
      NO_TEST=refactoring
      NO_CHANGELOG=refactoring
      
      (cherry picked from commit 7d10096c)
      972d909b
    • Vladislav Shpilevoy's avatar
      box: move recovery_journal creation · f4438449
      Vladislav Shpilevoy authored
      Recovery journal uses word "recovery" to say that it works with
      xlogs. For snapshot recovery there is bootstrap_journal. Lets use
      it during local snapshot recovery.
      
      The reasoning is that while right now there is no difference, in
      next commits the recovery_journal will do more.
      
      Part of #10113
      
      NO_DOC=refactoring
      NO_TEST=refactoring
      NO_CHANGELOG=refactoring
      
      (cherry picked from commit 2620eb9e)
      f4438449
Loading