Skip to content
Snippets Groups Projects
  1. Jan 29, 2024
    • Nikolay Shirokovskiy's avatar
      box: finish client fibers on shutdown · bf620650
      Nikolay Shirokovskiy authored
      In the process of graceful shutdown it is convenient to first finish
      all client (non system) fibers. Otherwise we should be ready for any
      subsystem to handle request from client fiber during or after subsystem
      shutdown. This would make code more complex.
      
      We first cancel client fibers and then wait for their finishing. The
      fiber may not respond to cancel and hang which cause shutdown hang
      but this is the approach we choose for iproto shutdown already.
      
      Note that as a result of this approach application will panic if
      it is shutdown during execution of initialization script (in
      particular if this script is doing box.cfg).
      
      There are changes in application/test to adopt to client fibers
      shutdown:
      
      - make code cancellable (only to pass existing tests, we did not
        investigate all the possible places that should be made such).
      
      - make console stop sending echo to client before client fibers
        shutdown. Otherwise as console server fiber is client one we will send
        message that fiber is cancelled on shutdown which breaks a lot of
        existing tests. This approach is on par with iproto shutdown.
      
      - some tests (7743, replication-luatest/shutdown, replication/anon,
        replication/force_recovery etc etc) test shutdown during execution of
        init script. Now panic is expected so change them accordingly.
      
      - some tests (8530, errinj_vylog) use injection that block client
        fiber finishing. In that tests we don't need graceful shutdown so
        let's just kill tarantool instead.
      
      - we change test in vinyl/errinj for gh-3225. We don't really need
        to check when vinyl reader is blocked as it executes small tasks
        (we assume reading syscall will not hang). Also change test for
        vinyl dump shutdown by slowing dump down instead of blocking it
        entirely. This is required to finish in time client fibers in
        the test.
      
      - other similar changes
      
      Also we can drop code from replication shutdown which is required to
      handle client requests during/after shutdown.
      
      Part of #8423
      
      NO_CHANGELOG=internal
      NO_DOC=internal
      bf620650
    • Pavel Balaev's avatar
      tools: remove tarantoolctl utility · 216b6243
      Pavel Balaev authored
      tarantoolctl has been removed. Systemd, sysvinit and logrotate scripts
      based on it were also removed. All this functionality is covered by the
      tt utility.
      
      Closes #9443
      
      @TarantoolBot document
      Title: utils: tarantoolctl has been removed.
      
      Tarantoolctl has been removed. You now need to use the `tt` utility.
      Documentation for the `tt` utility can be found at
      [tarantool.io](https://www.tarantool.io/ru/doc/latest/reference/tooling/tt_cli/)
      216b6243
    • Yaroslav Lobankov's avatar
      ci: update matrix strategy for osx testing · 47940311
      Yaroslav Lobankov authored
      It has been decided to have only release and debug builds for testing
      on two latest macOS versions (at this momemnt, 13 and 14). Static and
      LTO builds are dropped.
      
      NO_DOC=ci
      NO_TEST=ci
      NO_CHANGELOG=ci
      47940311
  2. Jan 26, 2024
  3. Jan 25, 2024
    • Sergey Kaplun's avatar
      test: fix fix_lua_loader_error_handling on macOS · 3fcd83ea
      Sergey Kaplun authored
      This patch is a follow-up for the commit
      b99a3541 ("lua: fix loader behaviour in
      case of syntax error").
      
      This test uses full matching of the error message and assumes that the
      temporary directory is short enough, so the full patch is less than 60
      symbols (size of `lua_Debug.short_src`). But for macOS, the temporary
      directory has a longer path, so the error message is shrunk and isn't
      equal to the expected string.
      
      This patch replaces full matching with matching of the last part of the
      error message.
      
      Relates to tarantool/tarantool-ee#585
      
      NO_DOC=test fix
      NO_CHANGELOG=test fix
      3fcd83ea
    • Ilya Verbin's avatar
      box: allow indexing of fixed-size numeric field types · 38b4915c
      Ilya Verbin authored
      The types added by commit 2e0f9a3c ("box: introduce fixed-size numeric
      field types") can't be indexed. Fix it.
      
      Perf results for `tuple_compare' benchmark with FIELD_TYPE_INTEGER:
      
      NO_WRAP
      $ taskset 0x2 ~/benchmark/tools/compare.py benchmarks \
        ./tuple.perftest.ref ./tuple.perftest.exp \
        --benchmark_min_warmup_time=10 \
        --benchmark_repetitions=30 \
        --benchmark_report_aggregates_only=true \
        --benchmark_filter=tuple_compare$
      [...]
      Comparing ./tuple.perftest.ref to ./tuple.perftest.exp
      Benchmark                                    Time             CPU      Time Old      Time New       CPU Old       CPU New
      -------------------------------------------------------------------------------------------------------------------------
      tuple_tuple_compare_mean                  +0.0158         +0.0158            27            27            27            27
      tuple_tuple_compare_median                +0.0153         +0.0152            27            27            27            27
      tuple_tuple_compare_stddev                +0.0042         +0.0039             1             1             1             1
      tuple_tuple_compare_cv                    -0.0114         -0.0117             0             0             0             0
      NO_WRAP
      
      Closes #9611
      
      NO_DOC=bugfix
      NO_CHANGELOG=unreleased
      38b4915c
  4. Jan 24, 2024
    • Igor Munkin's avatar
      lua: implement searchers-to-loaders mapping · a11cb66d
      Igor Munkin authored
      This patch finalizes the refactoring of Tarantool package.loaders
      required for Tarantool integrity protection.
      
      Finally, to make it possible to instrument the searchers with Tarantool
      integrity protection, two steps are left.
      * Implement mapping from searchers table to <package.loaders> using
        <gen_legacy_loader> helper.
      * Obtain the searcher from the table directly, so the runtime
        instrumentation takes the effect.
      
      As a result, <gen_legacy_loader> is reimplemented to use the table with
      searchers as an upvalue instead of the particular searcher routine, and
      <package.loaders> table is filled by the function generated with the
      updated helper. Furthermore, the aforementioned table with searchers is
      added to 'internal.loaders' module exports to make the instrumentation
      more convenient for users.
      
      Needed for tarantool/tarantool-ee#585
      
      NO_DOC=refactoring
      NO_TEST=refactoring
      NO_CHANGELOG=refactoring
      a11cb66d
    • Igor Munkin's avatar
      lua: implement common legacy loader generator · 38803a4c
      Igor Munkin authored
      This patch continues the refactoring of Tarantool package.loaders
      required for Tarantool integrity protection.
      
      Though all Lua 5.1 loaders machinery will be superseded by Lua 5.4
      searchers machinery, Tarantool still should follow Lua 5.1 semantics, so
      the "legacy" loader is required. This patch implements the common helper
      <gen_legacy_loader> to wrap all loaders and make the new machinery
      conform the Lua 5.1 semantics.
      
      Moreover, since all tweaks and helpers are implemented for searching
      machinery, <chain_loaders> is not required anymore.
      
      Needed for tarantool/tarantool-ee#585
      
      NO_DOC=refactoring
      NO_TEST=refactoring
      NO_CHANGELOG=refactoring
      38803a4c
    • Igor Munkin's avatar
      lua: implement croot searcher · dbfe4190
      Igor Munkin authored
      This patch continues the refactoring of Tarantool package.loaders
      required for Tarantool integrity protection.
      
      Though croot loader is not the popular way to work with Lua C libraries,
      some modules are organised using this mechanism, so they have to be
      verified via Tarantool integrity protection system, hence, croot loader
      has to be reimplemented to the new searching machinery.
      
      The corresponding croot searcher is implemented in this patch and added
      to the indexed part of the <package.searchers> table.
      
      It's worth to mention two peculiarities related to the croot loader and
      searcher:
      
      * Since Tarantool uses the original croot loader from LuaJIT, it ignores
        all paths except ones specified in <package.cpath>. However, there
        might be some croot-like modules in the Tarantool-specific cpaths (cwd
        or application root). It looks like croot machinery is not widely
        used, if this bug bothers nobody, but it still needs to be fixed.
      * A similar situation relates to the overriding mechanism that was
        recently introduced: there is no "prefixed" croot loader and all
        libraries using croot loading machinery are ignored for the overridden
        paths as a result.
      
      Needed for tarantool/tarantool-ee#585
      
      NO_DOC=refactoring
      NO_TEST=refactoring
      NO_CHANGELOG=refactoring
      dbfe4190
    • Igor Munkin's avatar
      lua: implement conditional searchers · 583d7462
      Igor Munkin authored
      This patch continues the refactoring of Tarantool package.loaders
      required for Tarantool integrity protection.
      
      Within this patch, the mechanism for enabling/disabling loaders
      introduced in the commit 87b4da31 ("lua:
      enable/disable override loader") is moved to the "searching" phase of
      the new loading machinery.
      
      Needed for tarantool/tarantool-ee#585
      
      NO_DOC=refactoring
      NO_TEST=refactoring
      NO_CHANGELOG=refactoring
      583d7462
    • Igor Munkin's avatar
      lua: implement prefixed searchers · 28e93ec9
      Igor Munkin authored
      This patch continues the refactoring of Tarantool package.loaders
      required for Tarantool integrity protection.
      
      Within this patch, the overriding mechanism introduced in the commit
      ec47c389 ("lua: allow to override a
      built-in module") is moved to the "searching" phase of the new loading
      machinery. Furthermore, package.path/package.cpath searchers are also
      added to the indexed list made for the mapping.
      
      Needed for tarantool/tarantool-ee#585
      
      NO_DOC=refactoring
      NO_TEST=refactoring
      NO_CHANGELOG=refactoring
      28e93ec9
    • Igor Munkin's avatar
      lua: implement chained searchers · 04495751
      Igor Munkin authored
      This patch continues the refactoring of Tarantool package.loaders
      required for Tarantool integrity protection.
      
      Considering the way "app" paths injection is implemented in Tarantool
      loaders to save the loading priority, the chained loaders machinery
      introduced in the commit 7e9051c4 ("lua:
      don't set built-in modules to package.loaded") is also implemented for
      the new searching mechanism. The approach is the same as the one used
      for chaining loaders. As a result, "app" paths searchers are injected
      into the "cwd" paths searchers the same way it has been done in the
      original loaders context.
      
      This patch also introduces numerical indices into <package.searchers>
      analogue to provide the convenient searchers <-> loaders mapping used in
      the final patch of this series.
      
      Needed for tarantool/tarantool-ee#585
      
      NO_DOC=refactoring
      NO_TEST=refactoring
      NO_CHANGELOG=refactoring
      04495751
    • Igor Munkin's avatar
      lua: implement searchers for file modules · fd1fb659
      Igor Munkin authored
      This patch continues the refactoring of Tarantool package.loaders
      required for Tarantool integrity protection.
      
      Originally, Lua uses only constant path patterns in package.path and
      package.cpath. Tarantool enhances the user experience here and provides
      more relaxed rules for module searching: some paths are calculated at
      the moment the module is being searched and are relative to the current
      working directory. Unfortunately, it makes "searchers" more complex,
      since all the paths can't be calculated when Tarantool starts up.
      
      Furthermore, there is the particular priority while loading Lua modules
      in Tarantool. Originally, Lua tries to load the particular Lua script
      using the hints from package.path and *only if* no Lua module is found,
      Lua tries to load the corresponding Lua C library using the hints from
      package.cpath. Tarantool loaders are prioritized by the locations at
      first and only then by the module implementation. This also makes
      "searchers" structure more complex than it could be.
      
      Anyway, "file" loaders are split into the three layers:
      1. "Pathogen": the function generated by <gen_path_builder>, building
         the hints for searchers
      2. "Searcher": the function generated by <gen_file_searcher> helper,
         using the particular file loading method (<load_lua> or <load_lib>)
         and the given pathogen to obtain the hints with path patterns to be
         used in <package.searchpath> routine
      3. "Loader": the function generated by <gen_file_loader> helper,
         following Lua 5.1 loaders semantics and using the particular searcher
      
      All the generated searchers are stored in the <package.searchers>
      analogue introduced in the first patch of this refactoring series.
      
      It's worth saying that "app" paths injection is still implemented via
      chained loaders, but this machinery will be reimplemented within the
      following patches in this series.
      
      Needed for tarantool/tarantool-ee#585
      
      NO_DOC=refactoring
      NO_TEST=refactoring
      NO_CHANGELOG=refactoring
      fd1fb659
    • Igor Munkin's avatar
      lua: implement searchers for preload and builtin · 6b8daa37
      Igor Munkin authored
      This patch initiates the refactoring of Tarantool package.loaders
      required for Tarantool integrity protection.
      
      Since Tarantool follows Lua 5.1[1] semantics, the set of routines in
      package.loaders is used to tweak Lua modules loading. E.g. overriding
      machinery has been implemented on top of the package.loaders.
      Unfortunately, package.loaders are not as flexible as they should be and
      this mechanism is redesigned in Lua 5.2[2].
      
      Starting from Lua 5.2 package.loaders has been superseded by
      package.searchers. This is quite a natural change, since nobody wants to
      tweak the "loading" rules, but rather the "searching" process. As a
      result of this redesign evolution up to the Lua 5.4[3], users are able
      to obtain the "source" of the module being loaded (i.e. file name or
      specific sentinel for preload packages).
      
      This patch introduces Tarantool internal table with package.searchers
      and reimplements package.preload and tarantool.builtin loaders using the
      machinery described above:
      * The new searcher function, respecting Lua 5.4 semantics, is
        implemented for each loader.
      * Each searcher is wrapped with the function to implement "dummy" loader
        respecting Lua 5.1 semantics.
      
      [1]: https://www.lua.org/manual/5.1/manual.html
      [2]: https://www.lua.org/manual/5.2/manual.html
      [3]: https://www.lua.org/manual/5.4/manual.html
      
      Needed for tarantool/tarantool-ee#585
      
      NO_DOC=refactoring
      NO_TEST=refactoring
      NO_CHANGELOG=refactoring
      6b8daa37
    • Igor Munkin's avatar
      lua: fix loader behaviour in case of syntax error · b99a3541
      Igor Munkin authored
      The patch fixes error handling between module "searching" and "loading"
      processes. Originally, if a module is not found, the error message is
      returned to <require> and is appended to the "not found" message. If no
      suitable module is found, the error is raised by <require>. In case,
      when the desired module is found, <require> tries to load it. If any
      error occurs while loading the module, the "loader" yields the
      corresponding error. The latter part was broken since the commit
      5be3a82b ("lua: fix loaders behavior"),
      so if the error occurs while loading the module, Tarantool proceeds with
      searching the module in other locations.
      
      This bug was found via the lua-Harness suite[1] used in the
      tarantool/luajit testing routine, so the test added in the patch
      partially duplicates the particular test case in lua-Harness.
      
      [1]: https://fperrad.frama.io/lua-Harness/
      
      Needed for tarantool/tarantool-ee#585
      
      NO_DOC=bugfix
      b99a3541
    • Yaroslav Lobankov's avatar
      ci: fix job name in freebsd failure report · 4007552a
      Yaroslav Lobankov authored
      The ChristopherHX/github-act-runner agent evaluates expression
      `join(matrix.*, ', ')` to `Object` instead of `a, b, c`. So let's
      have the facility to specify the job name manually to avoid the issue.
      
      NO_DOC=ci
      NO_TEST=ci
      NO_CHANGELOG=ci
      4007552a
  5. Jan 23, 2024
    • Yaroslav Lobankov's avatar
      ci: fix fail on artifact upload to S3 for freebsd · 9717c31f
      Yaroslav Lobankov authored
      This patch fixes the following issue:
      
          Error: Unable to retrieve job ID by provided job name
      
      The `${{ github.job }} (${{ join(matrix.*, ', ') }})` expression was
      evaluated to `freebsd (Object)` instead of proper job name. It looks
      like the ChristopherHX/github-act-runner agent has a bug somewhere.
      Now the job name is made manually to bypass the issue.
      
      NO_DOC=ci
      NO_TEST=ci
      NO_CHANGELOG=ci
      9717c31f
    • Sergey Bronnikov's avatar
      httpc: fix reading data in a chunked request · 9bdf2bab
      Sergey Bronnikov authored
      There was a problem: chunked request to etcd via TLS using builtin http
      client returns nothing. The problem was reproduced quite often by a test
      `integration.general.tls` in etcd-client Lua module. The patch updates
      curl submodule to a version based on 8.5.0 release with applied patch
      with fix [1].
      
      The patch also removes workaround for aforementioned problem made in
      commit aa58c212 ("httpc: use http 1.1 by default").
      
      1. https://github.com/curl/curl/commit/cdd905a9854305657ebbe645095e1189dcda28c7
      
      Fixes #9547
      
      NO_TEST=etcd-client tests
      NO_DOC=bugfix
      9bdf2bab
  6. Jan 18, 2024
    • Ilya Verbin's avatar
      Bump msgpuck submodule · f58bfc97
      Ilya Verbin authored
      This update pulls the following commit:
      
      * Introduce mp_read_int8 and mp_read_int16 functions
      
      Needed for tarantool/tarantool-ee#627
      
      NO_DOC=internal
      NO_CHANGELOG=internal
      NO_TEST=submodule update
      f58bfc97
    • Nikolay Shirokovskiy's avatar
      test: fix gh-2717-no-quit-sigint test · b4b9cc93
      Nikolay Shirokovskiy authored
      - First endswith returns boolean so fix where it compared with nil.
        Due to the invalid comparison we actually don't check for prompt
        in nested case.
      - Add trailing space to prompt in this test case.
      - Reset readline configuration in this test case.
      
      NO_CHANGELOG=test fix
      NO_DOC=test fix
      b4b9cc93
  7. Jan 17, 2024
    • Vladimir Davydov's avatar
      space: optimize lookup by id · e06b866a
      Vladimir Davydov authored
      In contrast to space_cache_find(), space_by_id() doesn't cache the last
      looked up space to speed up following lookups. This is confusing,
      because one would expect the only difference between space_cache_find()
      and space_by_id() functions to be that the former sets diag while the
      latter doesn't. Let's move the prev_space cache to space_by_id() to fix
      this issue.
      
      To achieve that we need to make space_by_id() inline, which is tricky
      because it's called via FFI. We work around that by defining a macro
      space_by_id() which expands to space_by_id_fast() while still keeping
      the space_by_id() symbol for FFI.
      
      Last but not least, we make the last cached space global (currently it's
      instantiated once per each space_cache_find() call site). This is a good
      thing because we often call space_by_id() a few times while processing
      the same DML request (e.g. in memtx mvcc).
      
      NO_DOC=no user-visible behavior changes
      NO_TEST=no user-visible behavior changes
      NO_CHANGELOG=no user-visible behavior changes
      e06b866a
    • Yaroslav Lobankov's avatar
      ci: drop testing for FreeBSD 12, add for FreeBSD 14 · 9ee99c5f
      Yaroslav Lobankov authored
      - Drop testing for FreeBSD 12 since FreeBSD 14 is available
      - Add testing for FreeBSD 14
      
      NO_DOC=ci
      NO_TEST=ci
      NO_CHANGELOG=ci
      9ee99c5f
  8. Jan 16, 2024
    • Ilya Verbin's avatar
      box: introduce fixed-size numeric field types · 2e0f9a3c
      Ilya Verbin authored
      See the docbot request for details.
      
      Closes #9548
      
      @TarantoolBot document
      Title: Document fixed-size numeric field types
      Product: Tarantool
      Since: 3.1
      Root document: https://www.tarantool.io/en/doc/latest/concepts/data_model/value_store/#field-type-details
      
       ### Fixed-size field types
      
      Tarantool supports the following fixed-size numeric field types:
      
       * `uint8`: integer in range [0 .. 255]
       * `int8`: integer in range [-128 .. 127]
       * `uint16`: integer in range [0 .. 65,535]
       * `int16`: integer in range [-32,768 .. 32,767]
       * `uint32`: integer in range [0 .. 4,294,967,295]
       * `int32`: integer in range [-2,147,483,648 .. 2,147,483,647]
       * `uint64`: integer in range [0 .. 18,446,744,073,709,551,615]
       * `int64`: integer in range [-9,223,372,036,854,775,808 ..
                                     9,223,372,036,854,775,807]
       * `float32`: 32-bit floating point number
       * `float64`: 64-bit floating point number
      
      These types don't affect how data is stored in Tarantool's memtx or vinyl
      engines but they enforce the limits when a tuple is inserted or updated.
      2e0f9a3c
    • Ilya Verbin's avatar
      box: rewrite field_type1_contains_type2() · fe4f0223
      Ilya Verbin authored
      All the information needed by this function is stored in `field_mp_type[]'
      and `field_ext_type[]'. Get rid of redundant `field_type_compatibility[]',
      which is hard to extend and which can introduce additional bugs.
      
      NO_DOC=refactoring
      NO_CHANGELOG=refactoring
      fe4f0223
    • Ilya Verbin's avatar
      box: fix field_type1_contains_type2(FIELD_TYPE_DOUBLE, ...) · 3ffefe3b
      Ilya Verbin authored
      commit 51af059c ("box: compare and hash msgpack value of double key
      field as double") changed `field_mp_type[FIELD_TYPE_DOUBLE]', but the
      corresponding column of `field_type_compatibility[]' wasn't updated.
      
      Now a field of type DOUBLE can store MP_UINT and MP_INT, i.e. it can
      store any value from fields of type UNSIGNED, NUMBER, INTEGER.
      In fact, FIELD_TYPE_DOUBLE is now an alias for FIELD_TYPE_NUMBER.
      
      It's hard to create a test, because e.g. during CheckSpaceFormat even if
      field_type1_contains_type2() erroneously returns false, then each tuple is
      correctly validated anyway. So this patch actually fixes performance issue.
      
      NO_TEST=see above
      NO_DOC=minor fix
      NO_CHANGELOG=minor fix
      3ffefe3b
  9. Jan 15, 2024
    • Vladimir Davydov's avatar
      space: drop space upgrade state ref counting · 87627322
      Vladimir Davydov authored
      Not used since we moved result processing to the memtx engine.
      
      NO_DOC=refactoring
      NO_TEST=refactoring
      NO_CHANGELOG=refactoring
      87627322
    • Vladimir Davydov's avatar
      index: drop result processor used for space upgrade · 5ce9a371
      Vladimir Davydov authored
      Initially, we tried to upgrade tuples in the engine-independent code,
      for which purpose we added the result processor, see commit 21e2def9
      ("box: introduce result processor"). It turned out, we still need to do
      some upgrade-related work in the memtx engine, see commit f7a520ef
      ("box: don't apply space upgrade func to DML result except for DELETE")
      and commit 921a0717 ("memtx: add stubs to keep track of upgraded
      read view tuples"). Since the memtx engine is the only engine that
      currently supports space upgrade, let's simplify the code by moving
      space upgrade handling completely in the memtx engine. This is easy to
      do because we already have memtx_prepare_result_tuple() helper that
      decompresses tuples before returning them to the user, which seems to be
      the perfect place for handling upgrade.
      
      NO_DOC=refactoring
      NO_TEST=refactoring
      NO_CHANGELOG=refactoring
      5ce9a371
  10. Jan 12, 2024
    • Vladimir Davydov's avatar
      test: make wal_off/alter robust against system tuple format count · 57b90288
      Vladimir Davydov authored
      We have to update the test every time we add a new system tuple format
      because it checks the number of spaces that can be created before the
      tuple format count limit is reached. Let's loosen the space count check
      to avoid that.
      
      We do this for the new memcs engine, which will be introduced to
      Tarantool Enterprise Edition, because it will create a tuple format to
      serve internal needs.
      
      Needed for tarantool/tarantool-ee#617
      
      NO_DOC=test fix
      NO_CHANGELOG=test fix
      57b90288
    • Vladimir Davydov's avatar
      index: add read view iterator destructor · f224b7ed
      Vladimir Davydov authored
      We need it for the memcs engine. The generic implementation used by
      other engines just trashes the iterator object.
      
      Needed for tarantool/tarantool-ee#617
      
      NO_DOC=internal
      NO_TEST=internal
      NO_CHANGELOG=internal
      f224b7ed
    • Vladimir Davydov's avatar
      box: add memcs engine initialization stub · 1112b5d8
      Vladimir Davydov authored
      The stub will be implemented in Tarantool Enterprise Edition.
      
      Needed for tarantool/tarantool-ee#617
      
      NO_DOC=stub
      NO_TEST=stub
      NO_CHANGELOG=stub
      1112b5d8
    • Vladimir Davydov's avatar
      engine: let other engines reuse memtx checkpoint and join · fd44364a
      Vladimir Davydov authored
      The implementation of the checkpoint and join memtx engine methods
      is actually generic: all they need is a read view. Let's allow
      other engines that support creation of a consistent read view to be
      checkpointed and joined by the memtx engine by setting the corresponding
      engine flags.
      
      Currently, there's no such engines, i.e. the only engine setting the new
      flags is memtx itself. We will use this feature for new engines.
      
      Needed for tarantool/tarantool-ee#617
      
      NO_DOC=internal
      NO_TEST=internal
      NO_CHANGELOG=internal
      fd44364a
    • Nikolay Shirokovskiy's avatar
      replication: replication graceful shutdown · 8d7af954
      Nikolay Shirokovskiy authored
      This implies finishing replication TX fibers and stopping applier
      threads. This is easy to do using existing applier_stop.
      
      We also need to make sure that there is no client fibers are in
      replication code after shutdown. Otherwise we may have difficulties
      (assertions) while freeing replication resources. This goal have two
      sides. First we have to finish client fibers waiting in replication code
      and second we should not allow to wait after shutdonw is done.
      
      Here we probably can achieve first side by just stopping appliers. But in
      this case client will get error other than FiberIsCancelled which is
      nice to have. So approach is to track client fibers in replication code
      and cancel them on shutdown. This approach is also aligned with
      iproto/relay shutdown.
      
      There is issue with graceful replication shutdown though. A good example
      (on which `replication/shutdown_test.lua` is based) is bootstrapping
      a replica with wrong auth in replication URI. In this case applier is
      sleeping in reconnect delay and bootstrap code waiting for READY state.
      Now comes server shutdown.
      
      Applier is stopped during shutdown and we hit assertion [1]. The issue
      is we miss bootstrap fiber notification that applier fiber is cancelled.
      That's why the change with `fiber_testcancel` in `applier_f`.
      
      We also drop the assertion in the `replica_on_applier_sync` because
      applier can switch to OFF state from any previous state if we cancel
      applier fiber.
      
      Part of #8423
      
      [1] Issue assertion stack:
      ```
        #5  0x00007fe877a54d26 in __assert_fail (
          assertion=0x5637b683b07c "fiber() == applier->fiber",
          file=0x5637b683a03e "./src/box/applier.cc", line=2809,
          function=0x5637b683b05f "void applier_pause(applier*)") at assert.c:101
        #6  0x00005637b62f0f20 in applier_pause (applier=0x5637b7a87a60)
          at /home/shiny/dev/tarantool/src/box/applier.cc:2809
        #7  0x00005637b62f104b in applier_on_state_f (trigger=0x7fe877380a60,
          event=0x5637b7a87a60) at /home/shiny/dev/tarantool/src/box/applier.cc:2845
        #8  0x00005637b645d2e3 in trigger_run_list (list=0x7fe877280de0,
          event=0x5637b7a87a60) at /home/shiny/dev/tarantool/src/lib/core/trigger.cc:100
        #9  0x00005637b645d401 in trigger_run (list=0x5637b7a88098, event=0x5637b7a87a60)
          at /home/shiny/dev/tarantool/src/lib/core/trigger.cc:133
        #10 0x00005637b62e4d6b in trigger_run_xc (list=0x5637b7a88098, event=0x5637b7a87a60)
          at /home/shiny/dev/tarantool/src/lib/core/trigger.h:173
        #11 0x00005637b62e703a in applier_set_state (applier=0x5637b7a87a60,
          state=APPLIER_OFF) at /home/shiny/dev/tarantool/src/box/applier.cc:83
        #12 0x00005637b62f0ab3 in applier_stop (applier=0x5637b7a87a60)
          at /home/shiny/dev/tarantool/src/box/applier.cc:2749
        #13 0x00005637b62dc189 in replication_shutdown ()
      ```
      
      NO_CHANGELOG=internal
      NO_DOC=internal
      8d7af954
    • Nikolay Shirokovskiy's avatar
      appiler: do not keep applier diag in applier fiber diag · 307ef45c
      Nikolay Shirokovskiy authored
      Currently last applier error is keeped in fiber diag. This leads to some
      hacks like the return of -1 from applier fiber to keep the diag (fiber
      diag is cleared if fiber return value is 0). Also we do not use applier
      fiber diag as it intended - as diag set to current fiber on applier fiber
      join. Let's instead keep applier diag in a different place.
      
      Also let's copy error in `applier_wait_for_state` so that fiber error is
      not disappeared in case of bootstrap.
      
      Part of #8423
      
      NO_TEST=refactoring
      NO_CHANGELOG=refactoring
      NO_DOC=refactoring
      307ef45c
  11. Jan 11, 2024
    • Vladimir Davydov's avatar
      index: introduce weak reference · a76eb242
      Vladimir Davydov authored
      An iterator doesn't pin the index it was created for. Instead, it holds
      a 'weak' reference to it, which is invalidated if the index is dropped.
      This is achieved by checking if the index can be found in the space
      cache by id every time the reference is accessed. Let's factor out this
      weak reference concept into a separate class so that we can reuse it
      because we'll also need it to implement the index scanner API.
      
      Needed for #9568
      
      NO_DOC=refactoring
      NO_TEST=refactoring
      NO_CHANGELOG=refactoring
      a76eb242
    • Yaroslav Lobankov's avatar
      ci: drop testing for macOS 12, add for macOS 14 · 19f9786f
      Yaroslav Lobankov authored
      - Drop testing for macOS 12 since macOS 14 is available
      - Add testing for macOS 14
      
      NO_DOC=ci
      NO_TEST=ci
      NO_CHANGELOG=ci
      19f9786f
    • Yaroslav Lobankov's avatar
      ci: rm workarounds from `install-deps-osx` action · 3142fb66
      Yaroslav Lobankov authored
      The issue [1] is closed, and it looks like the problem doesn't exist
      now. So removing the respective workaround from the action.
      
      Also, removing the workaround with upgrading packages as it is not
      needed anymore because now `brew` just gives a warning instead of
      failing when the package is already installed.
      
      [1] https://github.com/gevent/gevent/issues/1721
      
      NO_DOC=ci
      NO_TEST=ci
      NO_CHANGELOG=ci
      3142fb66
    • Yaroslav Lobankov's avatar
      ci: ignore fail on artifact upload to S3 for osx · 5137ea59
      Yaroslav Lobankov authored
      From time to time, the step to upload job artifacts to S3 fails due to
      the following error:
      
          Error: Unable to retrieve job ID by provided job name
      
      It happens in the `s3-upload-artifact` action that tries to get job ID
      by the name via the `get-job-id` action. Job ID is needed to compose the
      proper artifact name. `get-job-id` performs a request to GitHub for the
      list of workflow jobs and finds the job by its name and then gets its
      ID.
      
      For some reason, sometimes GitHub doesn't give the complete job list and
      there is no chance to find the job and get its ID. At least, it has been
      noticed for the osx.yml workflow with 36 parallel jobs.
      
      It looks like the issue resides on the GitHub side and the simplest way
      to resolve this is to ignore the failure and not fail the whole workflow
      due to this GitHub bug.
      
      NO_DOC=ci
      NO_TEST=ci
      NO_CHANGELOG=ci
      5137ea59
    • Yaroslav Lobankov's avatar
      ci: fix uploading build artifacts to S3 for freebsd · 5fa3d5a2
      Yaroslav Lobankov authored
      When the `freebsd` workflow has become a matrix one in 0bd15468
      ("ci: mv freebsd testing from per-commit to nightly"), we forgot
      to do the corresponding changes in the `Upload artifacts to S3` step.
      So adding missing changes.
      
      NO_DOC=ci
      NO_TEST=ci
      NO_CHANGELOG=ci
      5fa3d5a2
    • Yaroslav Lobankov's avatar
      ci: fix saving build artifacts for osx and freebsd · 64363b43
      Yaroslav Lobankov authored
      When `${{ matrix.tarantool-branch}}` is `release/3.0` or similar, the
      step to save build artifacts will fail with the following error:
      
          Error: Artifact name is not valid: osx-13-x86_64-release/3.0-debug.
          Contains the following character:  Forward slash /
      
      This commit fixes the issue.
      
      NO_DOC=ci
      NO_TEST=ci
      NO_CHANGELOG=ci
      64363b43
    • Sergey Kaplun's avatar
      test: fix flaky tests when JIT is enabled · 1a5e3bf3
      Sergey Kaplun authored
      <box/net.box_reconnect_after_gh-3164.test.lua> and
      <vinyl/tx_gap_lock.test.lua> checking some objects that should be
      collected may fail because these objects are referenced as the upvalues
      for a function inlined on some JIT trace. The objects aren't leaked;
      tests don't consider the JIT semantics. See more details in [1].
      
      This patch disables JIT for the first test.
      
      The second test itself isn't affected since the commit
      tarantool/test-run@33fbace8307e64e66b0b2178ac952b2697673917 ("Add
      default server restart to each test run"). Because of this commit, the
      `tx.gap_locks` can't be affected by the use of the `create_iterator()`
      function in other tests.
      But to avoid similar situations in the future, the JIT is disabled for
      the `res.next()` function in test utils.
      
      [1]: https://github.com/tarantool/tarantool/wiki/LuaJIT-function-inlining
      
      Closes tarantool/tarantool-qa#233
      Closes tarantool/tarantool-qa#276
      
      NO_DOC=fix flaky test
      NO_CHANGELOG=see NO_DOC
      1a5e3bf3
Loading