Skip to content
Snippets Groups Projects
  1. Sep 15, 2022
    • Georgiy Lebedev's avatar
      memtx: fix 'use after free' of garbage collected MVCC stories · 18e042f5
      Georgiy Lebedev authored
      `directly_replaced` stories can potentially get garbage collected in
      `memtx_tx_handle_gap_write`, which is unexpected and leads to 'use after
      free': in order to fix this, limit garbage collection points only to
      external API calls.
      
      Wrap all possible garbage collection points with explicit warnings (see
      c9981a56).
      
      Closes #7449
      
      NO_DOC=bugfix
      18e042f5
  2. Sep 14, 2022
  3. Sep 13, 2022
    • Georgiy Lebedev's avatar
      memtx: track read story when conflicting full scans due to gap write · 7f52f445
      Georgiy Lebedev authored
      When conflicting transactions that made full scans in
      `memtx_tx_handle_gap_write`, we need to also track that the conflicted
      transaction has read the inserted tuple, just like we do in gap tracking
      for ordered indexes — otherwise another transaction can overwrite the
      inserted tuple in which case no gap tracking will be handled.
      
      Closes #7493
      
      NO_DOC=bugfix
      7f52f445
    • Georgy Moshkin's avatar
      fiber: add fiber_set_ctx & fiber_get_ctx functions · 6a60768c
      Georgy Moshkin authored
      Before this change there was no way to create a fiber that accepts
      parameters without yielding from the current fiber using the c api. You
      could pass the function arguments when calling fiber_start, but that
      forces you to yield, which is not acceptable in some scenarios (e.g.
      within a transaction).
      
      This commit introduces 2 new functions to the api: fiber_set_ctx for
      setting an pointer to a context of the given fiber and fiber_get_ctx for
      accessing that context.
      
      Closes https://github.com/tarantool/tarantool/issues/7669
      
      @TarantoolBot document
      Title: fiber: add fiber_set_ctx & fiber_get_ctx functions
      
      Add 2 api functions: `fiber_set_ctx` & `fiber_get_ctx` which can be used
      for passing data to a fiber. Previously this could be done via the
      `fiber_start` function, except that this would force the current fiber
      to yield, which is not acceptable in some scenarios (e.g. during a
      transaction). Now you can create a fiber with `fiber_new`, set it's
      contents with `fiber_set_ctx`, make it ready for execution with
      `fiber_wakeup` and keep executing the current fiber.
      6a60768c
    • Yaroslav Lobankov's avatar
      test: slight refactoring of replication-py tests · d13b06bd
      Yaroslav Lobankov authored
      - Remove unused imports
      - Remove unnecessary creation of 'replica' instance objects
      - Use `<instance>.iproto.uri` object attribute instead of calling
        `box.cfg.listen` via admin connection
      
      NO_DOC=testing stuff
      NO_TEST=testing stuff
      NO_CHANGELOG=testing stuff
      d13b06bd
    • Yaroslav Lobankov's avatar
      test: bump test-run to new version · 4335b442
      Yaroslav Lobankov authored
      Bump test-run to new version with the following improvements:
      
      - Report job summary on GitHub Actions [1]
      - Free port auto resolving for TarantoolServer and AppServer [2]
      
      Also, this patch includes the following changes:
      
      - removing `use_unix_sockets` option from all suite.ini config files
        due to permanent using Unix sockets for admin connection recently
        introduced in test-run
      - switching replication-py tests to Unix sockets for iproto connection
      - fixing replication-py/swap.test.py and swim/swim.test.lua tests
      
      [1] tarantool/test-run#341
      [2] tarantool/test-run#348
      
      NO_DOC=testing stuff
      NO_TEST=testing stuff
      NO_CHANGELOG=testing stuff
      4335b442
  4. Sep 12, 2022
    • Vladimir Davydov's avatar
      read_view: add option to include temporary spaces · fa56fbf2
      Vladimir Davydov authored
       - Filter out temporary spaces on read view creation unless the
         read_view_opts::needs_temporary_spaces flag is set and drop
         temporary space filter from checkpoint and join code.
       - Pass read_view_opts to engine_create_read_view. In case of memtx,
         delay garbage collection of temporary tuples if the flag is set.
      
      Needed for https://github.com/tarantool/tarantool-ee/issues/213
      
      NO_DOC=internal
      NO_TEST=ee
      NO_CHANGELOG=internal
      fa56fbf2
    • Sergey Bronnikov's avatar
      httpc: decode body in http response · 6a364cbc
      Sergey Bronnikov authored
      @TarantoolBot document
      Title: Document a body decoding in http response
      
      New method "response:decode()" has been introduced for a HTTP response.
      It allows to decode body to a Lua object. Decoding depends on content
      type passed in HTTP header and decoding function for it defined in table
      `http.decoders` or `http.new().decoders`. Function `response:decode()`
      will raise an error when content type is present in the response, but
      there is no appropriate decoder.
      
      By default decoders for the following content types are defined:
      
      - to JSON with content-type "application/json"
      - to MsgPack with content-type "application/msgpack"
      - to YAML with content-type "application/yaml"
      
      If a content type is not present in the response, :decode() assumes
      "application/json".
      
      ```
      tarantool> resp = require('http.client').put(
          'http://127.0.0.1:8080', '{"productId": 123456, "quantity": 100}')
      ---
      ...
      
      tarantool> resp.body
      ---
      - '{"productId": 123456, "quantity": 100}'
      ...
      
      tarantool>
      tarantool> resp:decode()
      ---
      - productId: 123456
      - quantity: 100
      ...
      
      tarantool>
      ```
      
      For content types missed in `http.decoders` user could define it's own
      by defining a new record with a key equal to desired MIME type in
      lowercase and it's decoding function that must accept HTTP body, content
      type and must return a decoded value:
      
      ```
      local http = require("http")
      local xml = require("luarapidxml")
      
      http.decoders = {
          ['application/xml'] = function(body, _content_type)
              return xml.decode(body)
          end,
      }
      ```
      
      Closes #6833
      6a364cbc
    • Sergey Bronnikov's avatar
      httpc: encode body in http request · 4818e988
      Sergey Bronnikov authored
      client_object:request() could pass a body as a string value to HTTP
      request when body is used. User must encode body to a string value as a
      preparation step. However, we can do encoding for user
      automatically when it possible.
      
      @TarantoolBot document
      Title: Document body encoding in http request
      
      Now user could pass a "body" option as a value with standard Lua types
      (table, nil, string, userdata, cdata, boolean or number) as well as
      Tarantool's own data types like decimal, uuid, datetime. In such case
      that value will be encoded to a string automatically where it possible.
      
      Encoding depends on body's data type: `nil` converted to an empty
      string, types (`string`, `number`, `boolean` and `nil`) will be
      converted to a string using standard Lua function `tostring()`, types
      cdata, userdata and tables will be encoded by functions defined in
      `http.encoders` or `http.new().encoders` table. In latter case encoding
      function depends on a content type passed to HTTP request using HTTP
      header. Default content type "application/json" and appropriate encoder
      are used when content type is missed in HTTP request.
      
      By default encoders for the following content types are defined:
      
      - "application/json"
      - "application/msgpack"
      - "application/yaml"
      
      For content types missed in `http.encoders` user could define it's own
      encoder by defining a new record with a key equal to desired MIME type
      in lowercase and it's encoding function that must accept HTTP body and
      content type and must return a string with serialized data:
      
      ```
      local http = require("http")
      local xml = require("luarapidxml")
      
      http.encoders = {
          ['application/xml'] = function(body, _content_type)
              return xml.encode(body)
          end,
      }
      ```
      
      Be careful with defining your own encoding function - header with content
      type could contain content type like "text/html" as well as a header
      options [1], like "text/html; charset=UTF-8". Content type is passed to
      encoder function *with* options.
      
      JSON encoder will use default configuration settings defined in JSON
      module (see `json.cfg()`). Header with content type will be set to
      "application/json" if it was not defined by user in request's options.
      
      ```
      local http = require("http")
      local client = http.new()
      
      local body = {
          a = 1,
          b = 2,
          c = 3,
      }
      client:request("POST", uri, body)
      client:post(uri, body)
      ```
      
      1. https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Type
      
      Part of #6833
      4818e988
    • Vladimir Davydov's avatar
      Use MT-Safe strerror_r instead of strerror · 44f46dc8
      Vladimir Davydov authored
      strerror() is MT-Unsafe, because it uses a static buffer under the hood.
      We should use strerror_r() instead, which takes a user-provided buffer.
      The problem is there are two implementations of strerror_r(): XSI and
      GNU. The first one returns an error code and always writes the message
      to the beginning of the buffer while the second one returns a pointer to
      a location within the buffer where the message starts. Let's introduce a
      macro HAVE_STRERROR_R_GNU set if the GNU version is available and define
      tt_strerror() which writes the message to the static buffer, like
      tt_cstr() or tt_sprintf().
      
      Note, we have to export tt_strerror(), because it is used by Lua via
      FFI. We also need to make it available in the module API header, because
      the say_syserror() macro uses strerror() directly. In order to avoid
      adding tt_strerror() to the module API, we introduce an internal helper
      function _say_strerror(), which calls tt_strerror().
      
      NO_DOC=bug fix
      NO_TEST=code is covered by existing tests
      44f46dc8
  5. Sep 09, 2022
    • Alexander Turenko's avatar
      popen: fix a race between setpgrp() and killpg() · e2207fdc
      Alexander Turenko authored
      In brief: `vfork()` on Mac OS 12 and newer doesn't suspend the parent
      process, so we should wait for `setpgrp()` to use `killpg()`. See more
      detailed description of the problem in a comment of the
      `popen_wait_group_leadership()` function.
      
      The solution is to spin in a loop and check child's process group. It
      looks as the most simple and direct solution. Other possible solutions
      requires to estimate cons and pros of using extra file descriptor or
      assigning a signal number for the child -> parent communication.
      
      There are the following alternatives and variations:
      
      * Create a pipe and notify the parent from the child about the
        `setpgrp()` call.
      
        It costs extra file descriptor, so I decided to don't do that.
        However if we'll need some channel to deliver information from the
        child to the parent for another task, it'll worth to reimplement this
        function too.
      
        One possible place, where we may need such channel is delivery of
        child's errors to the parent. Now the child writes them directly to
        logger's fd and it requires some tricky code to keep and close the
        descriptor at right points. Also it doesn't allow to catch those
        errors in the parent, but we may need it for #4925.
      * Notify the parent about `setpgrp()` using a signal.
      
        It seems too greedly to assign a specific signal for such local
        problem. It is also unclear how to guarantee that it'll not break any
        user's code: a user can load a dynamic library, which uses some
        signals on its own.
      
        However we can consider using this approach here if we'll design some
        common interprocess notification system.
      * We can use the fiber cond or the `popen_wait_timeout()` function from
        PR #7648 to react to the child termination instantly.
      
        It would complicate the code and anyway wouldn't allow to react
        instantly on `setpgrp()` in the child.
      
        Also it assumes yielding during the wait (see below).
      * Wait until `setpgrp()` in `popen_send_signal()` instead of
        `popen_new()`.
      
        It would add yielding/waiting inside `popen_send_signal()` and likely
        will extend a set of its possible exit situations. It is undesirable:
        this function should have simple and predictable behavior.
      * Finally, we considered yielding in `popen_wait_group_leadership()`
        instead of sleeping the whole tx thread.
      
        `<popen handle>:new()` doesn't yield at the moment and a user's code
        may lean on this fact.
      
        Yielding would allow to achieve better throughtput (amount of parallel
        requests per second), but we don't take much care to performance on
        Mac OS. The primary goal for this platform is to offer the same
        behavior as on Linux to allow development of applications.
      
      I didn't replace `vfork()` with `fork()` on Mac OS, because `vfork()`
      works and I don't know consequences of calling `pthread_atfork()`
      handlers in a child created by popen. See the comment in `popen_new()`
      near to `vfork()` call: it warns about possible mutex double locks. This
      topic will be investigated further in #6674.
      
      Fixes #7658
      
      NO_DOC=fixes incorrect behavior, no need to document the bug
      NO_TEST=already tested by app-tap/popen.test.lua
      e2207fdc
  6. Sep 07, 2022
    • Vladislav Shpilevoy's avatar
      raft: persist new term and vote separately · c9155ac8
      Vladislav Shpilevoy authored
      If a node persisted a foreign term + vote request at the same
      time, it increased split-brain probability. A node could vote for
      a candidate having smaller vclock than the local one. For example,
      via the following scenario:
      
      - Node1, node2, node3 are started;
      - Node1 becomes a leader;
      - The topology becomes node1 <-> node2 <-> node3 due to network
          issues;
      - Node1 sends a synchro txn to node2. The txn starts a WAL write;
      - Node3 bumps term and votes for self. Sends it all to node2;
      - Node2 votes for node3, because their vclocks are equal;
      - Node2 finishes all pending WAL writes, including the txn from
          node1. Now its vclock is > node3's one and the vote was wrong.
      - Node3 wins, writes PROMOTE, and it conflicts with node1 writing
          CONFIRM.
      
      This patch makes so a node can't persist a vote in a new term in
      the same WAL write as the term bump. Term bump is written first
      and alone. It serves as a WAL sync after which the node's vclock
      is not supposed to change except for the 0 (local) component.
      
      The vote requests are re-checked after term bump is persisted to
      see if they still can be applied.
      
      Part of #7253
      
      NO_DOC=bugfix
      c9155ac8
    • Vladislav Shpilevoy's avatar
      qsync: fix txn fiber hang on fencing at CONFIRM · ec628100
      Vladislav Shpilevoy authored
      If the limbo was fenced during CONFIRM WAL write, then the
      confirmed txn was committed just fine, but its author-fiber kept
      hanging. This is because when it was woken up, it checked if the
      limbo is frozen and went to infinite waiting before actually
      checking if the txn is completed.
      
      The fiber would unfreeze if would be woken up explicitly as a
      workaround.
      
      The fix is simple - change the checks order.
      
      Part of #7253
      
      NO_DOC=bugfix
      ec628100
    • Vladislav Shpilevoy's avatar
      promote: abort it when become non-candidate · ab08dad9
      Vladislav Shpilevoy authored
      box.ctl.promote() bumps the term, makes the node a candidate, and
      waits for the term outcome. The waiting used to be until there is
      a leader elected or the node lost connection quorum or the term
      was bumped again.
      
      There was a bug that a node could hang in box.ctl.promote() even
      when became a voter. It could happen if the quorum was still there
      and a leader couldn't be elected in the current term at all. For
      instance, others could have `election_mode='off'`.
      
      The fix is to stop waiting for the term outcome if the node can't
      win anyway.
      
      NO_DOC=bugfix
      ab08dad9
    • Vladislav Shpilevoy's avatar
      promote: fix infinite elections with multi-promote · dd89c57e
      Vladislav Shpilevoy authored
      If box.ctl.promote() was called on more than one instance, then it
      could lead to infinite or extremely long elections bumping
      thousands of terms in just a few seconds.
      
      This was because box.ctl.promote() used to be a loop. The loop
      retried term bump + voted for self until the node won. Retry
      happened immediately as the node saw the term was bumped again
      and there was no leader elected or the connection quorum was lost.
      
      If 2 nodes would start box.ctl.promote() almost at the same time,
      they could bump each other's terms, not see any winner, bump them
      again, and so on. For example:
      
      - Node1 term=1, node2 term=2;
      - Promote is called on both;
      - Node1 term=2, node2 term=3. They receive the messages. Node2
          ignores node1's old term. Node1 term is bumped and it votes
          for node2, but it didn't win, so box.ctl.promote() bumps its
          term to 4.
      - Node2 receives term 4 from node1. Its own box.ctl.promote() sees
          the term was bumped and no winner, so it bumps it to 5 and the
          process continues for a long time.
      
      It worked good enough in tests - the problem happened sometimes,
      terms could roll like 80k times in a few seconds, but the tests
      ended fine anyway.
      
      One of the next commits will make term bump + vote written in
      separate WAL records. That aggravates the problem drastically.
      
      Basically, this mutual term bump loop could end only if one node
      would receive vote for self from another node and send back the
      message 'I am a leader' before the other node's box.ctl.promote()
      notices the term was bumped externally. This will get much harder
      to achieve.
      
      The patch simply drops the loop. Let box.ctl.promote() fail if the
      term was bumped outside.
      
      There was an alternative to keep running it in a loop with a
      randomized election timeout like it works inside of raft. But the
      current solution is just simpler.
      
      NO_DOC=bugfix
      NO_TEST=election_split_vote_test.lua catches it already
      dd89c57e
    • Vladimir Davydov's avatar
      read_view: add space upgrade function to space_read_view · fe594e3c
      Vladimir Davydov authored
      There may be a space upgrade in progress at the time when a read view is
      created. In this case, we should apply the upgrade function to tuples
      retrieved from the space read view. We can't just use the space upgrade
      function as is, because it may be dropped while the read view is still
      in use. So we need to create a special upgrade function for the read
      view. This commit adds stubs for this, which will be implemented in the
      EE repository.
      
      Needed for https://github.com/tarantool/tarantool-ee/issues/163
      
      NO_DOC=internal
      NO_TEST=ee
      NO_CHANGELOG=internal
      fe594e3c
    • Vladimir Davydov's avatar
      read_view: assert that read view is used only in one thread · 4156440a
      Vladimir Davydov authored
      Although currenlty feasible, using a read view from multiple threads
      simultaneously is dangerous on its own because of inevitably arising
      object life time issues. With introduction of space upgrade handling,
      it'll become hardly possible, because we'll have to attach Lua function
      to a read view, which can only be used from one thread. Let's add some
      assertions to guarantee that a read view is used only from one thread.
      To do that, we need to introduce the concept of an active read view:
      from now on, a read view must be activated before it can be used and
      deactivated before it can be closed; both operations must be done from
      the same thread - the thread that accesses the read view.
      
      https://github.com/tarantool/tarantool-ee/issues/163
      
      NO_DOC=debug
      NO_TEST=debug
      NO_CHANGELOG=debug
      4156440a
    • Ilya Verbin's avatar
      fiber: make fiber_set_cancellable a no-op · 20b06656
      Ilya Verbin authored
      Currently this function is not used inside Tarantool, however it is
      available via C module API. Deprecate it, because it is very confusing
      and has nothing to do with the fiber cancellation.
      
      Closes #7166
      
      @TarantoolBot document
      Title: fiber: get rid of fiber_set_cancellable
      Product: Tarantool
      Since: 2.11
      Audience/target: dev
      Root document: https://www.tarantool.io/en/doc/latest/dev_guide/reference_capi/fiber/#c.fiber_set_cancellable
      SME: @Gumix
      
      This function is a no-op since 2.11 and should be dropped from the
      documentation.
      20b06656
    • Ilya Verbin's avatar
      main: allow spurious wakeups in on_shutdown_f · dd65d5c6
      Ilya Verbin authored
      It will yield until the is_shutting_down flag is set by tarantool_exit().
      This allows to get rid of the FIBER_IS_CANCELLABLE flag, which is no
      longer used anywhere in Tarantool.
      
      Part of #7166
      
      NO_DOC=internal
      NO_CHANGELOG=internal
      dd65d5c6
  7. Sep 06, 2022
Loading