Skip to content
Snippets Groups Projects
  1. Jun 27, 2018
    • Konstantin Osipov's avatar
      Merge branch '1.9' into 1.10 · 2821872e
      Konstantin Osipov authored
      2821872e
    • Konstantin Osipov's avatar
      net.box: update a test case after cherry-pick · e04b5b23
      Konstantin Osipov authored
      schema_version must be passed to perform_request in 1.9
      e04b5b23
    • Vladislav Shpilevoy's avatar
      iproto: on input discard do nothing for closed con · a60c8dff
      Vladislav Shpilevoy authored
      When a connection is closed, some of long-poll requests still may
      by in TX thread with non-discarded input. If a connection is
      closed, and then an input is discarded, then connection must not
      try to read new data.
      
      The bug was introduced here:
      f4d66dae by me.
      
      Closes #3400
      a60c8dff
    • Vladimir Davydov's avatar
      iproto: fix IPROTO_SERVER_IS_RO key code · e5e8ef4f
      Vladimir Davydov authored
      IPROTO_SERVER_IS_RO currently has code 0x07 and is defined in the header
      key section, which is wrong, because this key is only used in request
      body. Let's move it to the body section, where it belongs, and set its
      code to 0x29. This shouldn't break anything even if 0x07 is reused in
      future, because the two codes belong to different sections and hence are
      never parsed in the same function. Worst that can happen is we fail to
      bootstrap a node in the cluster if it is running a newer tarantool
      version.
      
      While we are at it, let's also add the key name and change its type from
      MP_UINT to MP_BOOL.
      
      Fixes commit a8ecd1e1 ("replication: fix bug with read-only replica
      as a bootstrap leader").
      e5e8ef4f
    • Vladimir Davydov's avatar
      xrow: fix ret code on decode failure · 56c6f533
      Vladimir Davydov authored
      Throughout the code, we return -1 on error, but decode methods return 1
      for some reason, although according to comments they are supposed to
      return -1. This doesn't result in any errors, because we use != 0 to
      check for errors. Nevertheless, let's fix it to avoid confusion.
      56c6f533
    • Vladimir Davydov's avatar
      txn: remove unused C++ wrappers · 1a6432d3
      Vladimir Davydov authored
      1a6432d3
    • Vladimir Davydov's avatar
      xlog: erase eof marker when reopening existing file for writing · ce44a9e0
      Vladimir Davydov authored
      When reopening an existing xlog file (as in case of vylog), we do not
      erase the eof marker immediately. Instead we reposition file offset
      to (file_size - sizeof eof_marker), assuming the eof marker will be
      overwritten on the first write.
      
      However, it isn't enough if we want to reuse this function for reopening
      WAL files, because when scanning the WAL directory we close a file if we
      read eof marker and never reopen it again, see recover_remaining_wals().
      So to avoid skipping rows written to a once closed WAL, we have to erase
      the eof marker when reopening an xlog file. Let's do it with truncate().
      ce44a9e0
  2. Jun 26, 2018
    • Vladimir Davydov's avatar
      vinyl: fix read iterator skips source after reading cache · 13f4355c
      Vladimir Davydov authored
      If a source is used on a read iteration (i.e. the key at which it is
      positioned is the next best match or, in terms of the read iterator
      implementation, its front_id matches the read iterator front_id), its
      history is cleaned up, see vy_read_iterator_apply_history(). This breaks
      the logic behind vy_read_src_is_behind(), which assumes that the history
      always points to the last used key. As a result, a source may be
      mistakenly skipped, as illustrated below:
      
        Fiber 1                               Fiber 2
        -------                               -------
        1. Opens read iterator.
        2. Advances it to the next key.
           The returned key was read from
           a mem or run (not from cache).
           The source's history is emptied.
                                              Adds a chain containing
                                              the key read by fiber 1
                                              to the cache.
        3. Continues iteration, reads
           next few keys from the cache
           until the chain ends. The source
           used at step 2 is skipped.
        4. Calls vy_read_src_is_behind()
           on the source used at step 2 and
           skipped at step 3. It returns
           false, because its history is
           empty, thus skipping keys stored
           in it.
      
      Fix this bug by moving the code that checks whether a source iterator
      needs to be advanced from vy_read_src_is_behind() to source iterator
      'skip' method, because there we always know the last key returned by
      the iterator.
      
      Basically, this returns the code we had before commit b4d57284
      ("vinyl: consolidate skip optimization checks in read iterator").
      
      Closes #3477
      13f4355c
    • Georgy Kirichenko's avatar
      Introduce privileges for object groups · af35de96
      Georgy Kirichenko authored
      Allow define access privileges for all spaces, functions and sequences.
      Read and write privileges are supported for spaces, execute privilege
      for sequences. Privilege granting and revoking might be done through old api
      without object identification:
        box.schema.user.grant("guest", "read", "space")
      
      Prerequisite #945
      af35de96
    • Konstantin Osipov's avatar
      schema: misc changes to improve code readability · 4aaefbe4
      Konstantin Osipov authored
      Use constants, rename methods, invert acl-object
      compatibility matrix.
      
      In scope of gh-945.
      4aaefbe4
    • imarkov's avatar
      security: add limits on object_type-privilege pair · 983c194e
      imarkov authored
      Introduce constraints on object_type-privilege pairs.
      These constraints limit senseless grants/revokes, i.e.,
      sequence - execute, all space related privileges(insert, delete,
      update),
      function - alter, all space related privileges,
      role - all privileges except create, drop, alter, execute
      
      Prerequisite #945
      983c194e
  3. Jun 25, 2018
    • Konstantin Osipov's avatar
      Merge branch '1.9' into 1.10 · 3ec12041
      Konstantin Osipov authored
      3ec12041
    • Vladimir Davydov's avatar
      socket: fix race between unix tcp server stop and start · 80d379ee
      Vladimir Davydov authored
      If called on a unix socket, bind(2) creates a new file, see unix(7).
      When we stop a unix tcp server, we should remove that file. Currently,
      we do it from the tcp server fiber, after the server loop is broken,
      which happens when the socket is closed, see tcp_server_loop(). This
      opens a time window for another tcp server to reuse the same path:
      
          main fiber                  tcp server loop
          ----------                  ---------------
      
          -- Start a tcp server.
          s = socket.tcp_server('unix/', sock_path, ...)
          -- Stop the server.
          s:close()
      
                                      socket_readable? => no, break loop
      
          -- Start a new tcp server. Use the same path as before.
          -- This function succeeds, because the socket is closed
          -- so tcp_server_bind_addr() will clean up by itself.
          s = socket.tcp_server('unix/', sock_path, ...)
      
           tcp_server_bind
            tcp_server_bind_addr
             socket_bind => EADDRINUSE
             tcp_connect => ECONNREFUSED
             -- Remove dead unix socket.
             fio.unlink(addr.port)
             socket_bind => success
      
                                      -- Deletes unix socket used
                                      -- by the new server.
                                      fio.unlink(addr.port)
      
      In particular, the race results in sporadic failures of app-tap/console
      test, which restarts a tcp server using the same file path.
      
      To fix this issue, let's close the socket after removing the socket
      file. This is absolutely legit on any UNIX system, and this eliminates
      the race shown above, because a new server that tries to bind on the
      same path as the one already used by a dying server will not receive
      ECONNREFUSED until the socket fd is closed and hence the file is
      removed.
      
      A note about the app-tap/console test. After this patch is applied,
      socket.close() takes a little longer for unix tcp server, because it
      yields twice, once for removing the socket file and once for closing the
      socket file descriptor. As a result, on_disconnect() trigger left from
      the previous test case has time to run after session.type() check.
      Actually, those triggers have already been tested and we should have
      cleared them before proceeding to the next test case. So instead of
      adding two new on_disconnect checks to the test plan, let's clear the
      triggers before session.type() test case and remove 3 on_connect and 5
      on_auth checks from the test plan.
      
      Closes #3168
      80d379ee
    • Konstantin Osipov's avatar
    • Vladislav Shpilevoy's avatar
      iproto: protect from false-correct size in msg header · c6951c92
      Vladislav Shpilevoy authored
      Consider this packet:
      
          msgpack = require('msgpack')
          data = msgpack.encode(18400000000000000000)..'aaaaaaa'
      
      Tarantool interprets 18400000000000000000 as size of a coming
      iproto request, and tries with no any checks to allocate buffer
      of such size. It calculates needed capacity like this:
      
          capacity = start_value;
          while (capacity < size)
              capacity *= 2;
      
      Here it is possible that on i-th iteration 'capacity' < 'size',
      but 'capacity * 2' overflows 64 bits and becomes < 'size' again,
      so this loop never ends and occupies 100% CPU.
      
      Strictly speaking overflow has undefined behavior. On the
      original system it led to nullifying 'capacity'.
      
      Such size is improbable as a real packet gabarits, but can appear
      as a result of parsing of some invalid packet, first bytes of
      which accidentally appears to be valid MessagePack uint. This is
      how the bug emerged on the real system.
      
      Lets restrict the maximal packet size to 2GB.
      
      Closes #3464
      c6951c92
  4. Jun 24, 2018
    • Mergen Imeev's avatar
      box: create bigrefs for tuples · 3768d4bb
      Mergen Imeev authored
      Due to limitation of reference counters for tuple being only
      65535 it was possible to reach this limitation. This patch
      increases capacity of reference counters to 4 billions.
      
      Closes #3224
      3768d4bb
  5. Jun 14, 2018
    • Vladislav Shpilevoy's avatar
      session: fix box.session.sync() · 6cc31e04
      Vladislav Shpilevoy authored
      Before the patch box.session.sync() is global for the session and
      is updated on each new iproto request. When the connection is
      multiplexed, box.session.sync() can be changed with no finishing
      a current request, if a new one arrives.
      
      The patch makes box.session.push() local for the request,
      protecting it from multiplexing mess. Box.session.sync() after
      the patch can be safely used inside a request.
      
      Closes #3450
      
      @TarantoolBot document
      Title: box.session.sync() became request local
      Box.session.sync() was global for a session, so it was unusable
      when the connection behind the session is multiplexed. Now
      box.session.sync() is request local and can be safely used inside
      the request processor.
      6cc31e04
    • Vladislav Shpilevoy's avatar
      fiber: remove fiber local storage · 766feac2
      Vladislav Shpilevoy authored
      Replace it with more specific structures and pointers in order to
      prepare to add `net` storage.
      
      This allows to make the code working with fiber storage simpler,
      remove useless wrappers and casts, and in the next patch - remove
      broken session.sync and add fiber sync.
      
      Note that under no circumstances fiber.h is allowed to include
      application-specific headers like session.h or txn.h. One only
      is allowed to announce a struct and add opaque pointer to it.
      766feac2
    • Vladimir Davydov's avatar
      Merge branch '1.9' into 1.10 · 57ea7669
      Vladimir Davydov authored
      57ea7669
    • Vladimir Davydov's avatar
      memtx: don't delay deletion of temporary tuples during snapshot · f9299c43
      Vladimir Davydov authored
      Since tuples stored in temporary spaces are never written to disk, we
      can always delete them immediately, even when a snapshot is in progress.
      
      Closes #3432
      f9299c43
    • Vladimir Davydov's avatar
      Remove unused space_noop · 93ed36ea
      Vladimir Davydov authored
      93ed36ea
    • Vladimir Davydov's avatar
      test: fix vinyl/upgrade/fill.lua script · ec84f36b
      Vladimir Davydov authored
      Since commit 8f63d5d9 ("vinyl: fail transaction immediately if it does
      not fit in memory"), vinyl won't trigger memory dump if the size of
      memory needed by a transaction is greater than the memory limit, instead
      it will fail the transaction immediately. This broke the aforementioned
      script, which relied on this to trigger system-wide memory dump. Fix it
      by reworking the dump trigger logic used by the script: now it tries to
      insert two tuples, box.cfg.vinyl_memory / 2 size each, instead of one.
      
      Closes #3449
      ec84f36b
  6. Jun 09, 2018
  7. Jun 08, 2018
    • Vladislav Shpilevoy's avatar
      Fix build · c8b95be6
      Vladislav Shpilevoy authored
      c8b95be6
    • Alexander Turenko's avatar
    • Alexander Turenko's avatar
      debian: don't install systemd service file twice · e38d2762
      Alexander Turenko authored
      It fixes the following errors during tarantool installation from
      packages on debian / ubuntu:
      
      ```
      Unpacking tarantool (1.9.1.23.gacbd91c-1) ...
      dpkg: error processing archive /var/cache/apt/archives/tarantool_1.9.1.23.gacbd91c-1_amd64.deb (--unpack):
       trying to overwrite '/lib/systemd/system/tarantool.service', which is also in package tarantool-common 1.9.1.23.gacbd91c-1
      ```
      
      The problem is that tarantool.service file was shipped with
      tarantool-common and tarantool packages both. It is the regression after
      8925b862.
      
      The way to avoid installing / enabling the service file within tarantool
      package is to pass `--name` option to dh_systemd_enable, but do not pass
      the service file name. In that case dh_systemd_enable does not found the
      service file and does not enforce existence of the file.
      
      Hope there is less hacky way to do so, but I don't found one at the
      moment.
      Unverified
      e38d2762
    • Vladimir Davydov's avatar
      box: refactor hot standby recovery · 4b818e99
      Vladimir Davydov authored
      Currently, we start a hot standby fiber even if not in hot standby mode
      (see recovery_follow_local). And we scan the wal directory twice - first
      time in recovery_follow_local(), second time in recovery_finalize().
      Let's factor out recover_remaining_wals() from those functions and call
      it explicitly. And let's call follow_local() and stop_local() only if in
      hot standby mode.
      
      Needed for #461
      4b818e99
    • Vladimir Davydov's avatar
      box: retrieve instance uuid before starting local recovery · b22f8e80
      Vladimir Davydov authored
      In order to find out if the current instance fell too much behind its
      peers in the cluster and so needs to be rebootstrapped, we need to
      connect it to remote peers before proceeding to local recovery. The
      problem is box.cfg.replication may have an entry corresponding to the
      instance itself so before connecting we have to start listening to
      incoming connections. Since an instance is supposed to sent its uuid in
      the greeting message, we also have to initialize INSTANCE_UUID early,
      before we start local recovery. So this patch makes memtx engine
      constructor not only scan the snapshot directory, but also read the
      header of the most recent snapshot to initialize INSTANCE_UUID.
      
      Needed for #461
      b22f8e80
    • Konstantin Osipov's avatar
    • Georgy Kirichenko's avatar
      Fix libunwind segfault · 5c3b3001
      Georgy Kirichenko authored
      Use volatile asm modifier to prevent unwanted and awkward optimizations
      causing segfault while backtracing
      5c3b3001
    • Vladislav Shpilevoy's avatar
      netbox: introduce iterable future objects · 39709775
      Vladislav Shpilevoy authored
      Netbox has two major ways to execute a request: sync and async.
      During execution of any a server can send multiplie responses via
      IPROTO_CHUNK. And the execution ways differ in how to handle the
      chunks (called messages or pushes).
      
      For a sync request a one can specify on_push callback and its
      on_push_ctx argument called on each message.
      
      When a request is async a user has a future object only, and can
      not specify any callbacks. To get the pushed messages a one must
      iterate over future object like this:
      for i, message in future:pairs(one_iteration_timeout) do
      ...
      end
      Or ignore messages just calling future:wait_result(). Anyway
      messages are not deleted, so a one can iterate over future object
      again and again.
      
      Follow up #2677
      39709775
    • Vladislav Shpilevoy's avatar
      session: introduce binary box.session.push · 2b1143a7
      Vladislav Shpilevoy authored
      Box.session.push() allows to send a message to a client with no
      finishing a main request. Tarantool after this patch supports
      pushes over binary protocol.
      
      IProto message is encoded using a new header code - IPROTO_CHUNK.
      Push works as follows: a user calls box.session.push(message).
      The message is encoded into currently active obuf in TX thread,
      and then Kharon notifies IProto thread about new data.
      
      Originally Kharon is the ferryman of Hades who carries souls of
      the newly deceased across the rivers Styx and Acheron that
      divided the world of the living from the world of the dead. In
      Tarantool Kharon is a message and does the similar work. It
      notifies IProto thread about new data in an output buffer
      carrying pushed messages to IProto. Styx here is cpipe, and the
      boat is cbus message.
      
      One connection has single Kharon for all pushes. But Kharon can
      not be in two places at the time. So once he got away from TX to
      IProto, new messages can not send Kharon. They just set a special
      flag. When Kharon is back to TX and sees the flag is set, he
      immediately takes the road back to IProto.
      
      Herewith a user is not blocked to write to obuf when Kharon is
      busy. The user just updates obuf and set the flag if not set.
      There is no waiting for Kharon arrival back.
      
      Closes #2677
      2b1143a7
    • imarkov's avatar
      box: Add privilleges constants to lua · 8c9a6e99
      imarkov authored
      Add lua bindings of PRIV_XXX constants.
      
      This patch helps to avoid using numerical constants of privilleges
      in schema.lua code.
      
      Relates #945
      8c9a6e99
    • Vladimir Davydov's avatar
      xrow: add helper function for encoding vclock · a013f84b
      Vladimir Davydov authored
      So as not to duplicate the same code over and over again.
      a013f84b
    • Vladimir Davydov's avatar
      applier: remove extra new line in log message printed on connect · eaa2d482
      Vladimir Davydov authored
      An extra new line looks ugly in the log:
      
        2018-06-06 15:22:22.682 [9807] main/101/interactive C> Tarantool 1.10.1-58-gd2272132
        2018-06-06 15:22:22.682 [9807] main/101/interactive C> log level 5
        2018-06-06 15:22:22.682 [9807] main/101/interactive I> mapping 268435456 bytes for memtx tuple arena...
        2018-06-06 15:22:22.683 [9807] main/101/interactive I> mapping 134217728 bytes for vinyl tuple arena...
        2018-06-06 15:22:22.692 [9807] main/101/interactive I> recovery start
        2018-06-06 15:22:22.692 [9807] main/101/interactive I> recovering from `./00000000000000000006.snap'
        2018-06-06 15:22:22.721 [9807] main/106/applier/ I> remote master is 1.10.1 at 0.0.0.0:44441
      
        2018-06-06 15:22:22.723 [9807] main/106/applier/ C> leaving orphan mode
        2018-06-06 15:22:22.723 [9807] main/101/interactive C> replica set sync complete, quorum of 1 replicas formed
        2018-06-06 15:22:22.723 [9807] main/101/interactive I> ready to accept requests
      eaa2d482
    • Vladimir Davydov's avatar
      recovery: constify vclock argument · 5fba685c
      Vladimir Davydov authored
      Neither recovery_new() nor recover_remaining_wals() need to modify it.
      5fba685c
    • Vladimir Davydov's avatar
      recovery: drop unused recovery_exit · 1b0e5052
      Vladimir Davydov authored
      1b0e5052
    • Ilya Markov's avatar
      security: Use system views instead of system spaces · 01455946
      Ilya Markov authored
      System views are used instead of direct reads of corresponding system
      spaces to explore all accessible objects such as spaces, functions, users
      and e.g. An operation with an inaccessible object produces a 'not found'
      error even if the object exists.
      
      In scope of #3250
      
      Includes up fixes from Georgy
      01455946
  8. Jun 07, 2018
Loading