Skip to content
Snippets Groups Projects
  1. Jun 26, 2018
    • Georgy Kirichenko's avatar
      Introduce privileges for object groups · af35de96
      Georgy Kirichenko authored
      Allow define access privileges for all spaces, functions and sequences.
      Read and write privileges are supported for spaces, execute privilege
      for sequences. Privilege granting and revoking might be done through old api
      without object identification:
        box.schema.user.grant("guest", "read", "space")
      
      Prerequisite #945
      af35de96
    • Konstantin Osipov's avatar
      schema: misc changes to improve code readability · 4aaefbe4
      Konstantin Osipov authored
      Use constants, rename methods, invert acl-object
      compatibility matrix.
      
      In scope of gh-945.
      4aaefbe4
    • imarkov's avatar
      security: add limits on object_type-privilege pair · 983c194e
      imarkov authored
      Introduce constraints on object_type-privilege pairs.
      These constraints limit senseless grants/revokes, i.e.,
      sequence - execute, all space related privileges(insert, delete,
      update),
      function - alter, all space related privileges,
      role - all privileges except create, drop, alter, execute
      
      Prerequisite #945
      983c194e
  2. Jun 25, 2018
    • Konstantin Osipov's avatar
      Merge branch '1.9' into 1.10 · 3ec12041
      Konstantin Osipov authored
      3ec12041
    • Vladimir Davydov's avatar
      socket: fix race between unix tcp server stop and start · 80d379ee
      Vladimir Davydov authored
      If called on a unix socket, bind(2) creates a new file, see unix(7).
      When we stop a unix tcp server, we should remove that file. Currently,
      we do it from the tcp server fiber, after the server loop is broken,
      which happens when the socket is closed, see tcp_server_loop(). This
      opens a time window for another tcp server to reuse the same path:
      
          main fiber                  tcp server loop
          ----------                  ---------------
      
          -- Start a tcp server.
          s = socket.tcp_server('unix/', sock_path, ...)
          -- Stop the server.
          s:close()
      
                                      socket_readable? => no, break loop
      
          -- Start a new tcp server. Use the same path as before.
          -- This function succeeds, because the socket is closed
          -- so tcp_server_bind_addr() will clean up by itself.
          s = socket.tcp_server('unix/', sock_path, ...)
      
           tcp_server_bind
            tcp_server_bind_addr
             socket_bind => EADDRINUSE
             tcp_connect => ECONNREFUSED
             -- Remove dead unix socket.
             fio.unlink(addr.port)
             socket_bind => success
      
                                      -- Deletes unix socket used
                                      -- by the new server.
                                      fio.unlink(addr.port)
      
      In particular, the race results in sporadic failures of app-tap/console
      test, which restarts a tcp server using the same file path.
      
      To fix this issue, let's close the socket after removing the socket
      file. This is absolutely legit on any UNIX system, and this eliminates
      the race shown above, because a new server that tries to bind on the
      same path as the one already used by a dying server will not receive
      ECONNREFUSED until the socket fd is closed and hence the file is
      removed.
      
      A note about the app-tap/console test. After this patch is applied,
      socket.close() takes a little longer for unix tcp server, because it
      yields twice, once for removing the socket file and once for closing the
      socket file descriptor. As a result, on_disconnect() trigger left from
      the previous test case has time to run after session.type() check.
      Actually, those triggers have already been tested and we should have
      cleared them before proceeding to the next test case. So instead of
      adding two new on_disconnect checks to the test plan, let's clear the
      triggers before session.type() test case and remove 3 on_connect and 5
      on_auth checks from the test plan.
      
      Closes #3168
      80d379ee
    • Konstantin Osipov's avatar
    • Vladislav Shpilevoy's avatar
      iproto: protect from false-correct size in msg header · c6951c92
      Vladislav Shpilevoy authored
      Consider this packet:
      
          msgpack = require('msgpack')
          data = msgpack.encode(18400000000000000000)..'aaaaaaa'
      
      Tarantool interprets 18400000000000000000 as size of a coming
      iproto request, and tries with no any checks to allocate buffer
      of such size. It calculates needed capacity like this:
      
          capacity = start_value;
          while (capacity < size)
              capacity *= 2;
      
      Here it is possible that on i-th iteration 'capacity' < 'size',
      but 'capacity * 2' overflows 64 bits and becomes < 'size' again,
      so this loop never ends and occupies 100% CPU.
      
      Strictly speaking overflow has undefined behavior. On the
      original system it led to nullifying 'capacity'.
      
      Such size is improbable as a real packet gabarits, but can appear
      as a result of parsing of some invalid packet, first bytes of
      which accidentally appears to be valid MessagePack uint. This is
      how the bug emerged on the real system.
      
      Lets restrict the maximal packet size to 2GB.
      
      Closes #3464
      c6951c92
  3. Jun 24, 2018
    • Mergen Imeev's avatar
      box: create bigrefs for tuples · 3768d4bb
      Mergen Imeev authored
      Due to limitation of reference counters for tuple being only
      65535 it was possible to reach this limitation. This patch
      increases capacity of reference counters to 4 billions.
      
      Closes #3224
      3768d4bb
  4. Jun 14, 2018
    • Vladislav Shpilevoy's avatar
      session: fix box.session.sync() · 6cc31e04
      Vladislav Shpilevoy authored
      Before the patch box.session.sync() is global for the session and
      is updated on each new iproto request. When the connection is
      multiplexed, box.session.sync() can be changed with no finishing
      a current request, if a new one arrives.
      
      The patch makes box.session.push() local for the request,
      protecting it from multiplexing mess. Box.session.sync() after
      the patch can be safely used inside a request.
      
      Closes #3450
      
      @TarantoolBot document
      Title: box.session.sync() became request local
      Box.session.sync() was global for a session, so it was unusable
      when the connection behind the session is multiplexed. Now
      box.session.sync() is request local and can be safely used inside
      the request processor.
      6cc31e04
    • Vladislav Shpilevoy's avatar
      fiber: remove fiber local storage · 766feac2
      Vladislav Shpilevoy authored
      Replace it with more specific structures and pointers in order to
      prepare to add `net` storage.
      
      This allows to make the code working with fiber storage simpler,
      remove useless wrappers and casts, and in the next patch - remove
      broken session.sync and add fiber sync.
      
      Note that under no circumstances fiber.h is allowed to include
      application-specific headers like session.h or txn.h. One only
      is allowed to announce a struct and add opaque pointer to it.
      766feac2
    • Vladimir Davydov's avatar
      Merge branch '1.9' into 1.10 · 57ea7669
      Vladimir Davydov authored
      57ea7669
    • Vladimir Davydov's avatar
      memtx: don't delay deletion of temporary tuples during snapshot · f9299c43
      Vladimir Davydov authored
      Since tuples stored in temporary spaces are never written to disk, we
      can always delete them immediately, even when a snapshot is in progress.
      
      Closes #3432
      f9299c43
    • Vladimir Davydov's avatar
      Remove unused space_noop · 93ed36ea
      Vladimir Davydov authored
      93ed36ea
    • Vladimir Davydov's avatar
      test: fix vinyl/upgrade/fill.lua script · ec84f36b
      Vladimir Davydov authored
      Since commit 8f63d5d9 ("vinyl: fail transaction immediately if it does
      not fit in memory"), vinyl won't trigger memory dump if the size of
      memory needed by a transaction is greater than the memory limit, instead
      it will fail the transaction immediately. This broke the aforementioned
      script, which relied on this to trigger system-wide memory dump. Fix it
      by reworking the dump trigger logic used by the script: now it tries to
      insert two tuples, box.cfg.vinyl_memory / 2 size each, instead of one.
      
      Closes #3449
      ec84f36b
  5. Jun 09, 2018
  6. Jun 08, 2018
    • Vladislav Shpilevoy's avatar
      Fix build · c8b95be6
      Vladislav Shpilevoy authored
      c8b95be6
    • Alexander Turenko's avatar
    • Alexander Turenko's avatar
      debian: don't install systemd service file twice · e38d2762
      Alexander Turenko authored
      It fixes the following errors during tarantool installation from
      packages on debian / ubuntu:
      
      ```
      Unpacking tarantool (1.9.1.23.gacbd91c-1) ...
      dpkg: error processing archive /var/cache/apt/archives/tarantool_1.9.1.23.gacbd91c-1_amd64.deb (--unpack):
       trying to overwrite '/lib/systemd/system/tarantool.service', which is also in package tarantool-common 1.9.1.23.gacbd91c-1
      ```
      
      The problem is that tarantool.service file was shipped with
      tarantool-common and tarantool packages both. It is the regression after
      8925b862.
      
      The way to avoid installing / enabling the service file within tarantool
      package is to pass `--name` option to dh_systemd_enable, but do not pass
      the service file name. In that case dh_systemd_enable does not found the
      service file and does not enforce existence of the file.
      
      Hope there is less hacky way to do so, but I don't found one at the
      moment.
      Unverified
      e38d2762
    • Vladimir Davydov's avatar
      box: refactor hot standby recovery · 4b818e99
      Vladimir Davydov authored
      Currently, we start a hot standby fiber even if not in hot standby mode
      (see recovery_follow_local). And we scan the wal directory twice - first
      time in recovery_follow_local(), second time in recovery_finalize().
      Let's factor out recover_remaining_wals() from those functions and call
      it explicitly. And let's call follow_local() and stop_local() only if in
      hot standby mode.
      
      Needed for #461
      4b818e99
    • Vladimir Davydov's avatar
      box: retrieve instance uuid before starting local recovery · b22f8e80
      Vladimir Davydov authored
      In order to find out if the current instance fell too much behind its
      peers in the cluster and so needs to be rebootstrapped, we need to
      connect it to remote peers before proceeding to local recovery. The
      problem is box.cfg.replication may have an entry corresponding to the
      instance itself so before connecting we have to start listening to
      incoming connections. Since an instance is supposed to sent its uuid in
      the greeting message, we also have to initialize INSTANCE_UUID early,
      before we start local recovery. So this patch makes memtx engine
      constructor not only scan the snapshot directory, but also read the
      header of the most recent snapshot to initialize INSTANCE_UUID.
      
      Needed for #461
      b22f8e80
    • Konstantin Osipov's avatar
    • Georgy Kirichenko's avatar
      Fix libunwind segfault · 5c3b3001
      Georgy Kirichenko authored
      Use volatile asm modifier to prevent unwanted and awkward optimizations
      causing segfault while backtracing
      5c3b3001
    • Vladislav Shpilevoy's avatar
      netbox: introduce iterable future objects · 39709775
      Vladislav Shpilevoy authored
      Netbox has two major ways to execute a request: sync and async.
      During execution of any a server can send multiplie responses via
      IPROTO_CHUNK. And the execution ways differ in how to handle the
      chunks (called messages or pushes).
      
      For a sync request a one can specify on_push callback and its
      on_push_ctx argument called on each message.
      
      When a request is async a user has a future object only, and can
      not specify any callbacks. To get the pushed messages a one must
      iterate over future object like this:
      for i, message in future:pairs(one_iteration_timeout) do
      ...
      end
      Or ignore messages just calling future:wait_result(). Anyway
      messages are not deleted, so a one can iterate over future object
      again and again.
      
      Follow up #2677
      39709775
    • Vladislav Shpilevoy's avatar
      session: introduce binary box.session.push · 2b1143a7
      Vladislav Shpilevoy authored
      Box.session.push() allows to send a message to a client with no
      finishing a main request. Tarantool after this patch supports
      pushes over binary protocol.
      
      IProto message is encoded using a new header code - IPROTO_CHUNK.
      Push works as follows: a user calls box.session.push(message).
      The message is encoded into currently active obuf in TX thread,
      and then Kharon notifies IProto thread about new data.
      
      Originally Kharon is the ferryman of Hades who carries souls of
      the newly deceased across the rivers Styx and Acheron that
      divided the world of the living from the world of the dead. In
      Tarantool Kharon is a message and does the similar work. It
      notifies IProto thread about new data in an output buffer
      carrying pushed messages to IProto. Styx here is cpipe, and the
      boat is cbus message.
      
      One connection has single Kharon for all pushes. But Kharon can
      not be in two places at the time. So once he got away from TX to
      IProto, new messages can not send Kharon. They just set a special
      flag. When Kharon is back to TX and sees the flag is set, he
      immediately takes the road back to IProto.
      
      Herewith a user is not blocked to write to obuf when Kharon is
      busy. The user just updates obuf and set the flag if not set.
      There is no waiting for Kharon arrival back.
      
      Closes #2677
      2b1143a7
    • imarkov's avatar
      box: Add privilleges constants to lua · 8c9a6e99
      imarkov authored
      Add lua bindings of PRIV_XXX constants.
      
      This patch helps to avoid using numerical constants of privilleges
      in schema.lua code.
      
      Relates #945
      8c9a6e99
    • Vladimir Davydov's avatar
      xrow: add helper function for encoding vclock · a013f84b
      Vladimir Davydov authored
      So as not to duplicate the same code over and over again.
      a013f84b
    • Vladimir Davydov's avatar
      applier: remove extra new line in log message printed on connect · eaa2d482
      Vladimir Davydov authored
      An extra new line looks ugly in the log:
      
        2018-06-06 15:22:22.682 [9807] main/101/interactive C> Tarantool 1.10.1-58-gd2272132
        2018-06-06 15:22:22.682 [9807] main/101/interactive C> log level 5
        2018-06-06 15:22:22.682 [9807] main/101/interactive I> mapping 268435456 bytes for memtx tuple arena...
        2018-06-06 15:22:22.683 [9807] main/101/interactive I> mapping 134217728 bytes for vinyl tuple arena...
        2018-06-06 15:22:22.692 [9807] main/101/interactive I> recovery start
        2018-06-06 15:22:22.692 [9807] main/101/interactive I> recovering from `./00000000000000000006.snap'
        2018-06-06 15:22:22.721 [9807] main/106/applier/ I> remote master is 1.10.1 at 0.0.0.0:44441
      
        2018-06-06 15:22:22.723 [9807] main/106/applier/ C> leaving orphan mode
        2018-06-06 15:22:22.723 [9807] main/101/interactive C> replica set sync complete, quorum of 1 replicas formed
        2018-06-06 15:22:22.723 [9807] main/101/interactive I> ready to accept requests
      eaa2d482
    • Vladimir Davydov's avatar
      recovery: constify vclock argument · 5fba685c
      Vladimir Davydov authored
      Neither recovery_new() nor recover_remaining_wals() need to modify it.
      5fba685c
    • Vladimir Davydov's avatar
      recovery: drop unused recovery_exit · 1b0e5052
      Vladimir Davydov authored
      1b0e5052
    • Ilya Markov's avatar
      security: Use system views instead of system spaces · 01455946
      Ilya Markov authored
      System views are used instead of direct reads of corresponding system
      spaces to explore all accessible objects such as spaces, functions, users
      and e.g. An operation with an inaccessible object produces a 'not found'
      error even if the object exists.
      
      In scope of #3250
      
      Includes up fixes from Georgy
      01455946
  7. Jun 07, 2018
    • Alexander Turenko's avatar
    • Alexander Turenko's avatar
      test: update test-run · acbd91cc
      Alexander Turenko authored
      * added --verbose to show output of successful TAP13 test (#73)
      * allow to call create_cluster(), drop_cluster() multiple times (#83)
      * support configurations (*.cfg files) in core = app tests
      * added return_listen_uri = <boolean> option for create_cluster()
      * save and print at fail tarantool log for core = app tests (#87)
      Unverified
      acbd91cc
    • Vladimir Davydov's avatar
      vinyl: allow to build secondary index for non-empty space · 687f303e
      Vladimir Davydov authored
      This patch implements space_vtab::build_index callback for vinyl spaces.
      Now instead of returning an error in case the space is not empty, the
      callback will actually try to build a new index. The build procedure
      consists of four steps:
      
       1. Prepare the LSM tree for building. This implies writing a special
          record to vylog, VY_LOG_PREPARE_LSM, and adding the new index to
          the vinyl scheduler so that it can be dumped during build. We need
          to log the new LSM tree so that we can keep track of run files
          created for it during build and remove them if build procedure
          fails.
      
       2. Inserting tuples stored in the space into the new LSM tree. Since
          there may concurrent DML requests, we install a trigger to forward
          them to the new index.
      
       3. Dumping the index to disk so that we don't have to rebuild it after
          recovery.
      
       4. Committing the new LSM tree in vylog (VY_LOG_CREATE_LSM).
      
      Steps 1-3 are done from the space_vtab::build_index callback while
      step 4 is done after WAL write, from index_vtab::commit_create.
      
      While step 3 is being performed, new DML requests may be executed for
      the altered space. Those requests will be reflected in the new index
      thanks to the on_replace trigger, however they won't be recovered during
      WAL recovery as they will appear in WAL before the ALTER record that
      created the index. To recover them, we replay all statements stored in
      the primary key's memory level when replaying the ALTER record during
      WAL recovery.
      
      Closes #1653
      687f303e
    • Vladimir Davydov's avatar
      vinyl: relax limitation imposed on run min/max lsn · 2508ebb7
      Vladimir Davydov authored
      Currently, we assume that no two runs of the same range intersect by
      LSN. This holds, because LSNs grow strictly monotonically, and no
      transaction may be split between two runs (as we pin each affected
      vy_mem until the transaction is complete). We ensure this with an
      assertion in vy_task_dump_complete.
      
      However, tuples inserted during index build will have arbitrary (not
      monotonically growing) LSNs. This is OK as for each particular key, two
      statements will still have different LSNs, but this may break the
      assertion in vy_task_dump_complete in case dump occurs while build is in
      progress.
      
      To avoid that, let's relax this limitation and assume that a dumped run
      may intersect by LSN with runs dumped before. Moreover, let's assume
      that it may have max LSN less than the max LSN stored on disk so that
      we should update vy_lsm::dump_lsn only if the dumped run has newer data.
      
      Needed for #1653
      2508ebb7
    • Konstantin Osipov's avatar
    • Vladimir Davydov's avatar
      vinyl: do not yield on dump completion · 6605a5a4
      Vladimir Davydov authored
      The fact that we may yield after we added a new slice created by dump,
      but before we removed the dumped in-memory index from the LSM tree
      complicates read iterator logic, as it has to detect such a case and
      filter out sources that contain duplicates. This logic relies on the
      fact that no two slices of the same range intersect by LSN. For the
      sake of ALTER we have to relax this limitation, as statements inserted
      during index build can have arbitrary (not monotonically growing) LSNs,
      so the no-LSN-intersection property won't be fulfilled for whole slices,
      only for individual keys. Since there shouldn't be more than 1000 ranges
      in the same LSM tree, yielding doesn't make much sense as iteration over
      the whole range tree should be pretty fast. Besides, dump isn't done
      frequently. That said, let's remove yielding altogether.
      
      Needed for #1653
      6605a5a4
    • Alexander Turenko's avatar
    • lifemaker's avatar
      Travis CI: add Ubuntu Bionic · ab4416f8
      lifemaker authored
      Unverified
      ab4416f8
    • Vladimir Davydov's avatar
      vinyl: factor out vy_check_is_unique_secondary · 93f71dae
      Vladimir Davydov authored
      We need to check unique constraint when building a new index. So let's
      factor out this helper function and pass space_name, index_name, and
      read view to it explicitly (because index_name_by_id isn't going to work
      for an index that is under construction and there's no tx when we are
      building a new index). Suggested by @Gerold103.
      
      Needed for #1653
      93f71dae
    • Vladislav Shpilevoy's avatar
      session: enable box.session.push in local console · 5bb2fb7f
      Vladislav Shpilevoy authored
      It is quite simple - just use stdout file descriptor as the
      destination for push messages. It is needed to make remote and
      local console be similar.
      5bb2fb7f
Loading