Skip to content
Snippets Groups Projects
  1. Jun 08, 2018
    • Vladislav Shpilevoy's avatar
      session: introduce binary box.session.push · 2b1143a7
      Vladislav Shpilevoy authored
      Box.session.push() allows to send a message to a client with no
      finishing a main request. Tarantool after this patch supports
      pushes over binary protocol.
      
      IProto message is encoded using a new header code - IPROTO_CHUNK.
      Push works as follows: a user calls box.session.push(message).
      The message is encoded into currently active obuf in TX thread,
      and then Kharon notifies IProto thread about new data.
      
      Originally Kharon is the ferryman of Hades who carries souls of
      the newly deceased across the rivers Styx and Acheron that
      divided the world of the living from the world of the dead. In
      Tarantool Kharon is a message and does the similar work. It
      notifies IProto thread about new data in an output buffer
      carrying pushed messages to IProto. Styx here is cpipe, and the
      boat is cbus message.
      
      One connection has single Kharon for all pushes. But Kharon can
      not be in two places at the time. So once he got away from TX to
      IProto, new messages can not send Kharon. They just set a special
      flag. When Kharon is back to TX and sees the flag is set, he
      immediately takes the road back to IProto.
      
      Herewith a user is not blocked to write to obuf when Kharon is
      busy. The user just updates obuf and set the flag if not set.
      There is no waiting for Kharon arrival back.
      
      Closes #2677
      2b1143a7
    • imarkov's avatar
      box: Add privilleges constants to lua · 8c9a6e99
      imarkov authored
      Add lua bindings of PRIV_XXX constants.
      
      This patch helps to avoid using numerical constants of privilleges
      in schema.lua code.
      
      Relates #945
      8c9a6e99
    • Vladimir Davydov's avatar
      xrow: add helper function for encoding vclock · a013f84b
      Vladimir Davydov authored
      So as not to duplicate the same code over and over again.
      a013f84b
    • Vladimir Davydov's avatar
      applier: remove extra new line in log message printed on connect · eaa2d482
      Vladimir Davydov authored
      An extra new line looks ugly in the log:
      
        2018-06-06 15:22:22.682 [9807] main/101/interactive C> Tarantool 1.10.1-58-gd2272132
        2018-06-06 15:22:22.682 [9807] main/101/interactive C> log level 5
        2018-06-06 15:22:22.682 [9807] main/101/interactive I> mapping 268435456 bytes for memtx tuple arena...
        2018-06-06 15:22:22.683 [9807] main/101/interactive I> mapping 134217728 bytes for vinyl tuple arena...
        2018-06-06 15:22:22.692 [9807] main/101/interactive I> recovery start
        2018-06-06 15:22:22.692 [9807] main/101/interactive I> recovering from `./00000000000000000006.snap'
        2018-06-06 15:22:22.721 [9807] main/106/applier/ I> remote master is 1.10.1 at 0.0.0.0:44441
      
        2018-06-06 15:22:22.723 [9807] main/106/applier/ C> leaving orphan mode
        2018-06-06 15:22:22.723 [9807] main/101/interactive C> replica set sync complete, quorum of 1 replicas formed
        2018-06-06 15:22:22.723 [9807] main/101/interactive I> ready to accept requests
      eaa2d482
    • Vladimir Davydov's avatar
      recovery: constify vclock argument · 5fba685c
      Vladimir Davydov authored
      Neither recovery_new() nor recover_remaining_wals() need to modify it.
      5fba685c
    • Vladimir Davydov's avatar
      recovery: drop unused recovery_exit · 1b0e5052
      Vladimir Davydov authored
      1b0e5052
    • Ilya Markov's avatar
      security: Use system views instead of system spaces · 01455946
      Ilya Markov authored
      System views are used instead of direct reads of corresponding system
      spaces to explore all accessible objects such as spaces, functions, users
      and e.g. An operation with an inaccessible object produces a 'not found'
      error even if the object exists.
      
      In scope of #3250
      
      Includes up fixes from Georgy
      01455946
  2. Jun 07, 2018
    • Alexander Turenko's avatar
    • Alexander Turenko's avatar
      test: update test-run · acbd91cc
      Alexander Turenko authored
      * added --verbose to show output of successful TAP13 test (#73)
      * allow to call create_cluster(), drop_cluster() multiple times (#83)
      * support configurations (*.cfg files) in core = app tests
      * added return_listen_uri = <boolean> option for create_cluster()
      * save and print at fail tarantool log for core = app tests (#87)
      Unverified
      acbd91cc
    • Vladimir Davydov's avatar
      vinyl: allow to build secondary index for non-empty space · 687f303e
      Vladimir Davydov authored
      This patch implements space_vtab::build_index callback for vinyl spaces.
      Now instead of returning an error in case the space is not empty, the
      callback will actually try to build a new index. The build procedure
      consists of four steps:
      
       1. Prepare the LSM tree for building. This implies writing a special
          record to vylog, VY_LOG_PREPARE_LSM, and adding the new index to
          the vinyl scheduler so that it can be dumped during build. We need
          to log the new LSM tree so that we can keep track of run files
          created for it during build and remove them if build procedure
          fails.
      
       2. Inserting tuples stored in the space into the new LSM tree. Since
          there may concurrent DML requests, we install a trigger to forward
          them to the new index.
      
       3. Dumping the index to disk so that we don't have to rebuild it after
          recovery.
      
       4. Committing the new LSM tree in vylog (VY_LOG_CREATE_LSM).
      
      Steps 1-3 are done from the space_vtab::build_index callback while
      step 4 is done after WAL write, from index_vtab::commit_create.
      
      While step 3 is being performed, new DML requests may be executed for
      the altered space. Those requests will be reflected in the new index
      thanks to the on_replace trigger, however they won't be recovered during
      WAL recovery as they will appear in WAL before the ALTER record that
      created the index. To recover them, we replay all statements stored in
      the primary key's memory level when replaying the ALTER record during
      WAL recovery.
      
      Closes #1653
      687f303e
    • Vladimir Davydov's avatar
      vinyl: relax limitation imposed on run min/max lsn · 2508ebb7
      Vladimir Davydov authored
      Currently, we assume that no two runs of the same range intersect by
      LSN. This holds, because LSNs grow strictly monotonically, and no
      transaction may be split between two runs (as we pin each affected
      vy_mem until the transaction is complete). We ensure this with an
      assertion in vy_task_dump_complete.
      
      However, tuples inserted during index build will have arbitrary (not
      monotonically growing) LSNs. This is OK as for each particular key, two
      statements will still have different LSNs, but this may break the
      assertion in vy_task_dump_complete in case dump occurs while build is in
      progress.
      
      To avoid that, let's relax this limitation and assume that a dumped run
      may intersect by LSN with runs dumped before. Moreover, let's assume
      that it may have max LSN less than the max LSN stored on disk so that
      we should update vy_lsm::dump_lsn only if the dumped run has newer data.
      
      Needed for #1653
      2508ebb7
    • Konstantin Osipov's avatar
    • Vladimir Davydov's avatar
      vinyl: do not yield on dump completion · 6605a5a4
      Vladimir Davydov authored
      The fact that we may yield after we added a new slice created by dump,
      but before we removed the dumped in-memory index from the LSM tree
      complicates read iterator logic, as it has to detect such a case and
      filter out sources that contain duplicates. This logic relies on the
      fact that no two slices of the same range intersect by LSN. For the
      sake of ALTER we have to relax this limitation, as statements inserted
      during index build can have arbitrary (not monotonically growing) LSNs,
      so the no-LSN-intersection property won't be fulfilled for whole slices,
      only for individual keys. Since there shouldn't be more than 1000 ranges
      in the same LSM tree, yielding doesn't make much sense as iteration over
      the whole range tree should be pretty fast. Besides, dump isn't done
      frequently. That said, let's remove yielding altogether.
      
      Needed for #1653
      6605a5a4
    • Alexander Turenko's avatar
    • lifemaker's avatar
      Travis CI: add Ubuntu Bionic · ab4416f8
      lifemaker authored
      Unverified
      ab4416f8
    • Vladimir Davydov's avatar
      vinyl: factor out vy_check_is_unique_secondary · 93f71dae
      Vladimir Davydov authored
      We need to check unique constraint when building a new index. So let's
      factor out this helper function and pass space_name, index_name, and
      read view to it explicitly (because index_name_by_id isn't going to work
      for an index that is under construction and there's no tx when we are
      building a new index). Suggested by @Gerold103.
      
      Needed for #1653
      93f71dae
    • Vladislav Shpilevoy's avatar
      session: enable box.session.push in local console · 5bb2fb7f
      Vladislav Shpilevoy authored
      It is quite simple - just use stdout file descriptor as the
      destination for push messages. It is needed to make remote and
      local console be similar.
      5bb2fb7f
  3. Jun 06, 2018
    • Vladislav Shpilevoy's avatar
      session: introduce text box.session.push · 6d99d51e
      Vladislav Shpilevoy authored
      box.session.push allows to send some intermediate results in the
      scope of main request with no finalizing it. Messages can be
      sent over text and binary protocol. This patch allows to send
      text pushes.
      
      Text push is a YAML document tagged with '!push!' handle and
      'tag:tarantool.io/push,2018' prefix. YAML tags is a standard way
      to define a type of the document.
      
      Console received push message just prints it to the stdout (or
      sends to a next console, if it is remote console too).
      
      Part of #2677
      6d99d51e
    • Konstantin Osipov's avatar
      diag: preserve errno in diag_set() and sio_socketname() · a79ae4ba
      Konstantin Osipov authored
      malloc() or getsockname() may reset the errno.
      Let's ensure we preserve errno when setting the diagnostics.
      a79ae4ba
    • Vladimir Davydov's avatar
      vinyl: simplify index hash building during vylog recovery · 68f9f39f
      Vladimir Davydov authored
      For the sake of recovery, we keep all vy_lsm_recovery_info objects in
      vy_recovery::index_id_hash, which maps space_id/index_id to the latest
      index incarnation. We build this hash while loading records from vylog,
      which is rather difficult for understanding, because an LSM tree may
      have a counterpart created by incomplete ALTER (see ->prepared) so we
      have to link and unlink those objects very carefully. Let's simplify the
      procedure by postponing hash building until we have loaded all records.
      68f9f39f
    • Vladimir Davydov's avatar
      vinyl: allow to commit statements to mem in arbitrary order · f2e30507
      Vladimir Davydov authored
      vy_mem_commit_stmt() expects statements to be committed in the order
      of increasing LSN. Although this condition holds now, it won't once
      we start using this function for building indexes. So let's remove
      this limitation.
      
      Needed for #1653
      f2e30507
    • Konstantin Osipov's avatar
      47105cd9
    • Vladimir Davydov's avatar
      vinyl: bump mem version after committing statement · 2ae0902e
      Vladimir Davydov authored
      Since commit 1e1c1fdb ("vinyl: make read iterator always return
      newest tuple version") vinyl read iterator guarantees that any tuple it
      returns is the newest version in the iterator read view. However, if we
      don't bump mem version after assigning LSN to a mem statement, a read
      iterator using committed_read_view might not see it and return a stale
      tuple. Currently, there's no code that relies on this iterator feature,
      but we will need it for building new indexes. Without this patch, build
      (introduced later in the series) might return inconsistent results.
      
      Needed for #1653
      2ae0902e
    • Vladimir Davydov's avatar
      vinyl: log new index before WAL write on DDL · dd0827ba
      Vladimir Davydov authored
      Currently, we write new indexes to vylog only after successful WAL
      write (see vinyl_index_commit_create). This is incompatible with space
      ALTER - the problem is during ALTER vinyl may need to create new run
      files, which we need to track in order not to leave garbage if ALTER
      fails or tarantool exits before ALTER is complete.
      
      So this patch splits index creation in two stages, prepare and commit.
      The 'commit' stage is represented by existing VY_LOG_CREATE_LSM record,
      which is written from index_vtab::commit_create callback, just like
      before. For the 'prepare' stage we introduce a new record type,
      VY_LOG_REPARE_LSM, written from index_vtab::add_primary_key and
      index_vtab::build_index callbacks, i.e. before WAL write. For now, we
      don't write anything to prepared, but uncommitted indexes (this will be
      done later), but we do add prepared indexes to the scheduler so that
      they can be dumped and compacted. If ALTER fails, we drop prepared
      indexes in index_vtab::abort_create callback. Prepared but uncommitted
      indexes are ignored by backup and replication and cleaned up from vylog
      on restart.
      
      Note, we have to rework vinyl/errinj_vylog test in this patch, because
      index creation (and hence space truncation) commands now fail on vylog
      error, i.e. a situation when the same index is dropped and recreated
      multiple times in xlog without having corresponding records in vylog is
      now impossible.
      
      Also, space truncation is not linearizable for vinyl anymore as it may
      yield before WAL write, while trying to prepare an index in vylog. This
      is OK - we never promised it is. Remove the corresponding test case.
      
      Needed for #1653
      dd0827ba
  4. Jun 04, 2018
    • Vladimir Davydov's avatar
      vinyl: apply box.cfg.snap_io_rate_limit to dump/compaction · 62830170
      Vladimir Davydov authored
      Vinyl worker threads can consume all disk bandwidth while performing
      dump or compaction, thus stalling DML requests, which also need some
      disk bandwidth for WAL. Memtx has a similar problem - it needs to write
      snapshot files. In case of memtx, we cope with this problem by limiting
      the write rate with box.cfg.snap_io_rate_limit option. Let's reuse this
      option for limiting vinyl dump/compaction rate.
      
      Closes #3220
      62830170
    • Vladimir Davydov's avatar
      xlog: use ev_sleep instead of fiber_sleep for rate limiting · 5d5f0d1c
      Vladimir Davydov authored
      fiber_sleep() works only if the current thread was created with
      cord_costart(). Since vinyl worker threads don't need fibers, they
      are created with cord_start() and hence can't use fiber_sleep().
      So to be able to limit rate of vinyl dump/compaction, we have to
      use ev_sleep() instead of fiber_sleep() in xlog. This is fine by
      other xlog writers, because they don't use fibers either, neither
      they should as xlogs are written without coio.
      
      Needed for #3220
      5d5f0d1c
  5. Jun 02, 2018
  6. Jun 01, 2018
Loading