Skip to content
Snippets Groups Projects
  1. Dec 28, 2017
  2. Dec 26, 2017
    • Vladimir Davydov's avatar
      Add box.info.memory() to report aggregated memory statistics · 4240860f
      Vladimir Davydov authored
      This patch adds a new function, box.info.memory(). The functions returns
      a table with the following fields:
      
       - data - size of memory in bytes used for storing user data (i.e.
         tuples) in memtx and in vinyl level 0, without taking into account
         memory fragmentation.
      
       - index - size of memory in bytes used for indexing user data. This
         includes memtx and vinyl memory tree extents, vinyl page index, vinyl
         bloom filters.
      
       - cache - size of memory in bytes used for caching user data. Memtx
         doesn't have cache so basically this is the size of the vinyl tuple
         cache.
      
       - tx - size of memory in bytes used up by active transactions. For
         vinyl it is the total size of all allocated struct txv, struct vy_tx,
         struct vy_read_interval, plus tuples pinned by those objects. For
         memtx it is going to be 0 for now as memtx transaction manager
         shouldn't consume much memory. We may want to account struct txn and
         struct txn_stmt there too in future.
      
       - net - size of memory in bytes used up by network input and output
         buffers.
      
       - lua - size of memory used by the Lua runtime.
      
      It is supposed to be used by the admin to get a general knowledge about
      what's going on with a particular tarantool instance. For more info, per
      subsystem statistics are supposed to be used (e.g. box.info.vinyl()).
      
      Closes #934
      4240860f
    • Vladimir Davydov's avatar
      iproto: account memory used by network buffers · 8516cbc3
      Vladimir Davydov authored
      Add function iproto_mem_used() that returns the total amount of memory
      allocated for storing input and output buffers. It will be used by
      box.info.memory() implementation to show aggregated network statistics.
      
      Note, to account memory used by output buffers, we have to introduce a
      separate slab cache (currently, the cache of the tx cord is used).
      
      Needed for #934
      8516cbc3
    • Vladimir Davydov's avatar
      vinyl: account memory used by statements in read and write sets · 49b3f748
      Vladimir Davydov authored
      To be reported by box.info.memory().tx
      
      Needed for #934
      49b3f748
    • Vladimir Davydov's avatar
      vinyl: account total size of page index and bloom · 4fc2fbdf
      Vladimir Davydov authored
      To be reported by box.info.memory().index
      
      Needed for #934
      4fc2fbdf
    • Vladimir Davydov's avatar
      vinyl: account memory occupied by tree extents · 28a24984
      Vladimir Davydov authored
      To be reported by box.info.memory().index
      
      Needed for #934
      28a24984
    • Ilya's avatar
      Fix access checks on CALL · 62ec6c1b
      Ilya authored
      CALL should check only EXECUTE access on universe instead of
      READ, WRITE, EXECUTE.
      
      Closes #3017
      62ec6c1b
  3. Dec 22, 2017
    • Konstantin Osipov's avatar
      say: workaround gh-3014 · 3359b88b
      Konstantin Osipov authored
      At least print an error message when we try to set json log format
      for syslog log type.
      Do not iterate over all loggers on each SIGHUP.
      Style fixes.
      3359b88b
  4. Dec 21, 2017
    • Konstantin Osipov's avatar
      say: a few review fixes · 9737269d
      Konstantin Osipov authored
      * fix a bug in opening syslog file descriptor on Mac (Darwin)
      * add comments
      9737269d
    • Vladislav Shpilevoy's avatar
      schema: inherit default index parts from space format · 4bbd8d1b
      Vladislav Shpilevoy authored
      If index parts are not specied, default part types ignore space format.
      Lets use in default index parts types from a space format.
      
      Closes #2893
      4bbd8d1b
    • Ilya's avatar
      say: fix format functions · 3c61cd03
      Ilya authored
      * Add check on NULL filename in format functions
      * The need of this fix was inspired by possible need of custom loggers
      which want to reuse our format functions
      3c61cd03
    • Ilya's avatar
      say: Add several loggers support · 6d7437da
      Ilya authored
      * Add struct log
      * Add possibility to add logger configuration to global scope
      * Refactor functions in say to use them with specified config,
       not only global variable
      
       This patch was inspired by need of additional logger in audit log
       Relates #2912
      6d7437da
    • Ilya's avatar
      say: Refactor system error processing · f59ac234
      Ilya authored
      * Pull error processing from functions say_XXX_init(file, pipe, sys)
      to say_log_init
      * Remove error processing with passing point error
      message as an argument
      * Add new exception IllegalParameters
      f59ac234
  5. Dec 20, 2017
  6. Dec 19, 2017
    • Vladislav Shpilevoy's avatar
      alter: swap key_def memory on index alter · 56f970e1
      Vladislav Shpilevoy authored
      If an index is modified, in index_def_swap it does not swap key_defs.
      But index alter can change part types and comparators. Lets swap
      key_defs too. Note, that we can not swap key_defs by pointers - they
      are stored in memtx trees.
      
      Closes #3000
      56f970e1
    • Konstantin Osipov's avatar
    • Vladimir Davydov's avatar
      vinyl: force read view in iterator in autocommit mode · a31c2c10
      Vladimir Davydov authored
      Every iteration over a secondary index tracks a point in the transaction
      manager (due to lookup in the primary index). As a result, if the user
      calls 'select' or 'pairs' over a huge data set, it will consume a lot of
      memory due to this tracked points, even if the user doesn't uses
      transactions.
      
      To mitigate this, let's send all read only transactions to read view
      immediately so that tracking is disabled completely during iteration.
      Note, with this patch select() called outside a transaction doesn't
      populate the cache any more, but it seems to be OK as caching large
      select() requests results in cache thrashing.
      
      Closes #2534
      a31c2c10
    • Konstantin Osipov's avatar
      schema.cc: remove unused includes · c45f4e37
      Konstantin Osipov authored
      c45f4e37
    • Vladimir Davydov's avatar
      vinyl: pass read view to vy_index_get explicitly · 3c2a6d30
      Vladimir Davydov authored
      Currently, if tx is not NULL, tx->read_view is used, otherwise the
      global read view is used. For the sake of #2534 (read view for read only
      autocommit statements), we will need to pass an arbitrary read view to
      this function. So let's add the corresponding argument. This also allows
      us to drop env from the argument list.
      
      While we are at it, let's also inline vy_index_full_by_stmt() as it is a
      trivial wrapper around vy_index_get().
      
      Needed for #2534
      3c2a6d30
    • Vladimir Davydov's avatar
      vinyl: fix latency accounting for point lookups · 2084b8df
      Vladimir Davydov authored
      We don't account latency of point lookups, neither we emit a warning if
      a point lookup took > too_long_threshold. To fix this, let's move the
      too_long_threshold configuration parameter from vy_env to vy_index_env
      and check it in both vy_read_iterator_next() and vy_point_lookup().
      This will also allow us not to pass too_long_threshold (or vy_env for
      that matter) all the way down to read iterator initialization.
      2084b8df
    • Konstantin Osipov's avatar
      vinyl: properly initialize vy_index_env::index_count · 8db04112
      Konstantin Osipov authored
      The variable is only used for asserts, which
      started to fail sporadically because of a missing initialization
      since the last patch.
      8db04112
    • Vladimir Davydov's avatar
      vinyl: turn point iterator into a function · a59f93b7
      Vladimir Davydov authored
      The point iterator is not actually an iterator: it doesn't have an
      internal state and it acts as a function. Wrapping it into the iterator
      protocol only complicates its usage. Let's turn it into a function.
      a59f93b7
    • Vladimir Davydov's avatar
      vinyl: do not extract key for lookup in primary index · 0808cddc
      Vladimir Davydov authored
      This is not necessary, because if the index is primary, its key_def and
      cmp_def are the same and so the read iterator will not compare extra
      tuple parts as it is the case for secondary indexes.
      0808cddc
    • Vladimir Davydov's avatar
      vinyl: truncate region after extracting key from tuple · 939c2f9a
      Vladimir Davydov authored
      tuple_extract_key() uses region to store the result. If the region is
      not truncated, as it is currently the case, the memory consumption can
      rocket sky high during a transaction execution. This is especially
      critical in case of select() or pairs() over a secondary index.
      
      To fix that, let's introduce vy_stmt_extract_key() wrapper, which would
      store the result on malloc, and use it throughout the code. Note, this
      doesn't add malloc() invocations, because malloc() has to be invoked by
      vy_index_get() anyway - we just move it up.
      939c2f9a
    • Konstantin Osipov's avatar
  7. Dec 18, 2017
  8. Dec 17, 2017
    • Vladimir Davydov's avatar
      vinyl: cleanup usage of tuple comparison functions · ad7dc787
      Vladimir Davydov authored
       - Use vy_tuple_compare() instead of vy_stmt_compare() in places where
         we know that both arguments are tuples (type != SELECT).
      
       - Use vy_tuple_compare() instead of tuple_compare() in the write
         iterator to assure that none of its arguments happens to be a key.
      
       - Use vy_stmt_compare() instead of vy_tuple_compare_with_key() in the
         read iterator when comparing the resulting statement to the search
         key, because the search key may be a tuple.
      ad7dc787
  9. Dec 16, 2017
    • Vladimir Davydov's avatar
      vinyl: fix crash in vy_read_iterator_evaluate_src · 310e0715
      Vladimir Davydov authored
      There's an optimization in vy_read_iterator_evaluate_src(): in case
      the evaluated statement exactly matches the search key, we assume
      that it precedes all statements that have been evaluated so far.
      This assumption holds when we are looking for the next key, but it
      results in a crash when we are applying UPSERTs:
      
        vy_read_iterator_evaluate_src: Assertion `vy_read_iterator_cmp_stmt(itr, src->stmt, itr->curr_stmt) < 0' failed
      
      Actually, this optimization is rather dubious. It saves us one
      comparison on the first iteration if the iterator type is LE/GE, the
      search key is full, and there's an exact match. However, if there's
      no exact match, it will add an extra comparison instead. One might
      argue that we should optimize for exact match because of space.get(),
      but for space.get(), the read iterator is not invoked at all - we use
      point lookup instead. That being said, let's simply zap this
      "optimization" altogether to fix this crash.
      
      Closes #3003
      310e0715
  10. Dec 15, 2017
    • Konstantin Osipov's avatar
    • Vladislav Shpilevoy's avatar
      schema: remove unused error codes · e37329ed
      Vladislav Shpilevoy authored
      e37329ed
    • Georgy Kirichenko's avatar
      Handle SystemError in applier fiber · 33874b44
      Georgy Kirichenko authored
      A DNS issue can raise a SystemError, applier should handle this error
      
      Fixed #3001
      33874b44
    • Vladimir Davydov's avatar
      Get rid of iobuf · 46773dd3
      Vladimir Davydov authored
      It is not used anywhere anymore. Move readahead configuration
      to iproto.cc and zap ibuf.cc
      46773dd3
    • Vladimir Davydov's avatar
      applier: use ibuf instead of iobuf · e7d1390e
      Vladimir Davydov authored
      The obuf part of applier->iobuf is unused so let's replace the iobuf
      with ibuf. Couple of notes:
      
       - Do not bother about setting readahead for the ibuf (iobuf_readahead).
         We read rows one by one here anyway.
      
       - Do not free memory from the ibuf (iobuf_reset() does). The buffer may
         contain two rows at max, which is not a big deal.
      e7d1390e
    • Vladimir Davydov's avatar
      iproto: decouple input buffer from output buffer · e66b8f07
      Vladimir Davydov authored
      Output buffers are now rotated independently of input buffers. The
      rotation is done by the tx thread according to the following rules:
      
       - If both buffers are empty, choose any one.
       - If neither of buffers is empty, write to the current one.
       - If one of the buffers is empty while the other is not, choose
         the empty one (rotate).
      
      The output buffer is modified (rotated, written, reset) exclusively by
      the tx thread. The iproto thread just flushes its content to the socket.
      To propagate the output buffer state (iproto flush position and tx write
      position) between the two threads, we pass it in iproto_msg::wpos.
      
      The patch was originally written by @kostja. I just rebased it, fixed
      a couple of bugs, and added some comments.
      
      Note, it updates the tarantool/small version to bring obuf_svp_reset().
      
      Needed for #946
      e66b8f07
    • Vladimir Davydov's avatar
      vinyl: fold vy_stmt_extract_key · 4de8b4cb
      Vladimir Davydov authored
      Rationale:
       - It's used in the only place.
       - I want to reuse it for key extraction into a tuple on malloc.
      4de8b4cb
    • Vladimir Davydov's avatar
      vinyl: store pointer to vy_run_env in vy_run · bd6d138b
      Vladimir Davydov authored
      It's really annoying to pass vy_run_env along with vy_run or vy_slice
      every time we want to read a run. Let's store a pointer to vy_run_env
      immediately in vy_run. This is a widely accepted practice throughout
      vinyl - we already do this in case of vy_index, vy_cache, and vy_mem.
      bd6d138b
    • Vladimir Davydov's avatar
      vinyl: rename vy_stmt_env to vy_mem_env · 1125276f
      Vladimir Davydov authored
      vy_stmt_env doesn't really belong to vy_stmt infrastructure, actually it
      isn't used there at all, only created and destroyed. Let's rename it to
      vy_mem_env, move it to vy_mem.c, and replace vy_mem->allocator with a
      pointer to vy_mem_env. This will allow us to account memory tree blocks
      there.
      
      Needed for #934
      1125276f
    • Georgy Kirichenko's avatar
      replication: applier state channel can be closed · 8095ca65
      Georgy Kirichenko authored
      If applier state channel is closed by connect_all cleanup code then
      applier_connect function could throw an unwanted exception on a write
      to this channel.
      
      Interrupt applier_pause() on fiber_cancel().
      
      Applier orchestration uses fiber_cancel() to stop applier,
      so applier_pause() should exit if the fiber is cancelled.
      
      Add test.
      
      Fixes #2991.
      8095ca65
Loading