Skip to content
Snippets Groups Projects
  1. Mar 20, 2020
    • Vladislav Shpilevoy's avatar
      box: on cfg properly check memory quota · 74cec5f5
      Vladislav Shpilevoy authored
      
      box_check_config() didn't check memtx_memory and vinyl_memory
      upper bound. As a result, it was possible to set memory size
      higher than what the quota allows as maximum.
      
      That worked only when box.cfg() was called first time, because
      quota_init() does not check its value. Subsequent box.cfg() calls
      use quota_set(), which aborts the program if a size is too big.
      Only in debug mode. In release quota_set() also worked with any
      sizes.
      
      Closes #4705
      
      Reviewed-by: default avatarIgor Munkin <imun@tarantool.org>
      Reviewed-by: default avatarNikita Pettik <korablev@tarantool.org>
      74cec5f5
  2. Mar 19, 2020
    • Vladislav Shpilevoy's avatar
      iproto: show real port in logs and box.info.listen · 2c3acf2b
      Vladislav Shpilevoy authored
      Box.cfg{listen = 0} automatically chooses a port. But it was
      impossible to obtain a real port the instance is bound to.
      
      An ability to see a real port may help to make test-run more
      robust, because it won't depend on which ports are free, and
      won't need to pre-choose them in advance.
      
      Now box.info.listen shows a real address, or nil when listen is
      turned off. Also a real address is logged instead of the dummy
      0-port one.
      
      Closes #4620
      
      @TarantoolBot document
      Title: box.info.listen - real address
      
      New value in box.info - listen. It is a real address to which the
      instance was bound. For example, if box.cfg.listen was set with
      a zero port, box.info.listen will show a real port. The address
      is stored as a string:
      
          - unix/:<path> for UNIX domain sockets;
          - <ip>:<port> for IPv4;
          - [ip]:<port> for IPv6.
      
      If the instance does not listen anything, box.info.listen is nil.
      2c3acf2b
  3. Mar 18, 2020
    • Oleg Babin's avatar
      box: allow to retrieve the last generated value of sequence · 64c69fe0
      Oleg Babin authored
      
      This patch introduces "current" function for sequences.
      It returns the last retrieved value of specified sequence or
      throws an error if no value has been generated yet.
      
      This patch partially reverts 3ff1f1e3
      (box: remove sequence_get) here similar function "get" was removed
      to avoid possible misleading with "currval" function of PosgreSQL
      that returns the last obtained value of the sequence in the scope
      of current session. In contrast "current" returns the last globally
      retrieved value of the sequence.
      
      Closes #4752
      
      Reviewed-by: default avatarVladislav Shpilevoy <v.shpilevoy@tarantool.org>
      Reviewed-by: default avatarNikita Pettik <korablev@tarantool.org>
      
      @TarantoolBot document
      Title: sequence:current()
      
      This patch introduces "current" function for sequences.
      It returns the last retrieved value of specified sequence or
      throws an error if no value has been generated yet ("next"
      has not been called yet or right after "reset" is called).
      
      Lua:
      
      Example:
      
      ```lua
      sq = box.schema.sequence.create('test')
      ---
      ...
      sq:current()
      ---
      - error: Sequence 'test' is not started
      ...
      sq:next()
      ---
      - 1
      ...
      sq:current()
      ---
      - 1
      ...
      sq:set(42)
      ---
      ...
      sq:current()
      ---
      - 42
      ...
      sq:reset()
      ---
      ...
      sq:current()  -- error
      ---
      - error: Sequence 'test' is not started
      ...
      ```
      
      C API:
      
      ```C
      int
      box_sequence_current(uint32_t seq_id, int64_t *result);
      ```
      
      Where:
        * seq_id - sequence identifier;
        * result - pointer to a variable where the current sequence
        value will be stored on success.
      
      Returns 0 on success and -1 otherwise. In case of an error user
      could get it via `box_error_last()`.
      64c69fe0
  4. Mar 17, 2020
  5. Mar 16, 2020
    • Vladislav Shpilevoy's avatar
      tuple: document all missed box.tuple.* methods · adc98575
      Vladislav Shpilevoy authored
      In #4684 it was found that box.tuple.* contained some private
      functions: bless(), encode(), and is().
      
      Bless() and encode() didn't make any sense for a user, so they
      were hidden into box.internal.tuple.*.
      
      But box.tuple.is() is actually a useful thing. It is harnessed in
      the tests a lot, and is likely to be already used by customers,
      because it is available in box.tuple.* for a long time. It is a
      matter of time when someone will open a doc ticket saying that
      box.tuple.is() is not documented. The patch makes it legally
      public.
      
      Follow-up #4684
      
      @TarantoolBot document
      Title: box.tuple.is()
      ```Lua
      box.tuple.is(object)
      ```
      A function to check whether a given object is a tuple cdata
      object. Returns true or false. Never raises nor returns an error.
      adc98575
    • Chris Sosnin's avatar
      sequence: return correct error on using dropped sequence · afe4007d
      Chris Sosnin authored
      
      This code is called from C, so it shouldn't throw.
      
      Closes #4753
      
      Reviewed-by: default avatarVladislav Shpilevoy <v.shpilevoy@tarantool.org>
      Reviewed-by: default avatarNikita Pettik <korablev@tarantool.org>
      afe4007d
  6. Mar 10, 2020
    • Olga Arkhangelskaia's avatar
      memtx: fix out of memory handling for rtree · 3b4fbdc0
      Olga Arkhangelskaia authored
      When tarantool tries to recover rtree from a snapshot and memtx_memory value
      is lower than it has been when the snapshot was created, server suffers from
      segmentation fault. This happens because there is no out of memory error
      handling in rtree lib. In another words, we do not check the result of
      malloc operation.
      The execution flow in case of recovery uses different way and the secondary
      keys are build in batches. That way has no checks and reservations.
      The patch adds memtx_rtree_index_reserve implementation to make sure that any
      memory allocation in rtree will fail. Although this gives us no additional
      optimization as in case of memtx_tree, the memory reservation prevents
      tarantool from segmentation fault. If there is not enough memory to be reserved
      server will fail gently with the "Failed to allocate" error message.
      
      Closes #4619
      3b4fbdc0
  7. Mar 08, 2020
    • Maria's avatar
      box: netbox.self and connect should work interchangeably · 0bcf9a59
      Maria authored
      Despite what was stated in the documentation, netbox.connect was not always
      equivalent to netbox.self. In particular, they converted tuple to different
      types - table and cdata respectively.
      
      The patch fixes the issue and covers all cases where netbox.self and connect
      perform conversion of types - e.g., for box.error.
      
      Closes #4513
      0bcf9a59
  8. Mar 05, 2020
    • Maria's avatar
      box: replication shouldn't leak user password · a806549d
      Maria authored
      It was possible to leak user password through setting 'replication'
      configuration option in first box.cfg invocation. This happened due
      to unconditional logging in load_cfg function. The patch introduces
      conditional logging.
      
      Closes #4493
      a806549d
  9. Mar 02, 2020
    • Serge Petrenko's avatar
      replication: do not relay rows coming from a remote instance back to it · ed2e1430
      Serge Petrenko authored
      We have a mechanism for restoring rows originating from an instance that
      suffered a sudden power loss: remote masters resend the isntance's rows
      received before a certain point in time, defined by remote master vclock
      at the moment of subscribe.
      However, this is useful only on initial replication configuraiton, when
      an instance has just recovered, so that it can receive what it has
      relayed but haven't synced to disk.
      In other cases, when an instance is operating normally and master-master
      replication is configured, the mechanism described above may lead to
      instance re-applying instance's own rows, coming from a master it has just
      subscribed to.
      To fix the problem do not relay rows coming from a remote instance, if
      the instance has already recovered.
      
      Closes #4739
      ed2e1430
  10. Feb 27, 2020
  11. Feb 25, 2020
    • Maria's avatar
      box: user.grant error should be versatile · e8009f41
      Maria authored
      Error message on granted privileges was not flexible and
      did not distinguish between universal or any other privileges
      leaving either 'nil' or simply '' at the end.
      
      Closes #714
      e8009f41
  12. Feb 24, 2020
  13. Feb 20, 2020
    • Vladislav Shpilevoy's avatar
      tuple: use field type in update of a float/double · daad1cb0
      Vladislav Shpilevoy authored
      There was a bug that float +/- float could result into infinity
      even if result fits a double. It was fixed by storing double or
      float depending on a result value. But it didn't take result field
      type into account. That led to a bug when a double field +/- a
      value fit the float range, and could be stored as float resulting
      into an error at attempt to create a tuple.
      
      Now if a field type is double in the tuple format, it will store
      double always, even if it fits the float range.
      
      Follow-up #4701
      daad1cb0
    • Vladislav Shpilevoy's avatar
      tuple: don't truncate float in :update() · fef4fdfc
      Vladislav Shpilevoy authored
      Before the patch there were the rules:
      * float +/- double = double
      * double +/- double = double
      * float +/- float = float
      
      The rules were applied regardless of values. That led to a problem
      when float + float exceeding maximal float value could fit into
      double, but was stored as an infinity.
      
      The patch makes so that if a floating point arithmetic operation
      result fits into float, it is stored as float. Otherwise as
      double. Regardless of initial types.
      
      This alongside saves some memory for cases when doubles can be
      stored as floats, and therefore takes 4 less bytes. Although
      these cases are rare, because any not integer value stored in a
      double may have a long garbage tail in its fraction.
      
      Closes #4701
      fef4fdfc
  14. Feb 14, 2020
  15. Jan 16, 2020
    • Oleg Babin's avatar
      error: add __concat method to error object · 935db173
      Oleg Babin authored
      Usually functions return pair `nil, err` and expected that err is string.
      Let's make the behaviour of error object closer to string
      and define __concat metamethod.
      
      The case of error "error_mt.__concat(): neither of args is an error"
      is not covered by tests because of #4723
      
      Closes #4489
      Unverified
      935db173
  16. Jan 14, 2020
    • Chris Sosnin's avatar
      box: frommap() bug fix · f89b5ab0
      Chris Sosnin authored
      - If an optional argument is provided for
        space_object:frommap() (which is {table = true|false}),
        type match for first arguments is omitted, which is
        incorrect. We should return the result only after making
        sure it is possible to build a tuple.
      
      - If there is a type mismatch, however, frommap() does not
        return nil, err as it is mentioned in the description, so we
        change it to be this way.
      
      Closes #4262
      f89b5ab0
  17. Jan 13, 2020
    • Chris Sosnin's avatar
      tuple: add argument length check for update() · b73fb421
      Chris Sosnin authored
      Currently tuple_object:update() does not check the length
      of operation string and just takes the first character
      after decoding. This patch fixes this problem.
      
      Follow-up #3884
      b73fb421
    • Chris Sosnin's avatar
      tuple: fix non-informative update() error message · d4fcec0c
      Chris Sosnin authored
      Calling tuple_object:update() with invalid argument number
      yields 'Unknown UPDATE operation' error. Instead, we replace this
      error with explicit "wrong argument number", mentioning which operation
      failed, or poiniting out at invalid operation code.
      
      Fixes #3884
      d4fcec0c
  18. Dec 31, 2019
    • Nikita Pettik's avatar
      sql: add cache statistics to box.info · 5a1a220e
      Nikita Pettik authored
      To track current memory occupied by prepared statements and number of
      them, let's extend box.info submodule with .sql statistics: now it
      contains current total size of prepared statements and their count.
      
      @TarantoolBot document
      Title: Prepared statements in SQL
      
      Now it is possible to prepare (i.e. compile into byte-code and save to
      the cache) statement and execute it several times. Mechanism is similar
      to ones in other DBs. Prepared statement is identified by numeric
      ID, which are returned alongside with prepared statement handle.
      Note that they are not sequential and represent value of hash function
      applied to the string containing original SQL request.
      Prepared statement holder is shared among all sessions. However, session
      has access only to statements which have been prepared in scope of it.
      There's no eviction policy like in any cache; to remove statement from
      holder explicit unprepare request is required. Alternatively, session's
      disconnect also removes statements from holder.
      Several sessions can share one prepared statement, which will be
      destroyed when all related sessions are disconnected or send unprepare
      request. Memory limit for prepared statements is adjusted by
      box.cfg{sql_cache_size} handle (can be set dynamically;
      
      Any DDL operation leads to expiration of all prepared statements: they
      should be manually removed or re-prepared.
      Prepared statements are available in local mode (i.e. via box.prepare()
      function) and are supported in IProto protocol. In the latter case
      next IProto keys are used to make up/receive requests/responses:
      IPROTO_PREPARE - new IProto command; key is 0x13. It can be sent with
      one of two mandatory keys: IPROTO_SQL_TEXT (0x40 and assumes string value)
      or IPROTO_STMT_ID (0x43 and assumes integer value). Depending on body it
      means to prepare or unprepare SQL statement: IPROTO_SQL_TEXT implies prepare
      request, meanwhile IPROTO_STMT_ID - unprepare;
      IPROTO_BIND_METADATA (0x33 and contains parameters metadata of type map)
      and IPROTO_BIND_COUNT (0x34 and corresponds to the count of parameters to
      be bound) are response keys. They are mandatory members of result of
      IPROTO_PREPARE execution.
      
      To track statistics of used memory and number of currently prepared
      statements, box.info is extended with SQL statistics:
      
      box.info:sql().cache.stmt_count - number of prepared statements;
      box.info:sql().cache.size - size of occupied by prepared statements memory.
      
      Typical workflow with prepared statements is following:
      
      s = box.prepare("SELECT * FROM t WHERE id = ?;")
      s:execute({1}) or box.execute(s.sql_str, {1})
      s:execute({2}) or box.execute(s.sql_str, {2})
      s:unprepare() or box.unprepare(s.query_id)
      
      Structure of object is following (member : type):
      
      - stmt_id: integer
        execute: function
        params: map [name : string, type : integer]
        unprepare: function
        metadata: map [name : string, type : integer]
        param_count: integer
      ...
      
      In terms of remote connection:
      
      cn = netbox:connect(addr)
      s = cn:prepare("SELECT * FROM t WHERE id = ?;")
      cn:execute(s.sql_str, {1})
      cn:unprepare(s.query_id)
      
      Closes #2592
    • Nikita Pettik's avatar
      netbox: introduce prepared statements · 0e1b20c3
      Nikita Pettik authored
      This patch introduces support of prepared statements in IProto
      protocol. To achieve this new IProto command is added - IPROTO_PREPARE
      (key is 0x13). It is sent with one of two mandatory keys:
      IPROTO_SQL_TEXT (0x40 and assumes string value) or IPROTO_STMT_ID (0x43
      and assumes integer value). Depending on body it means to prepare or
      unprepare SQL statement: IPROTO_SQL_TEXT implies prepare request,
      meanwhile IPROTO_STMT_ID - unprepare.  Also to reply on PREPARE request a
      few response keys are added: IPROTO_BIND_METADATA (0x33 and contains
      parameters metadata of type map) and IPROTO_BIND_COUNT (0x34 and
      corresponds to the count of parameters to be bound).
      
      Part of #2592
      0e1b20c3
    • Nikita Pettik's avatar
      box: introduce prepared statements · 7bea3d5b
      Nikita Pettik authored
      This patch introduces local prepared statements. Support of prepared
      statements in IProto protocol and netbox is added in the next patch.
      
      Prepared statement is an opaque instance of SQL Virtual Machine. It can
      be executed several times without necessity of query recompilation. To
      achieve this one can use box.prepare(...) function. It takes string of
      SQL query to be prepared; returns extended set of meta-information
      including statement's ID, parameter's types and names, types and names
      of columns of the resulting set, count of parameters to be bound.  Lua
      object representing result of :prepare() invocation also features two
      methods - :execute() and :unprepare(). They correspond to
      box.execute(stmt.stmt_id) and box.unprepare(stmt.stmt_id), i.e.
      automatically substitute string of prepared statement to be executed.
      Statements are held in prepared statement cache - for details see
      previous commit.  After schema changes all prepared statement located in
      cache are considered to be expired - they must be re-prepared by
      separate :prepare() call (or be invalidated with :unrepare()).
      
      Two sessions can share one prepared statements. But in the current
      implementation if statement is executed by one session, another is
      not able to use it and will compile it from scratch and than execute.
      
      SQL cache memory limit is regulated by box{sql_cache_size} which can be
      set dynamically. However, it can be set to the value which is less than
      the size of current free space in cache (since otherwise some statements
      can disappear from cache).
      
      Part of #2592
      7bea3d5b
    • Nikita Pettik's avatar
      sql: introduce holder for prepared statemets · 10ebc2d5
      Nikita Pettik authored
      This patch introduces holder (as data structure) to handle prepared
      statements and a set of interface functions (insert, delete, find) to
      operate on it. Holder under the hood is implemented as a global hash
      (keys are values of hash function applied to the original string containing
      SQL query; values are pointer to wrappers around compiled VDBE objects) and
      GC queue. Each entry in hash has reference counter. When it reaches 0
      value, entry is moved to GC queue. In case prepared statements holder is
      out of memory, it launches GC process: each entry in GC queue is deleted
      and all resources are released. Such approach allows to avoid workload
      spikes on session's disconnect (since on such event all statements must
      be deallocated).
      Each session is extended with local hash to map statement ids available
      for it. That is, session is allowed to execute and deallocate only
      statements which are previously prepared in scope of this session.
      On the other hand, global hash makes it possible to share same prepared
      statement object among different sessions.
      Size of cache is regulated via box.cfg{sql_cache_size} parameter.
      
      Part of #2592
      10ebc2d5
    • Mergen Imeev's avatar
      sql: remove control pragmas · eafadc13
      Mergen Imeev authored
      This patch removes control pragmas. They are not needed now, after
      the introduction of the _session_settings system space.
      
      Closes #4511
      eafadc13
    • Mergen Imeev's avatar
      box: add SQL settings to _session_settings · 4655447c
      Mergen Imeev authored
      Part of #4511
      
      @TarantoolBot document
      Title: _session_settings system space
      The _session_settings system space used to view or change session
      settings.
      
      This space uses a new engine. This allows us to create tuples on
      the fly when the get() or select() methods are called. This
      engine does not support the insert(), replace(), and delete()
      methods. The only way to change the setting value is update(),
      which can only be used with the "=" operation.
      
      Because space creates a tuple on the fly, it allows us to get a
      tuple without saving it anywhere. But this means that every time
      we get a tuple from this system space, it is a new tuple, even if
      they look the same:
      
      tarantool> s = box.space._session_settings
      tarantool> name = 'sql_default_engine'
      tarantool> s:get({name}) == s:get({name})
      ---
      - false
      ...
      
      Currently, this space contains only SQL settings, since the only
      session settings are SQL settings.
      
      List of currently available session settings:
      
      sql_default_engine
      sql_defer_foreign_keys
      sql_full_column_names
      sql_parser_debug
      sql_recursive_triggers
      sql_reverse_unordered_selects
      sql_select_debug
      sql_vdbe_debug
      
      The default values of these settings cannot be changed by the
      user.
      
      Debug settings are disabled by default and can only be enabled in
      the debug build.
      
      Example of usage:
      tarantool> s = box.space._session_settings
      -- View session settings values.
      tarantool> s:get({'sql_default_engine'})
      ---
      - ['sql_default_engine', 'memtx']
      ...
      
      tarantool> s:select()
      
      s:select()
      ---
      - - ['sql_default_engine', 'memtx']
        - ['sql_defer_foreign_keys', false]
        - ['sql_full_column_names', false]
        - ['sql_full_metadata', false]
        - ['sql_parser_debug', false]
        - ['sql_recursive_triggers', true]
        - ['sql_reverse_unordered_selects', false]
        - ['sql_select_debug', false]
        - ['sql_vdbe_debug', false]
      ...
      
      tarantool> s:select('sql_g', {iterator='LE'})
      ---
      - - ['sql_full_metadata', false]
        - ['sql_full_column_names', false]
        - ['sql_defer_foreign_keys', false]
        - ['sql_default_engine', 'memtx']
      ...
      
      -- Change session setting value.
      tarantool> s:update('sql_default_engine', {{'=', 'value', 'vinyl'}})
      ---
      - ['sql_default_engine', 'vinyl']
      ...
      4655447c
    • Mergen Imeev's avatar
      box: introduce _session_settings system space · 10d79f7a
      Mergen Imeev authored
      This patch creates _session_settings system space. This space is
      used to view and change session settings. This space is one of the
      special spaces that have a "service" engine. The main idea of this
      space is that it will not store tuples, but when you call the
      get() or select() methods, it creates tuples on the fly. Because
      of this, even if the same setting is asked, the returned tuples
      will be different. In addition, this space allows you to change
      the setting value using the update() method, in which case it
      directly changes the setting value.
      
      There are no settings at the moment, some will be added in the
      next patch.
      
      Part of #4511
      10d79f7a
    • Mergen Imeev's avatar
      box: introduce 'service' engine · 81bb565c
      Mergen Imeev authored
      This patch introduces a new engine called "service" that will be
      used to create a new system space. The main idea of this engine is
      that it will not have a predefined space_vtab. With this engine,
      we can create unusual spaces with their own vtab and behavior.
      
      Due to the nature of this engine, it can only be used to create
      system spaces.
      
      Part of #4511
      81bb565c
  19. Dec 30, 2019
    • Vladislav Shpilevoy's avatar
      tuple: JSON path update intersection at maps · 6e97d6a9
      Vladislav Shpilevoy authored
      Previous commits introduced isolated JSON updates, and then
      allowed intersection at array. This one completes the puzzle,
      adding intersection at maps, so now both these samples work:
      
      Allowed in the previous commit:
      
          [1][2][3].a.b.c = 20
          [1][2][4].e.f.g = 30
                 ^
      
          First difference is [3] vs [4] - intersection by an array.
      
      Allowed in this commit:
      
          [1][2][3].a.b.c = 20
          [1][2][3].a.e.f = 30
                      ^
      
          First difference is 'b' vs 'e' - intersection by a map.
      
      Now JSON updates are fully available.
      
      Closes #1261
      
      @TarantoolBot document
      Title: JSON updates
      Tuple/space/index:update/upsert now support JSON paths. All the
      same paths as allowed in tuple["..."].
      
      Example:
      box.cfg{}
      format = {}
      format[1] = {'field1', 'unsigned'}
      format[2] = {'field2', 'map'}
      format[3] = {'field3', 'array'}
      format[4] = {'field4', 'string', is_nullable = true}
      s = box.schema.create_space('test', {format = format})
      _ = s:create_index('pk')
      t = {
          1,
          {
              key1 = 'value',
              key2 = 10
          },
          {
              2,
              3,
              {key3 = 20}
          }
      }
      t = s:replace(t)
      
      tarantool> t:update({{'=', 'field2.key1', 'new_value'}})
      ---
      - [1, {'key1': 'new_value', 'key2': 10}, [2, 3, {'key3': 20}]]
      ...
      
      tarantool> t:update({{'+', 'field3[2]', 1}})
      ---
      - [1, {'key1': 'value', 'key2': 10}, [2, 4, {'key3': 20}]]
      ...
      
      tarantool> s:update({1}, {{'!', 'field4', 'inserted value'}})
      ---
      - [1, {'key1': 'value', 'key2': 10}, [2, 3, {'key3': 20}], 'inserted value']
      ...
      
      tarantool> s:update({1}, {{'#', '[2].key2', 1}, {'=', '[3][3].key4', 'value4'}})
      ---
      - [1, {'key1': 'value'}, [2, 3, {'key3': 20, 'key4': 'value4'}], 'inserted value']
      ...
      
      tarantool> s:upsert({1, {k = 'v'}, {}}, {{'#', '[2].key1', 1}})
      ---
      ...
      
      tarantool> s:select{}
      ---
      - - [1, {}, [2, 3, {'key3': 20, 'key4': 'value4'}], 'inserted value']
      ...
      
      Note, that there is the same rule, as in tuple field access by
      JSON, for field names looking like JSON paths. The rule is that
      firstly the whole path is interpreted as a field name. If such a
      name does not exist, then it is treated as a path. For example,
      if there is a field name 'field.name.like.json', then this update
      
          <obj>:update({..., 'field.name.like.json', ...})
      
      will update this field, instead of keys 'field' -> 'name' ->
      'like' -> 'json'. If such a name is needed as a part of a bigger
      path, then it should be wrapped in quotes and []:
      
          <obj>:update({..., '["field.name.like.json"].next.fields', ...})
      
      There are some new rules for JSON updates:
      
      - Operation '!' can't be used to create all intermediate nodes of
        a path. For example, {'!', 'field1[1].field3', ...} can't
        create fields 'field1' and '[1]', they should exist.
      
      - Operation '#', when applied to maps, can't delete more than one
        key at once. That is, its argument should be always 1 for maps.
      
          {'#', 'field1.field2', 1} - this is allowed;
          {'#', 'field1.field2', 10} - this is not.
      
        That limitation originates from a problem, that keys in a map
        are not ordered anyhow, and '#' with more than 1 key would lead
        to undefined behaviour.
      
      - Operation '!' on maps can't create a key, if it exists already.
      
      - If a map contains non-string keys (booleans, numbers, maps,
        arrays - anything), then these keys can't be updated via JSON
        paths. But it is still allowed to update string keys in such a
        map.
      
      Why JSON updates are good, and should be preferred when only a
      part of a tuple needs to be updated?
      
      - They consume less space in WAL, because for an update only its
        keys, operations, and arguments are stored. It is cheaper to
        store update of one deep field, than the whole tuple.
      
      - They are faster. Firstly, this is because they are implemented
        in C, and have no problems with Lua GC and dynamic typing.
        Secondly, some cases of JSON paths are highly optimized. For
        example, an update with a single JSON path costs O(1) memory
        regardless of how deep that path goes (not counting update
        arguments).
      
      - They are available from remote clients, as well as any other
        DML. Before JSON updates to update one deep part of a tuple it
        would be necessary to download that tuple, update it in memory,
        send it back - 2 network hops. With JSON paths it can be 1 when
        the update can be described in paths.
      6e97d6a9
    • Vladislav Shpilevoy's avatar
      tuple: JSON path update intersection at arrays · 8cad025a
      Vladislav Shpilevoy authored
      Before the patch only isolated JSON updates were supported as the
      simplest and fastest to implement. This patch allows update
      operations with paths having the same prefix. But difference of
      the paths should start from an array index.
      
      For example, this is allowed:
      
          [1][2][3].a.b.c = 20
          [1][2][4].e.f.g = 30
      
          First difference is [3] vs [4] - intersection by an array, ok.
      
      This is not allowed yet:
      
          [1][2][3].a.b.c = 20
          [1][2][3].a.e.f = 30
      
          First difference is 'b' vs 'e' - intersection by a map,
          not ok.
      
      For that a new update tree node type is added - XUPDATE_ROUTE.
      When several update operations have the same prefix, this prefix
      becomes an XUPDATE_ROUTE tree field. It stores the prefix and a
      subtree with these operations.
      
      Bar and route update nodes can branch and produce more bars and
      routes, when new operations come.
      
      Part of #1261
      8cad025a
    • Serge Petrenko's avatar
      replication: introduce anonymous replica. · e17beed8
      Serge Petrenko authored
      This commit introduces anonymous replicas. Such replicas do not pollute
      _cluster table (they can only be read-only and have a zero id in return).
      An anonymous replica can be promoted to a normal one if needed.
      
      Closes #3186
      
      @TarantoolBot document
      Title: Document anonymous replica
      
      There is a new type of replica in tarantool, anonymous one. Anonymous
      replica is read-only (but you still can write to temporary and
      replica-local spaces), and it isn't present in _cluster table.
      
      Since anonymous replica isn't registered in _cluster table, there is no
      limitation for anonymous replica count in a replicaset. You can have as
      many of them as you want.
      
      In order to make a replica anonymous, you have to pass an option
      `replication_anon=true` to `box.cfg`. You also have to set 'read_only'
      to true.
      
      Let's go through anonymous replica bootstrap.
      Suppose we have a master configured with
      ```
      box.cfg{listen=3301}
      ```
      And created a local space called "loc"
      ```
      box.schema.space.create('loc', {is_local=true})
      box.space.loc:create_index("pk")
      ```
      Now, to configure an anonymous replica, we have to issue `box.cfg`,
      as usual.
      ```
      box.cfg{replication_anon=true, read_only=true, replication=3301}
      ```
      As mentioned above, `replication_anon` may be set to true only together
      with `read_only`
      The instance will fetch masters snapshot and proceed to following its
      changes. It will not receive an id so its id will remain zero.
      ```
      tarantool> box.info.id
      ---
      - 0
      ...
      ```
      ```
      tarantool> box.info.replication
      ---
      - 1:
          id: 1
          uuid: 3c84f8d9-e34d-4651-969c-3d0ed214c60f
          lsn: 4
          upstream:
            status: follow
            idle: 0.6912029999985
            peer:
            lag: 0.00014615058898926
      ...
      ```
      Now we can use the replica.
      For example, we may do inserts into the local space:
      ```
      tarantool> for i = 1,10 do
               > box.space.loc:insert{i}
               > end
      ---
      ...
      ```
      Note, that while the instance is anonymous, it will increase the 0-th
      component of its vclock:
      ```
      tarantool> box.info.vclock
      ---
      - {0: 10, 1: 4}
      ...
      ```
      Let's now promote the replica to a normal one:
      ```
      tarantool> box.cfg{replication_anon=false}
      2019-12-13 20:34:37.423 [71329] main I> assigned id 2 to replica 6a9c2ed2-b9e1-4c57-a0e8-51a46def7661
      2019-12-13 20:34:37.424 [71329] main/102/interactive I> set 'replication_anon' configuration option to false
      ---
      ...
      
      tarantool> 2019-12-13 20:34:37.424 [71329] main/117/applier/ I> subscribed
      2019-12-13 20:34:37.424 [71329] main/117/applier/ I> remote vclock {1: 5} local vclock {0: 10, 1: 5}
      2019-12-13 20:34:37.425 [71329] main/118/applierw/ C> leaving orphan mode
      ```
      The replica just received id 2. We can make it read-write now.
      ```
      box.cfg{read_only=false}
      2019-12-13 20:35:46.392 [71329] main/102/interactive I> set 'read_only' configuration option to false
      ---
      ...
      
      tarantool> box.schema.space.create('test')
      ---
      - engine: memtx
        before_replace: 'function: 0x01109f9dc8'
        on_replace: 'function: 0x01109f9d90'
        ck_constraint: []
        field_count: 0
        temporary: false
        index: []
        is_local: false
        enabled: false
        name: test
        id: 513
      - created
      ...
      
      tarantool> box.info.vclock
      ---
      - {0: 10, 1: 5, 2: 2}
      ...
      ```
      Now replica tracks its changes in 2nd vclock component, as expected.
      It can also become replication master from now on.
      
      Side notes:
        * You cannot replicate from an anonymous instance.
        * To promote an anonymous instance to a regular one,
          you first have to start it as anonymous, and only
          then issue `box.cfg{replication_anon=false}`
        * In order for the deanonymization to succeed, the
          instance must replicate from some read-write instance,
          otherwise noone will be able to add it to _cluster table.
      e17beed8
  20. Dec 29, 2019
  21. Dec 28, 2019
    • Nikita Pettik's avatar
      sql: display line and position in syntax errors · ea958f41
      Nikita Pettik authored
      When it comes for huge queries, it may turn out to be useful to see
      exact position of occurred error. Hence, let's now display line and
      position within line near which syntax error takes place. Note that it
      can be done only during parsing process (since AST can be analysed only
      after its construction is completed), so most of semantic errors still
      don't contain position. A few errors have been reworked to match new
      formatting patterns.
      
      First iteration of this patch is implemented by @romanhabibov
      
      Closes #2611
      ea958f41
  22. Dec 19, 2019
    • Mergen Imeev's avatar
      lua/serializer: encode Lua number -2^63 as integer · 06aeb768
      Mergen Imeev authored
      This patch fixes a bug that appeared after commit
      3a13be1d ('lua: fix lua_tofield()
      for 2**64 value') .
      
      Due to this bug, -2^63 was serialized as double, although it
      should be serialized as integer.
      
      Closes #4672
      06aeb768
    • Vladislav Shpilevoy's avatar
      tuple: enable isolated JSON updates · 940133ae
      Vladislav Shpilevoy authored
      Isolated tuple update is an update by JSON path, which hasn't a
      common prefix with any other JSON update operation in the same
      set. For example, these JSON update operations are isolated:
      
          {'=', '[1][2][3]', 100},
          {'+', '[2].b.c', 200}
      
      Their JSON paths has no a common prefix. But these operations are
      not isolated:
      
          {'=', '[1][2][3]', 100},
          {'+', '[1].b.c', 200}
      
      They have a common prefix '[1]'.
      
      Isolated updates are a first part of fully functional JSON
      updates. Their feature is that their implementation is relatively
      simple and lightweight - an isolated JSON update does not store
      each part of the JSON path as a separate object. Isolated update
      stores just string with JSON and pointer to the MessagePack
      object to update.
      
      Such isolated updates are called 'bar update'. They are a basic
      brick of more complex JSON updates.
      
      Part of #1261
      940133ae
  23. Dec 11, 2019
  24. Dec 10, 2019
    • Vladislav Shpilevoy's avatar
      func: fix not unloading of unused modules · ca07088c
      Vladislav Shpilevoy authored
      C functions are loaded from .so/.dylib dynamic libraries. A
      library is loaded when any function from there is called first
      time. And was supposed to be unloaded, when all its functions are
      dropped from the schema (box.schema.func.drop()), and none of them
      is still in a call. But the unloading part was broken.
      
      In fact, box.schema.func.drop() never unloaded anything. Moreover,
      when functions from the module were added again without a restart,
      it led to a second mmap of the same module. And so on, the same
      library could be loaded any number of times.
      
      The problem was in a useless flag in struct module preventing its
      unloading even when it is totally unused. It is dropped.
      
      Closes #4648
      ca07088c
Loading