Skip to content
Snippets Groups Projects
  1. Apr 20, 2020
    • Kirill Yukhin's avatar
      test: fix LTO build for popen unit test · c44ed3c0
      Kirill Yukhin authored
      Fixa clash in struct names detected by LTO.
      c44ed3c0
    • Igor Munkin's avatar
      build: temporary fix for luajit-tap tests cmake · dfba0512
      Igor Munkin authored
      Fixes the regression from 335f80a0
      ('test: adjust luajit-tap testing machinery').
      dfba0512
    • Vladislav Shpilevoy's avatar
      iproto: rename IPROTO_ERROR and IPROTO_ERROR_STACK · 2b4263d3
      Vladislav Shpilevoy authored
      IPROTO_ERROR in fact is not an error. It is an error message.
      Secondly, this key is deprecated in favor of IPROTO_ERROR_STACK,
      which contains all attributes of the whole error stack. It uses
      MP_ERROR MessagePack extenstion for that.
      
      So IPROTO_ERROR is renamed to IPROTO_ERROR_24 (similar to how old
      call was renamed to IPROTO_CALL_16). IPROTO_ERROR_STACK becomes
      new IPROTO_ERROR.
      
      Follow up #4398
      2b4263d3
    • Vladislav Shpilevoy's avatar
      error: make iproto errors reuse mp_error module · 712af455
      Vladislav Shpilevoy authored
      After error objects marshaling was implemented in #4398, there
      were essentially 2 versions of the marshaling - when an error is
      sent inside response body, and when it is thrown and is encoded
      in iproto fields IPROTO_ERROR and IPROTO_ERROR_STACK. That is not
      really useful to have 2 implementation of the same feature. This
      commit drops the old iproto error encoding (its IPROTO_ERROR_STACK
      part), and makes it reuse the common error encoder.
      
      Note, the encoder skips MP_EXT header. This is because
      
      * The header is not needed - error is encoded as a value of
        IPROTO_ERROR_STACK key, so it is known this is an error. MP_EXT
        is needed only when type is unknown on decoding side in advance;
      
      * Old clients may not expect MP_EXT in iproto fields. That is the
        case of netbox connector, at least.
      
      Follow up #4398
      
      @TarantoolBot document
      Title: Stacked diagnostics binary protocol
      Stacked diagnostics is described in details in
      https://github.com/tarantool/doc/issues/1224. This commit
      changes nothing except binary protocol. The old protocol should
      not be documented anywhere.
      
      `IPROTO_ERROR_STACK` is still 0x52, but format of its value is
      different now. It looks exactly like `MP_ERROR` object, without
      `MP_EXT` header.
      
      ```
      IPROTO_ERROR_STACK: <MP_MAP> {
          MP_ERROR_STACK: <MP_ARRAY> [
              <MP_MAP> {
                  ... <all the other fields of MP_ERROR> ...
              },
              ...
          ]
      }
      ```
      
      It is easy to see, that key `IPROTO_ERROR_STACK` is called
      'stack', and `MP_ERROR_STACK` is also 'stack'. So it may be good
      to rename the former key in the documentation. For example, the
      old `IPROTO_ERROR` can be renamed to `IPROTO_ERROR_24` and
      `IPROTO_ERROR_STACK` can be renamed to just `IPROTO_ERROR`.
      712af455
    • Leonid Vasiliev's avatar
      error: add error MsgPack encoding · 345877df
      Leonid Vasiliev authored
      
      Co-authored-by: default avatarVladislav <Shpilevoy&lt;v.shpilevoy@tarantool.org>
      
      Closes #4398
      
      @TarantoolBot document
      Title: Error objects encoding in MessagePack
      
      Until now an error sent over IProto, or serialized into
      MessagePack was turned into a string consisting of the error
      message. As a result, all other error object attributes were lost,
      including type of the object. On client side seeing a string it
      was not possible to tell whether the string is a real string, or
      it is a serialized error.
      
      To deal with that the error objects encoding is reworked from the
      scratch. Now, when session setting `error_marshaling_enabled` is
      true, all fibers of that session will encode error objects as a
      new MP_EXT type - MP_ERROR (0x03).
      
      ```
          +--------+----------+========+
          | MP_EXT | MP_ERROR | MP_MAP |
          +--------+----------+========+
      
          MP_ERROR: <MP_MAP> {
              MP_ERROR_STACK: <MP_ARRAY> [
                  <MP_MAP> {
                      MP_ERROR_TYPE: <MP_STR>,
                      MP_ERROR_FILE: <MP_STR>,
                      MP_ERROR_LINE: <MP_UINT>,
                      MP_ERROR_MESSAGE: <MP_STR>,
                      MP_ERROR_ERRNO: <MP_UINT>,
                      MP_ERROR_CODE: <MP_UINT>,
                      MP_ERROR_FIELDS: <MP_MAP> {
                          <MP_STR>: ...,
                          <MP_STR>: ...,
                          ...
                      },
                      ...
                  },
                  ...
              ]
          }
      ```
      
      On the top level there is a single key: `MP_ERROR_STACK = 0x00`.
      More keys can be added in future, and a client should ignore all
      unknown keys to keep compatibility with new versions.
      
      Every error in the stack is a map with the following keys:
      * `MP_ERROR_TYPE = 0x00` - error type. This is what is visible in
        `<error_object>.base_type` field;
      * `MP_ERROR_FILE = 0x01` - file name from `<error_object>.trace`;
      * `MP_ERROR_LINE = 0x02` - line from `<error_object>.trace`;
      * `MP_ERROR_MESSAGE = 0x03` - error message from
        `<error_object>.message`;
      * `MP_ERROR_ERRNO = 0x04` - errno saved at the moment of the error
        creation. Visible in `<error_object>.errno`;
      * `MP_ERROR_CODE = 0x05` - error code. Visible in
        `<error_object>.code` and in C function `box_error_code()`.
      * `MP_ERROR_FIELDS = 0x06` - additional fields depending on error
        type. For example, AccessDenied error type stores here fields
        `access_type`, `object_type`, `object_name`. Connector's code
        should ignore unknown keys met here, and be ready, that for some
        existing errors new fields can be added, old can be dropped.
      345877df
    • Leonid Vasiliev's avatar
      error: add session setting for error type marshaling · c7a7f1cb
      Leonid Vasiliev authored
      
      Errors are encoded as a string when serialized to MessagePack to
      be sent over IProto or when just saved into a buffer via Lua
      modules msgpackffi and msgpack.
      
      That is not very useful on client-side, because most of the error
      metadata is lost: code, type, trace - everything except the
      message.
      
      Next commits are going to dedicate a new MP_EXT type to error
      objects so as everything could be encoded, and on client side it
      would be possible to restore types.
      
      But this is a breaking change in case some users use old
      connectors when work with newer Tarantool instances. So to smooth
      the upgrade there is a new session setting -
      'error_marshaling_enabled'.
      
      By default it is false. When it is true, all fibers of the given
      session will serialize error objects as MP_EXT.
      
      Co-authored-by: default avatarVladislav <Shpilevoy&lt;v.shpilevoy@tarantool.org>
      
      Needed for #4398
      c7a7f1cb
    • Leonid Vasiliev's avatar
      error: add custom error type · b728e7af
      Leonid Vasiliev authored
      
      Co-authored-by: default avatarVladislav <Shpilevoy&lt;v.shpilevoy@tarantool.org>
      
      Part of #4398
      
      @TarantoolBot document
      Title: Custom error types for Lua errors
      
      Errors can be created in 2 ways: `box.error.new()` and `box.error()`.
      
      Both used to take either `code, reason, <reason string args>` or
      `{code = code, reason = reason, ...}` arguments.
      
      Now in the first option instead of code a user can specify a
      string as its own error type. In the second option a user can
      specify both code and type. For example:
      
      ```Lua
      box.error('MyErrorType', 'Message')
      box.error({type = 'MyErrorType', code = 1024, reason = 'Message'})
      ```
      Or no-throw version:
      ```Lua
      box.error.new('MyErrorType', 'Message')
      box.error.new({type = 'MyErrorType', code = 1024, reason = 'Message'})
      ```
      When a custom type is specified, it is shown in `err.type`
      attribute. When it is not specified, `err.type` shows one of
      built-in errors such as 'ClientError', 'OurOfMemory', etc.
      
      Name length limit on the custom type is 63 bytes. All what is
      longer is truncated.
      
      Original error type can be checked using `err.base_type` member,
      although normally it should not be used. For user-defined types
      base type is 'CustomError'.
      
      For example:
      ```
      tarantool> e = box.error.new({type = 'MyErrorType', code = 1024, reason = 'Message'})
      ---
      ...
      
      tarantool> e:unpack()
      ---
      - code: 1024
        trace:
        - file: '[string "e = box.error.new({type = ''MyErrorType'', code..."]'
          line: 1
        type: MyErrorType
        custom_type: MyErrorType
        message: Message
        base_type: CustomError
      ...
      ```
      b728e7af
    • Alexander Turenko's avatar
      popen: add popen Lua module · 6d1c5ff5
      Alexander Turenko authored
      
      Fixes #4031
      
      Co-developed-by: default avatarCyrill Gorcunov <gorcunov@gmail.com>
      Co-developed-by: default avatarIgor Munkin <imun@tarantool.org>
      Acked-by: default avatarCyrill Gorcunov <gorcunov@gmail.com>
      Reviewed-by: default avatarIgor Munkin <imun@tarantool.org>
      
      @TarantoolBot document
      Title: popen module
      
      ```
      Overview
      ========
      
      Tarantool supports execution of external programs similarly to well
      known Python's `subprocess` or Ruby's `Open3`. Note though the `popen`
      module does not match one to one to the helpers these languages provide
      and provides only basic functions. The popen object creation is
      implemented via `vfork()` system call which means the caller thread is
      blocked until execution of a child process begins.
      
      Module functions
      ================
      
      The `popen` module provides two functions to create that named popen
      object: `popen.shell` which is the similar to libc `popen` syscall and
      `popen.new` to create popen object with more specific options.
      
      `popen.shell(command[, mode]) -> handle, err`
      ---------------------------------------------
      
      Execute a shell command.
      
      @param command  a command to run, mandatory
      @param mode     communication mode, optional
                      'w'    to use ph:write()
                      'r'    to use ph:read()
                      'R'    to use ph:read({stderr = true})
                      nil    inherit parent's std* file descriptors
      
      Several mode characters can be set together: 'rw', 'rRw', etc.
      
      This function is just shortcut for popen.new({command}, opts)
      with opts.{shell,setsid,group_signal} set to `true` and
      and opts.{stdin,stdout,stderr} set based on `mode` parameter.
      
      All std* streams are inherited from parent by default if it is
      not changed using mode: 'r' for stdout, 'R' for stderr, 'w' for
      stdin.
      
      Raise an error on incorrect parameters:
      
      - IllegalParams: incorrect type or value of a parameter.
      
      Return a popen handle on success.
      
      Return `nil, err` on a failure.
      @see popen.new() for possible reasons.
      
      Example:
      
       | local popen = require('popen')
       |
       | -- Run the program and save its handle.
       | local ph = popen.shell('date', 'r')
       |
       | -- Read program's output, strip trailing newline.
       | local date = ph:read():rstrip()
       |
       | -- Free resources. The process is killed (but 'date'
       | -- exits itself anyway).
       | ph:close()
       |
       | print(date)
      
      Execute 'sh -c date' command, read the output and close the
      popen object.
      
      Unix defines a text file as a sequence of lines, each ends
      with the newline symbol. The same convention is usually
      applied for a text output of a command (so when it is
      redirected to a file, the file will be correct).
      
      However internally an application usually operates on
      strings, which are NOT newline terminated (e.g. literals
      for error messages). The newline is usually added right
      before a string is written to the outside world (stdout,
      console or log). :rstrip() in the example above is shown
      for this sake.
      
      `popen.new(argv[, opts]) -> handle, err`
      ----------------------------------------
      
      Execute a child program in a new process.
      
      @param argv  an array of a program to run with
                   command line options, mandatory;
                   absolute path to the program is required
                   when @a opts.shell is false (default)
      
      @param opts  table of options
      
      @param opts.stdin   action on STDIN_FILENO
      @param opts.stdout  action on STDOUT_FILENO
      @param opts.stderr  action on STDERR_FILENO
      
      File descriptor actions:
      
          popen.opts.INHERIT  (== 'inherit') [default]
                              inherit the fd from the parent
          popen.opts.DEVNULL  (== 'devnull')
                              open /dev/null on the fd
          popen.opts.CLOSE    (== 'close')
                              close the fd
          popen.opts.PIPE     (== 'pipe')
                              feed data from/to the fd to parent
                              using a pipe
      
      @param opts.env  a table of environment variables to
                       be used inside a process; key is a
                       variable name, value is a variable
                       value.
                       - when is not set then the current
                         environment is inherited;
                       - if set to an empty table then the
                         environment will be dropped
                       - if set then the environment will be
                         replaced
      
      @param opts.shell            (boolean, default: false)
             true                  run a child process via
                                   'sh -c "${opts.argv}"'
             false                 call the executable directly
      
      @param opts.setsid           (boolean, default: false)
             true                  run the program in a new
                                   session
             false                 run the program in the
                                   tarantool instance's
                                   session and process group
      
      @param opts.close_fds        (boolean, default: true)
             true                  close all inherited fds from a
                                   parent
             false                 don't do that
      
      @param opts.restore_signals  (boolean, default: true)
             true                  reset all signal actions
                                   modified in parent's process
             false                 inherit changed actions
      
      @param opts.group_signal     (boolean, default: false)
             true                  send signal to a child process
                                   group (only when opts.setsid is
                                   enabled)
             false                 send signal to a child process
                                   only
      
      @param opts.keep_child       (boolean, default: false)
             true                  don't send SIGKILL to a child
                                   process at freeing (by :close()
                                   or Lua GC)
             false                 send SIGKILL to a child process
                                   (or a process group if
                                   opts.group_signal is enabled) at
                                   :close() or collecting of the
                                   handle by Lua GC
      
      The returned handle provides :close() method to explicitly
      release all occupied resources (including the child process
      itself if @a opts.keep_child is not set). However if the
      method is not called for a handle during its lifetime, the
      same freeing actions will be triggered by Lua GC.
      
      It is recommended to use opts.setsid + opts.group_signal
      if a child process may spawn its own childs and they all
      should be killed together.
      
      Note: A signal will not be sent if the child process is
      already dead: otherwise we might kill another process that
      occupies the same PID later. This means that if the child
      process dies before its own childs, the function will not
      send a signal to the process group even when opts.setsid and
      opts.group_signal are set.
      
      Use os.environ() to pass copy of current environment with
      several replacements (see example 2 below).
      
      Raise an error on incorrect parameters:
      
      - IllegalParams: incorrect type or value of a parameter.
      - IllegalParams: group signal is set, while setsid is not.
      
      Return a popen handle on success.
      
      Return `nil, err` on a failure. Possible reasons:
      
      - SystemError: dup(), fcntl(), pipe(), vfork() or close()
                     fails in the parent process.
      - SystemError: (temporary restriction) the parent process
                     has closed stdin, stdout or stderr.
      - OutOfMemory: unable to allocate the handle or a temporary
                     buffer.
      
      Example 1:
      
       | local popen = require('popen')
       |
       | local ph = popen.new({'/bin/date'}, {
       |     stdout = popen.opts.PIPE,
       | })
       | local date = ph:read():rstrip()
       | ph:close()
       | print(date) -- Thu 16 Apr 2020 01:40:56 AM MSK
      
      Execute 'date' command, read the result and close the
      popen object.
      
      Example 2:
      
       | local popen = require('popen')
       |
       | local env = os.environ()
       | env['FOO'] = 'bar'
       |
       | local ph = popen.new({'echo "${FOO}"'}, {
       |     stdout = popen.opts.PIPE,
       |     shell = true,
       |     env = env,
       | })
       | local res = ph:read():rstrip()
       | ph:close()
       | print(res) -- bar
      
      It is quite similar to the previous one, but sets the
      environment variable and uses shell builtin 'echo' to
      show it.
      
      Example 3:
      
       | local popen = require('popen')
       |
       | local ph = popen.new({'echo hello >&2'}, { -- !!
       |     stderr = popen.opts.PIPE,              -- !!
       |     shell = true,
       | })
       | local res = ph:read({stderr = true}):rstrip()
       | ph:close()
       | print(res) -- hello
      
      This example demonstrates how to capture child's stderr.
      
      Example 4:
      
       | local function call_jq(input, filter)
       |     -- Start jq process, connect to stdin, stdout and stderr.
       |     local jq_argv = {'/usr/bin/jq', '-M', '--unbuffered', filter}
       |     local ph, err = popen.new(jq_argv, {
       |         stdin = popen.opts.PIPE,
       |         stdout = popen.opts.PIPE,
       |         stderr = popen.opts.PIPE,
       |     })
       |     if ph == nil then return nil, err end
       |
       |     -- Write input data to child's stdin and send EOF.
       |     local ok, err = ph:write(input)
       |     if not ok then return nil, err end
       |     ph:shutdown({stdin = true})
       |
       |     -- Read everything until EOF.
       |     local chunks = {}
       |     while true do
       |         local chunk, err = ph:read()
       |         if chunk == nil then
       |             ph:close()
       |             return nil, err
       |         end
       |         if chunk == '' then break end -- EOF
       |         table.insert(chunks, chunk)
       |     end
       |
       |     -- Read diagnostics from stderr if any.
       |     local err = ph:read({stderr = true})
       |     if err ~= '' then
       |         ph:close()
       |         return nil, err
       |     end
       |
       |     -- Glue all chunks, strip trailing newline.
       |     return table.concat(chunks):rstrip()
       | end
      
      Demonstrates how to run a stream program (like `grep`, `sed`
      and so), write to its stdin and read from its stdout.
      
      The example assumes that input data are small enough to fit
      a pipe buffer (typically 64 KiB, but depends on a platform
      and its configuration). It will stuck in :write() for large
      data. How to handle this case: call :read() in a loop in
      another fiber (start it before a first :write()).
      
      If a process writes large text to stderr, it may fill out
      stderr pipe buffer and stuck in write(2, ...). So we need
      to read stderr in a separate fiber to handle this case.
      
      Handle methods
      ==============
      
      `popen_handle:read([opts]) -> str, err`
      ---------------------------------------
      
      Read data from a child peer.
      
      @param handle        handle of a child process
      @param opts          an options table
      @param opts.stdout   whether to read from stdout, boolean
                           (default: true)
      @param opts.stderr   whether to read from stderr, boolean
                           (default: false)
      @param opts.timeout  time quota in seconds
                           (default: 100 years)
      
      Read data from stdout or stderr streams with @a timeout.
      By default it reads from stdout. Set @a opts.stderr to
      `true` to read from stderr.
      
      It is not possible to read from stdout and stderr both in
      one call. Set either @a opts.stdout or @a opts.stderr.
      
      Raise an error on incorrect parameters or when the fiber is
      cancelled:
      
      - IllegalParams:    incorrect type or value of a parameter.
      - IllegalParams:    called on a closed handle.
      - IllegalParams:    opts.stdout and opts.stderr are set both
      - IllegalParams:    a requested IO operation is not supported
                          by the handle (stdout / stderr is not
                          piped).
      - IllegalParams:    attempt to operate on a closed file
                          descriptor.
      - FiberIsCancelled: cancelled by an outside code.
      
      Return a string on success, an empty string at EOF.
      
      Return `nil, err` on a failure. Possible reasons:
      
      - SocketError: an IO error occurs at read().
      - TimedOut:    @a timeout quota is exceeded.
      - OutOfMemory: no memory space for a buffer to read into.
      - LuajitError: ("not enough memory"): no memory space for
                     the Lua string.
      
      `popen_handle:write(str[, opts]) -> str, err`
      ---------------------------------------------
      
      Write data to a child peer.
      
      @param handle        a handle of a child process
      @param str           a string to write
      @param opts          table of options
      @param opts.timeout  time quota in seconds
                           (default: 100 years)
      
      Write string @a str to stdin stream of a child process.
      
      The function may yield forever if a child process does
      not read data from stdin and a pipe buffer becomes full.
      Size of this buffer depends on a platform. Use
      @a opts.timeout when unsure.
      
      When @a opts.timeout is not set, the function blocks
      (yields the fiber) until all data is written or an error
      happened.
      
      Raise an error on incorrect parameters or when the fiber is
      cancelled:
      
      - IllegalParams:    incorrect type or value of a parameter.
      - IllegalParams:    called on a closed handle.
      - IllegalParams:    string length is greater then SSIZE_MAX.
      - IllegalParams:    a requested IO operation is not supported
                          by the handle (stdin is not piped).
      - IllegalParams:    attempt to operate on a closed file
                          descriptor.
      - FiberIsCancelled: cancelled by an outside code.
      
      Return `true` on success.
      
      Return `nil, err` on a failure. Possible reasons:
      
      - SocketError: an IO error occurs at write().
      - TimedOut:    @a timeout quota is exceeded.
      
      `popen_handle:shutdown(opts) -> true`
      ------------------------------------------
      
      Close parent's ends of std* fds.
      
      @param handle        handle of a child process
      @param opts          an options table
      @param opts.stdin    close parent's end of stdin, boolean
      @param opts.stdout   close parent's end of stdout, boolean
      @param opts.stderr   close parent's end of stderr, boolean
      
      The main reason to use this function is to send EOF to
      child's stdin. However parent's end of stdout / stderr
      may be closed too.
      
      The function does not fail on already closed fds (idempotence).
      However it fails on attempt to close the end of a pipe that was
      never exist. In other words, only those std* options that
      were set to popen.opts.PIPE at a handle creation may be used
      here (for popen.shell: 'r' corresponds to stdout, 'R' to stderr
      and 'w' to stdin).
      
      The function does not close any fds on a failure: either all
      requested fds are closed or neither of them.
      
      Example:
      
       | local popen = require('popen')
       |
       | local ph = popen.shell('sed s/foo/bar/', 'rw')
       | ph:write('lorem foo ipsum')
       | ph:shutdown({stdin = true})
       | local res = ph:read()
       | ph:close()
       | print(res) -- lorem bar ipsum
      
      Raise an error on incorrect parameters:
      
      - IllegalParams:  an incorrect handle parameter.
      - IllegalParams:  called on a closed handle.
      - IllegalParams:  neither stdin, stdout nor stderr is choosen.
      - IllegalParams:  a requested IO operation is not supported
                        by the handle (one of std* is not piped).
      
      Return `true` on success.
      
      `popen_handle:terminate() -> ok, err`
      -------------------------------------
      
      Send SIGTERM signal to a child process.
      
      @param handle  a handle carries child process to terminate
      
      The function only sends SIGTERM signal and does NOT
      free any resources (popen handle memory and file
      descriptors).
      
      @see popen_handle:signal() for errors and return values.
      
      `popen_handle:kill() -> ok, err`
      --------------------------------
      
      Send SIGKILL signal to a child process.
      
      @param handle  a handle carries child process to kill
      
      The function only sends SIGKILL signal and does NOT
      free any resources (popen handle memory and file
      descriptors).
      
      @see popen_handle:signal() for errors and return values.
      
      `popen_handle:signal(signo) -> ok, err`
      ---------------------------------------
      
      Send signal to a child process.
      
      @param handle  a handle carries child process to be signaled
      @param signo   signal number to send
      
      When opts.setsid and opts.group_signal are set on the handle
      the signal is sent to the process group rather than to the
      process. @see popen.new() for details about group
      signaling.
      
      Note: The module offers popen.signal.SIG* constants, because
      some signals have different numbers on different platforms.
      
      Raise an error on incorrect parameters:
      
      - IllegalParams:    an incorrect handle parameter.
      - IllegalParams:    called on a closed handle.
      
      Return `true` if signal is sent.
      
      Return `nil, err` on a failure. Possible reasons:
      
      - SystemError: a process does not exists anymore
      
                     Aside of a non-exist process it is also
                     returned for a zombie process or when all
                     processes in a group are zombies (but
                     see note re Mac OS below).
      
      - SystemError: invalid signal number
      
      - SystemError: no permission to send a signal to
                     a process or a process group
      
                     It is returned on Mac OS when a signal is
                     sent to a process group, where a group leader
                     is zombie (or when all processes in it
                     are zombies, don't sure).
      
                     Whether it may appear due to other
                     reasons is unclear.
      
      `popen_handle:info() -> res`
      ----------------------------
      
      Return information about popen handle.
      
      @param handle  a handle of a child process
      
      Raise an error on incorrect parameters:
      
      - IllegalParams: an incorrect handle parameter.
      - IllegalParams: called on a closed handle.
      
      Return information about the handle in the following
      format:
      
          {
              pid = <number> or <nil>,
              command = <string>,
              opts = <table>,
              status = <table>,
              stdin = one-of(
                  popen.stream.OPEN   (== 'open'),
                  popen.stream.CLOSED (== 'closed'),
                  nil,
              ),
              stdout = one-of(
                  popen.stream.OPEN   (== 'open'),
                  popen.stream.CLOSED (== 'closed'),
                  nil,
              ),
              stderr = one-of(
                  popen.stream.OPEN   (== 'open'),
                  popen.stream.CLOSED (== 'closed'),
                  nil,
              ),
          }
      
      `pid` is a process id of the process when it is alive,
      otherwise `pid` is nil.
      
      `command` is a concatenation of space separated arguments
      that were passed to execve(). Multiword arguments are quoted.
      Quotes inside arguments are not escaped.
      
      `opts` is a table of handle options in the format of
      popen.new() `opts` parameter. `opts.env` is not shown here,
      because the environment variables map is not stored in a
      handle.
      
      `status` is a table that represents a process status in the
      following format:
      
          {
              state = one-of(
                  popen.state.ALIVE    (== 'alive'),
                  popen.state.EXITED   (== 'exited'),
                  popen.state.SIGNALED (== 'signaled'),
              )
      
              -- Present when `state` is 'exited'.
              exit_code = <number>,
      
              -- Present when `state` is 'signaled'.
              signo = <number>,
              signame = <string>,
          }
      
      `stdin`, `stdout`, `stderr` reflect status of parent's end
      of a piped stream. When a stream is not piped the field is
      not present (`nil`). When it is piped, the status may be
      one of the following:
      
      - popen.stream.OPEN    (== 'open')
      - popen.stream.CLOSED  (== 'closed')
      
      The status may be changed from 'open' to 'closed'
      by :shutdown({std... = true}) call.
      
      Example 1 (tarantool console):
      
       | tarantool> require('popen').new({'/usr/bin/touch', '/tmp/foo'})
       | ---
       | - command: /usr/bin/touch /tmp/foo
       |   status:
       |     state: alive
       |   opts:
       |     stdout: inherit
       |     stdin: inherit
       |     group_signal: false
       |     keep_child: false
       |     close_fds: true
       |     restore_signals: true
       |     shell: false
       |     setsid: false
       |     stderr: inherit
       |   pid: 9499
       | ...
      
      Example 2 (tarantool console):
      
       | tarantool> require('popen').shell('grep foo', 'wrR')
       | ---
       | - stdout: open
       |   command: sh -c 'grep foo'
       |   stderr: open
       |   status:
       |     state: alive
       |   stdin: open
       |   opts:
       |     stdout: pipe
       |     stdin: pipe
       |     group_signal: true
       |     keep_child: false
       |     close_fds: true
       |     restore_signals: true
       |     shell: true
       |     setsid: true
       |     stderr: pipe
       |   pid: 10497
       | ...
      
      `popen_handle:wait() -> res`
      ----------------------------
      
      Wait until a child process get exited or signaled.
      
      @param handle  a handle of process to wait
      
      Raise an error on incorrect parameters or when the fiber is
      cancelled:
      
      - IllegalParams:    an incorrect handle parameter.
      - IllegalParams:    called on a closed handle.
      - FiberIsCancelled: cancelled by an outside code.
      
      Return a process status table (the same as ph.status and
      ph.info().status). @see popen_handle:info() for the format
      of the table.
      
      `popen_handle:close() -> ok, err`
      ---------------------------------
      
      Close a popen handle.
      
      @param handle  a handle to close
      
      Basically it kills a process using SIGKILL and releases all
      resources assosiated with the popen handle.
      
      Details about signaling:
      
      - The signal is sent only when opts.keep_child is not set.
      - The signal is sent only when a process is alive according
        to the information available on current even loop iteration.
        (There is a gap here: a zombie may be signaled; it is
        harmless.)
      - The signal is sent to a process or a grocess group depending
        of opts.group_signal. (@see lbox_popen_new() for details of
        group signaling).
      
      Resources are released disregarding of whether a signal
      sending succeeds: fds are closed, memory is released,
      the handle is marked as closed.
      
      No operation is possible on a closed handle except
      :close(), which always successful on closed handle
      (idempotence).
      
      Raise an error on incorrect parameters:
      
      - IllegalParams: an incorrect handle parameter.
      
      The function may return `true` or `nil, err`, but it always
      frees the handle resources. So any return value usually
      means success for a caller. The return values are purely
      informational: it is for logging or same kind of reporting.
      
      Possible diagnostics (don't consider them as errors):
      
      - SystemError: no permission to send a signal to
                     a process or a process group
      
                     This diagnostics may appear due to
                     Mac OS behaviour on zombies when
                     opts.group_signal is set,
                     @see lbox_popen_signal().
      
                     Whether it may appear due to other
                     reasons is unclear.
      
      Always return `true` when a process is known as dead (say,
      after ph:wait()): no signal will be send, so no 'failure'
      may appear.
      
      Handle fields
      =============
      
      - popen_handle.pid
      - popen_handle.command
      - popen_handle.opts
      - popen_handle.status
      - popen_handle.stdin
      - popen_handle.stdout
      - popen_handle.stderr
      
      See popen_handle:info() for description of those fields.
      
      Module constants
      ================
      
      - popen.opts
        - INHERIT (== 'inherit')
        - DEVNULL (== 'devnull')
        - CLOSE   (== 'close')
        - PIPE    (== 'pipe')
      
      - popen.signal
        - SIGTERM (== 9)
        - SIGKILL (== 15)
        - ...
      
      - popen.state
        - ALIVE    (== 'alive')
        - EXITED   (== 'exited')
        - SIGNALED (== 'signaled')
      
      - popen.stream
        - OPEN    (== 'open')
        - CLOSED  (== 'closed')
      ```
      
      (cherry picked from commit 34c2789b48eadd36da742f7f761a998889e70544)
      6d1c5ff5
    • Igor Munkin's avatar
      test: adjust luajit-tap testing machinery · 335f80a0
      Igor Munkin authored
      
      This changeset makes possible to run luajit-tap tests requiring
      libraries implemented in C:
      * symlink to luajit test is created on configuration phase instead of
        build one.
      * introduced a CMake function for building shared libraries required for
        luajit tests.
      
      Furthermore this commit enables CMake build for the following luajit-tap
      tests:
      * gh-4427-ffi-sandwich
      * lj-flush-on-trace
      
      Reviewed-by: default avatarVladislav Shpilevoy <v.shpilevoy@tarantool.org>
      Reviewed-by: default avatarSergey Ostanevich <sergos@tarantool.org>
      Signed-off-by: default avatarIgor Munkin <imun@tarantool.org>
      335f80a0
  2. Apr 17, 2020
    • Vladislav Shpilevoy's avatar
      feedback: add cmake option to disable the daemon · 7b443650
      Vladislav Shpilevoy authored
      There is a complaint that the feedback daemon is a 'spying' tool
      and because of that can't be used on Gentoo. Its default disabled
      option also is not acceptable, the daemon should be eliminated
      completely.
      
      The patch introduces cmake option ENABLE_FEEDBACK_DAEMON. It is
      ON by default. When set to OFF, all feedback daemon's code is not
      included into the binary, its configuration options disappear.
      
      Closes #3308
      7b443650
  3. Apr 16, 2020
    • Mergen Imeev's avatar
      sql: do not change order of inserted values · 2cc7e608
      Mergen Imeev authored
      Before this patch, if an ephemeral space was used during INSERT or
      REPLACE, the inserted values were sorted by the first column,
      since this was the first part of the index. This can lead to an
      error when using the AUTOINCREMENT feature, since changing the
      order of the inserted value can change the value inserted instead
      of NULL. To avoid this, the patch makes the rowid of the inserted
      row in the ephemeral space the only part of the ephemeral space
      index.
      
      Closes #4256
      2cc7e608
  4. Apr 15, 2020
    • Mergen Imeev's avatar
      sql: add '\0' to the BLOB when it is cast to INTEGER · a39e6a01
      Mergen Imeev authored
      Prior to this patch, due to the absence of the '\0' character at
      the end of the BLOB, it was possible to get an error or incorrect
      result when using CAST() from BLOB to INTEGER or UNSIGNED. This
      has now been fixed, but the maximum length of a BLOB that could be
      cast to INTEGER or UNSIGNED was limited to 12287 bytes.
      
      Examples of wrong CAST() from BLOB to INTEGER:
      
      CREATE TABLE t (i INT PRIMARY KEY, a VARBINARY, b INT, c INT);
      INSERT INTO t VALUES (1, X'33', 0x33, 0x00), (2, X'34', 0x41, 0);
      
      Example of wrong result:
      
      SELECT CAST(a AS INTEGER) FROM t WHERE i = 1;
      
      Result: 33
      
      Example of error during CAST():
      
      SELECT CAST(a AS INTEGER) FROM t WHERE i = 2;
      
      Result: 'Type mismatch: can not convert varbinary to integer'
      
      Closes #4766
      a39e6a01
    • Mergen Imeev's avatar
      sql: fix implicit cast from STRING to INTEGER · 6e6de43c
      Mergen Imeev authored
      Prior to this patch, STRING, which contains the DOUBLE value,
      could be implicitly cast to INTEGER. This was done by converting
      STRING to DOUBLE and then converting this DOUBLE value to INTEGER.
      This may affect the accuracy of CAST(), so it was forbidden. It
      is worth noting that these changes will not affect the comparison,
      since the implicit cast in this case has different mechanics.
      
      Example:
      box.execute("CREATE TABLE t(i INT PRIMARY KEY);")
      
      Before patch:
      box.execute("INSERT INTO t VALUES ('111.1');")
      box.execute("SELECT * FROM t;")
      Result: 111
      
      After patch:
      box.execute("INSERT INTO t VALUES ('1.1');")
      Result: 'Type mismatch: can not convert 1.1 to integer'
      
      box.execute("INSERT INTO t VALUES ('1.0');")
      Result: 'Type mismatch: can not convert 1.0 to integer'
      
      box.execute("INSERT INTO t VALUES ('1.');")
      Result: 'Type mismatch: can not convert 1. to integer'
      
      @TarantoolBot document
      Title: disallow cast from STRING contains DOUBLE to INTEGER
      
      After the last two patches, explicit and implicit casting from the
      string containing DOUBLE to INTEGER directly will be prohibited.
      The user must use the explicit cast to DOUBLE before the explicit
      or implicit cast to INTEGER. The reason for this is that before
      these patches, such STRINGs were implicitly cast to DOUBLE, and
      then this DOUBLE was implicitly or explicitly cast to INTEGER.
      Because of this, the result of such a cast may differ from what
      the user expects, and the user may not know why.
      
      It is worth noting that these changes will not affect the
      comparison, since the implicit cast in this case has different
      mechanics.
      
      Example for implicit cast:
      
      box.execute("CREATE TABLE t(i INT PRIMARY KEY);")
      -- Does not work anymore:
      box.execute("INSERT INTO t VALUES ('1.1');")
      -- Right way:
      box.execute("INSERT INTO t VALUES (CAST('1.1' AS DOUBLE));")
      
      Example for explicit cast:
      
      -- Does not work anymore:
      box.execute("SELECT CAST('1.1' AS INTEGER);")
      -- Right way:
      box.execute("SELECT CAST(CAST('1.1' AS DOUBLE) AS INTEGER);")
      6e6de43c
    • Mergen Imeev's avatar
      sql: fix CAST() from STRING to INTEGER · 11352a32
      Mergen Imeev authored
      Prior to this patch, STRING, which contains the DOUBLE value,
      could be cast to INTEGER. This was done by converting STRING to
      DOUBLE and then converting this DOUBLE value to INTEGER. This may
      affect the accuracy of CAST(), so it was forbidden.
      
      Before patch:
      box.execute("SELECT CAST('111.1' as INTEGER);")
      Result: 111
      
      After patch:
      box.execute("SELECT CAST('1.1' as INTEGER);")
      Result: 'Type mismatch: can not convert 1.1 to integer'
      
      box.execute("SELECT CAST('1.0' as INTEGER);")
      Result: 'Type mismatch: can not convert 1.0 to integer'
      
      box.execute("SELECT CAST('1.' as INTEGER);")
      Result: 'Type mismatch: can not convert 1. to integer'
      11352a32
    • Alexander V. Tikhonov's avatar
      Divide box/ddl.test.lua test · 4a8d1ebd
      Alexander V. Tikhonov authored
      Divided into tests:
      - box/ddl_alter.test.lua
      - box/ddl_collation.test.lua
      - box/ddl_collation_types.test.lua
      - box/ddl_collation_wrong_id.test.lua
      - box/ddl_no_collation.test.lua
      - box/ddl_parallel.test.lua
      - box/ddl_tuple.test.lua
      - box/gh-2336-ddl_call_twice.test.lua
      - box/gh-2783-ddl_lock.test.lua
      - box/gh-2839-ddl_custom_fields.test.lua
      - box/gh-2937-ddl_collation_field_def.test.lua
      - box/gh-3290-ddl_collation_deleted.test.lua
      - box/gh-928-ddl_truncate.test.lua
      4a8d1ebd
    • Sergey Bronnikov's avatar
      Fix flaky test engine/ddl · 5f96ee59
      Sergey Bronnikov authored
      Test was a flaky from the beginning 39d0e427
      Time of building indexes varies from time to time and the problem was due to
      abcense of synchronization in index building and checking numbers of these
      indexes.
      
      Fixes #4353
      5f96ee59
    • Serge Petrenko's avatar
      box: start counting local space requests separately · 7eb4650e
      Serge Petrenko authored
      Sign local space requests with a zero instance id. This allows to split
      local changes aside from the changes, which should be visible to the whole
      cluster, and stop sending NOPs to replicas to follow local vclock.
      
      Moreover, it fixes the following bug with local spaces and replication.
      In a situation when there are a master and a replica set up, replica may
      still write to local spaces even if it's read-only. Local space
      operations used to promote instance's lsn before this patch. Eventually,
      master would have vclock {1:x} and replica'd have vclock {1:x, 2:y},
      where y > 0, due to local space requests performed by the replica.
      If a snapshot happens on replica side, replica will delete it's .xlog
      files prior to the snapshot, since no one replicates from it and thus it
      doesn't have any registered GC consumers.
      From this point, every attempt to configure replication from replica to
      master will fail, since master will try to fetch records which account
      for the difference in master's and replica's vclocks: {1:x} vs {1:x,2:y},
      even though master will never need the rows in range {2:1} - {2:y},
      since they'll be turned to NOPs during replication.
      
      Starting from this patch, in the situation described above, replica's
      clock will be {0:y, 1:x}, and, since local space requests are now not
      replicated at all, master will be able to follow replica, turning the
      configuration to master-master.
      
      Closes #4114
      7eb4650e
    • Serge Petrenko's avatar
      replication: omit 0-th vclock component in replication responses · 93ae77fc
      Serge Petrenko authored
      If an anonymous replica is promoted to a normal one and becomes
      replication master later, its vclock contains a non-empty zero
      component, tracking local changes on this replica from the time when it
      had been anonymous. No need to pollute joining instance's vclock with
      our non-empty 0 component.
      When an anonymous replica reports its status to a remote instance it
      should also hide its 0-th vclock component.
      
      This is needed for backward compatibility with old instances, which
      don't ignore 0th vclock component coming from a remote instance by
      default.
      
      In order to do so, introduce a new function - vclock_size_ignore0(),
      which doesn't count 0th clock component, and patch xrow_encode_vclock()
      to skip 0th clock component if it's present.
      
      Also make sure that new instances ignore 0th vclock component coming
      from an unpatched remote instance.
      
      Follow-up #3186
      Prerequisite #4114
      93ae77fc
  5. Apr 14, 2020
    • Vladislav Shpilevoy's avatar
      box: export box_session_push to the public C API · 5d795527
      Vladislav Shpilevoy authored
      API is different from box.session.push() - sync argument was
      removed. It will disappear from Lua API as well, because it just
      is not needed here. Session is omitted as well. Indeed, a user
      can't push to a foreign session, and the current session can be
      obtained inside box_session_push(). And anyway session is not in
      the public C API.
      
      Internally dump into iproto is done using obuf_dup(), just like
      tuple_to_obuf() does. obuf_alloc() would be a bad call here,
      because it wouldn't be able to split the pushed data into several
      obuf chunks, and would cause obuf fragmentation.
      
      Dump into plain text behaves just like a Lua push - it produces a
      YAML formatted text or Lua text depending on output format. But to
      turn MessagePack into YAML or Lua text an intermediate Lua
      representation is used, because there are no a MessagePack -> YAML
      and MessagePack -> Lua text translators yet.
      
      Closes #4734
      
      @TarantoolBot document
      Title: box_session_push() C API
      
      There is a new function in the public C API:
      ```C
          int
          box_session_push(const char *data, const char *data_end);
      ```
      
      It takes raw MessagePack, and behaves just like Lua
      `box.session.push()`.
      5d795527
  6. Apr 13, 2020
    • Chris Sosnin's avatar
      box: fix formatting in session.push · 568b16ce
      Chris Sosnin authored
      box.session.push() encodes data as a YAML document independent on
      the current console output format. This patch adds handling for Lua
      as well.
      
      Closes #4686
      568b16ce
    • Chris Sosnin's avatar
      sql: provide a user friendly frontend for accessing session settings · 6706f659
      Chris Sosnin authored
      Currently if a user wants to change session setting with SQL, one has
      to execute UPDATE query like:
      [[UPDATE "_session_settings" SET "value" = true WHERE "name" = 'name']]
      However, direct access to system spaces isn't considered to be a good practice.
      To avoid that and a bit simplify user's life, we introduce SQL shortcut command
      SET SESSION.
      
      Closes #4711
      
      @TarantoolBot document
      Title: API for accessing _session_settings space.
      There are two ways of updating values of session settings:
      via Lua and SQL.
      
      Lua:
      box.session.settings is a table, which is always accessible
      to user. The syntax is the following:
      `box.session.settings.<setting_name> = <new_value>`.
      
      Example of usage:
      ```
      tarantool> box.session.settings.sql_default_engine
      ---
      - memtx
      ...
      
      tarantool> box.session.settings.sql_default_engine = 'vinyl'
      ---
      ...
      
      ```
      
      The table itself represents the (unordered) result of select
      from _session_settings space.
      
      SQL:
      Instead of typing long UPDATE query one can use the SET SESSION command:
      `box.execute([[SET SESSION "<setting_name>" = <new_value>]])`.
      Note, that this query is case sensitive so the name must be quoted.
      Also, SET SESSION doesn't provide any implicit casts, so <new_value> must
      be of the type corresponding to the setting being updated.
      
      Example:
      ```
      tarantool> box.execute([[set session "sql_default_engine" = 'memtx']])
      ---
      - row_count: 1
      ...
      
      tarantool> box.execute([[set session "sql_defer_foreign_keys" = true]])
      ---
      - row_count: 1
      ...
      
      ```
      6706f659
    • Chris Sosnin's avatar
      box: provide a user friendly frontend for accessing session settings · 5ea94c51
      Chris Sosnin authored
      - space_object:update() is hard to use for configuring session settings,
      so we provide box.session.settings table, which can be used in a much more
      native way.
      
      - Prior to this patch sql settings were not accessible before box.cfg()
      call, even though these flags can be set right after session creation.
      
      Part of #4711
      5ea94c51
    • Serge Petrenko's avatar
      box: introduce indices by UUID · b238def8
      Serge Petrenko authored
      It is now possible to create an index over UUID values, returned by
      `uuid.new()`.
      
      Closes #4268
      Closes #2916
      
      @TarantoolBot document
      Title: Document uuid field type.
      
      There's a new field type -- UUID, it accepts values returned by
      `uuid.new()`.
      
      The index may be either unique or non-unique, nullable or non-nullable,
      and may be a primary key.
      
      The values in an index are ordered lexicographically by their string
      representation.
      
      To create an index over a uuid field for space `test`, say:
      ```
      box.space.test:create_index("pk", {parts={1, 'uuid'}})
      ```
      Now you may insert uuids into the space:
      ```
      tarantool> box.space.test:insert{uuid.new()}
      ---
      - [e631fdcc-0e8a-4d2f-83fd-b0ce6762b13f]
      ...
      
      tarantool> box.space.test:insert{uuid.fromstr('64d22e4d-ac92-4a23-899a-e59f34af5479')}
      ---
      - [64d22e4d-ac92-4a23-899a-e59f34af5479]
      ...
      
      tarantool> box.space.test:select{}
      ---
      - - [64d22e4d-ac92-4a23-899a-e59f34af5479]
        - [e631fdcc-0e8a-4d2f-83fd-b0ce6762b13f]
      ...
      
      ```
      b238def8
    • Serge Petrenko's avatar
      box: add MsgPack encoding/decoding for UUID · d68fc292
      Serge Petrenko authored
      A special format for encoding UUIDs to MsgPack is introduced.
      It is supported by both lua and C encoders/decoders, and it is now
      possible to insert UUIDs into spaces, but only into unindexed fields
      without format for now.
      
      Prerequisite #4268
      
      @TarantoolBot document
      Title: Internals: msgpack format for UUID
      
      UUID values share the MessagePack type with decimals:
      both use MP_EXT. A new subtype is introduced for UUIDs,
      MP_UUID = `0x02`
      UUID is encoded as follows:
      ```
          +--------+---------+-----------+
          | MP_EXT | MP_UUID | UuidValue |
          +--------+---------+-----------+
      ```
      Since UUID is 16 bytes in size, the header, MP_EXT, is always the same:
      `0xd8`. MP_UUID = `0x02` follows. The header is followed by the 16
      bytes of the UuidValue.
      
      UuidValue consists of 11 fields, which are encoded as big endian
      unsigned integers in the following order: `time_low` (4 bytes), `time_mid`
      (2 bytes), `time_hi_and_version` (2 bytes), `clock_seq_hi_and_reserved` (1
      byte), `clock_seq_low` (1 byte), `node[0], ..., node[5]` (1 byte each).
      
      The total size of such a representation is 18 bytes, whereas storing
      uuids as strings requires from 34 (when '-'s are ommitted) to 38 bytes
      per UUID, giving a 2x space usage improvement.
      d68fc292
  7. Apr 10, 2020
  8. Apr 08, 2020
  9. Apr 07, 2020
    • Nikita Pettik's avatar
      sql: reset values to be bound after execution · df03a7e8
      Nikita Pettik authored
      Before this patch prepared statements didn't reset bound values after
      its execution. As a result, if during next execution cycle not all
      parameters were provided, cached values would appear. For instance:
      
      prep = box.prepare('select :a, :b, :c')
      prep:execute({{[':a'] = 1}, {[':b'] = 2}, {[':c'] = 3}}
      -- [1, 2, 3]
      prep:execute({{[':a'] = 1}, {[':b'] = 2}})
      -- [1, 2, 3]
      
      However, expected result for the last query should be [1, 2, NULL].
      Let's fix it and always reset all binding values before next execution.
      
      Closes #4825
      df03a7e8
    • Nikita Pettik's avatar
      iproto: support error stacked diagnostic area · 4c465312
      Nikita Pettik authored
      This patch introduces support of stacked errors in IProto protocol and
      in net.box module.
      
      Closes #1148
      
      @TarantoolBot document
      Title: Stacked error diagnostic area
      
      Starting from now errors can be organized into lists. To achieve this
      Lua table representing error object is extended with .prev field and
      e:set_prev(err) method. .prev field returns previous error if any exist.
      e:set_prev(err) method expects err to be error object or nil and sets
      err as previous error of e. For instance:
      ```
      e1 = box.error.new({code = 111, reason = "cause"})
      e2 = box.error.new({code = 111, reason = "cause of cause"})
      
      e1:set_prev(e2)
      assert(e1.prev == e2) -- true
      ```
      Cycles are not allowed for error lists:
      ```
      e2:set_prev(e1)
      - error: 'builtin/error.lua: Cycles are not allowed'
      ```
      Nil is valid input to :set_prev() method:
      ```
      e1:set_prev(nil)
      assert(e1.prev == nil) -- true
      ```
      Note that error can be 'previous' only to the one error at once:
      ```
      e1:set_prev(e2)
      e3:set_prev(e2)
      assert(e1.prev == nil) -- true
      assert(e3.prev == e2) -- true
      ```
      Setting previous error does not erase its own previous members:
      ```
      -- e1 -> e2 -> e3 -> e4
      e1:set_prev(e2)
      e2:set_prev(e3)
      e3:set_prev(e4)
      e2:set_prev(e5)
      -- Now there are two lists: e1->e2->e5 and e3->e4
      assert(e1.prev == e2) -- true
      assert(e2.prev == e5) -- true
      assert(e3.prev == e4) -- true
      ```
      Alternatively:
      ```
      e1:set_prev(e2)
      e2:set_prev(e3)
      e3:set_prev(e4)
      e5:set_prev(e3)
      -- Now there are two lists: e1->e2 and e5->e3->e4
      assert(e1.prev == e2) -- true
      assert(e2.prev == nil) -- true
      assert(e5.prev == e3) -- true
      assert(e3.prev == e4) -- true
      ``
      Stacked diagnostics is also supported by IProto protocol. Now responses
      containing errors always (even if there's only one error to be returned)
      include new IProto key: IPROTO_ERROR_STACK (0x51). So, body corresponding to
      error response now looks like:
      ```
      MAP{IPROTO_ERROR : string, IPROTO_ERROR_STACK : ARRAY[MAP{ERROR_CODE : uint, ERROR_MESSAGE : string}, MAP{...}, ...]}
      ```
      where IPROTO_ERROR is 0x31 key, IPROTO_ERROR_STACK is 0x52, ERROR_CODE
      is 0x01 and ERROR_MESSAGE is 0x02.
      Instances of older versions (without support of stacked errors in
      protocol) simply ignore unknown keys and still rely only on IPROTO_ERROR
      key.
      4c465312
    • Nikita Pettik's avatar
      box: always promote error created via box.error() to diag · 4bcaf15e
      Nikita Pettik authored
      This patch makes box.error() always promote error to the diagnostic
      area despite of passed arguments.
      
      Closes #4829
      
      @TarantoolBot document
      Title: always promote error created via box.error() to diag
      
      box.error() is able to accept two types of argument: either pair of code
      and reason (box.error{code = 555, reason = 'Arbitrary message'}) or error
      object (box.error(err)). In the first case error is promoted to
      diagnostic area, meanwhile in the latter - it is not:
      ```
      e1 = box.error.new({code = 111, reason = "cause"})
      box.error({code = 111, reason = "err"})
      - error: err
      box.error.last()
      - err
      box.error(e1)
      - error: cause
      box.error.last()
      - err
      ```
      From now box.error(e1) sets error to diagnostic area as well:
      ```
      box.error(e1)
      - error: cause
      box.error.last()
      - cause
      ```
      4bcaf15e
    • Nikita Pettik's avatar
      box: use stacked diagnostic area for functional indexes · c15cef54
      Nikita Pettik authored
      Since we've introduced stacked diagnostic in previous commit, let's use
      it in the code implementing functional indexes.
      
      Part of #1148
      c15cef54
    • Nikita Pettik's avatar
      box: introduce stacked diagnostic area · 3b887d04
      Nikita Pettik authored
      In terms of implementation, now struct error objects can be organized
      into double-linked lists. To achieve this pointers to the next and
      previous elements (cause and effect correspondingly) have been added to
      struct error. It is worth mentioning that already existing rlist and
      stailq list implementations are not suitable: rlist is cycled list, as a
      result it is impossible to start iteration over the list from random
      list entry and finish it at the logical end of the list; stailq is
      single-linked list leaving no possibility to remove elements from the
      middle of the list.
      
      As a part of C interface, box_error_add() has been introduced. In
      contrast to box_error_set() it does not replace last raised error, but
      instead it adds error to the list of diagnostic errors having already
      been set. If error is to be deleted (its reference counter hits 0 value)
      it is unlinked from the list it belongs to and destroyed. Meanwhile,
      error destruction leads to decrement of reference counter of its
      previous error and so on.
      
      To organize errors into lists in Lua, table representing error object in
      Lua now has .prev field (corresponding to 'previous' error) and method
      :set_prev(e). The latter accepts error object (i.e. created via
      box.error.new() or box.error.last()) and nil value. Both field .prev and
      :set_prev() method are implemented as ffi functions. Also note that
      cycles are not allowed while organizing errors into lists:
      e1 -> e2 -> e3; e3:set_prev(e1) -- would lead to error.
      
      Part of #1148
      3b887d04
  10. Apr 02, 2020
    • Alexander V. Tikhonov's avatar
      gitlab-ci: add memory leaks detection via LSAN · e737400d
      Alexander V. Tikhonov authored
      The change enables memory leaks detection to existing ASAN testing
      routine and introduces suppression files with the corresponding
      exception list:
       * address sanitizer for compile-time: asan/asan.supp
       * memory leak sanitizer for run-time: asan/lsan.supp
      
      Furthermore, added engine and replication suites for ASAN testing
      routine.
      
      Additionally to the tests blacklisted within #4359,
      'box/on_shutdown.test.lua' is also disabled since it fails the
      introduced leak check. All blacklisted tests have to be enabled
      within #4360.
      
      Close #2058
      e737400d
    • Alexander V. Tikhonov's avatar
      test: use default replication connection timeout · fb892b7a
      Alexander V. Tikhonov authored
      All local connection timeout settings not related to the testing
      scenario are removed within this change. Instead of removed values
      the default one from src/box/lua/load_cfg.lua will be used.
      The approach with a single default value helps to avoid flaky test
      results regarding different timeout values and makes the future
      maintainence easier.
      
      The change is required for LSAN and ASAN testing machinery since it
      introduces a little overhead and cause failures for tests with
      excessively strict time limits.
      
      Needed for #2058
      fb892b7a
  11. Mar 27, 2020
    • Nikita Pettik's avatar
      box/error: don't set error created via box.error.new to diag · eaa86088
      Nikita Pettik authored
      To achieve this let's refactor luaT_error_create() to return error
      object instead of setting it via box_error_set().
      luaT_error_create() is used both to handle box.error() and
      box.error.new() invocations, and box.error() is still expected to set
      error to diagnostic area. So, luaT_error_call() which implements
      box.error() processing at the end calls diag_set_error().
      It is worth mentioning that net.box module relied on the fact that
      box.error.new() set error to diagnostic area: otherwise request errors
      don't get to diagnostic area on client side.
      
      Needed for #1148
      Closes #4778
      
      @TarantoolBot document
      Title: Don't promote error created via box.error.new to diagnostic area
      
      Now box.error.new() only creates error object, but doesn't set it to
      Tarantool's diagnostic area:
      ```
      box.error.clear()
      e = box.error.new({code = 111, reason = "cause"})
      assert(box.error.last() == nil)
      ---
      - true
      ...
      ```
      To set error in diagnostic area explicitly box.error.set() has been
      introduced. It accepts error object which is set as last system error
      (i.e. becomes available via box.error.last()).
      Finally, box.error.new() does not longer accept error object as an
      argument (this was undocumented feature).
      Note that patch does not affect box.error(), which still pushes error to
      diagnostic area. This fact is reflected in docs:
      '''
      Emulate a request error, with text based on one of the pre-defined
      Tarantool errors...
      '''
      eaa86088
  12. Mar 26, 2020
    • Vladislav Shpilevoy's avatar
      fio: close unused descriptors automatically · 3d5b4daa
      Vladislav Shpilevoy authored
      Fio.open() returned a file descriptor, which was not closed
      automatically after all its links were nullified. In other words,
      GC didn't close the descriptor.
      
      This was not really useful, because after fio.open() an exception
      may appear, and user needed to workaround this to manually call
      fio_object:close(). Also this was not consistent with io.open().
      
      Now fio.open() object closes the descriptor automatically when
      GCed.
      
      Closes #4727
      
      @TarantoolBot document
      Title: fio descriptor is closed automatically by GC
      
      fio.open() returns a descriptor which can be closed manually by
      calling :close() method, or it will be closed automatically, when
      it has no references, and GC deletes it.
      
      :close() method existed always, auto GC was added just now.
      
      Keep in mind, that the number of file descriptors is limited, and
      they can end earlier than GC will be triggered to collect not
      used descriptors. It is always better to close them manually as
      soon as possible.
      3d5b4daa
    • Vladislav Shpilevoy's avatar
      fiber: introduce schedule_task() internal function · 8443bd93
      Vladislav Shpilevoy authored
      fiber._internal.schedule_task() is an API for a singleton fiber
      worker object. It serves for not urgent delayed execution of
      functions. Main purpose - schedule execution of a function, which
      is going to yield, from a context, where a yield is not allowed.
      Such as an FFI object's GC callback.
      
      It will be used by SWIM and by fio, whose destruction yields, but
      they need to use GC finalizer, where a yield is not allowed.
      
      Part of #4727
      8443bd93
    • Nikita Pettik's avatar
      box/error: introduce box.error.set() method · f00945a1
      Nikita Pettik authored
      box.error.set(err) sets err to instance's diagnostics area. Argument err
      is supposed to be instance of error object. This method is required
      since we are going to avoid adding created via box.error.new() errors to
      Tarantool's diagnostic area.
      
      Needed for #1148
      Part of #4778
      f00945a1
    • Nikita Pettik's avatar
      test: move box.error tests to box/error.test.lua · 195d99d0
      Nikita Pettik authored
      We are going to introduce more tests related to error module, so let's
      move all error-related tests from box/misc.test.lua to a separate test
      file (box/error.test.lua).
      
      Needed for #1148
      195d99d0
    • Cyrill Gorcunov's avatar
      test: unit/popen -- provide a child process · 12086678
      Cyrill Gorcunov authored
      
      Testing via plain C interface with a shell is not stable,
      the shell might simply be misconfigured or not found and
      we will simply stuck forever (the signal handling in libev
      is tricky and requires at least idle cycles or similar to
      pass event processing). Thus lets rather run a program we
      know is presenting in the system (popen-child executable).
      
      Fixes #4811
      
      Signed-off-by: default avatarCyrill Gorcunov <gorcunov@gmail.com>
      12086678
  13. Mar 25, 2020
Loading