Skip to content
Snippets Groups Projects
  1. Apr 07, 2020
    • Nikita Pettik's avatar
      iproto: support error stacked diagnostic area · 4c465312
      Nikita Pettik authored
      This patch introduces support of stacked errors in IProto protocol and
      in net.box module.
      
      Closes #1148
      
      @TarantoolBot document
      Title: Stacked error diagnostic area
      
      Starting from now errors can be organized into lists. To achieve this
      Lua table representing error object is extended with .prev field and
      e:set_prev(err) method. .prev field returns previous error if any exist.
      e:set_prev(err) method expects err to be error object or nil and sets
      err as previous error of e. For instance:
      ```
      e1 = box.error.new({code = 111, reason = "cause"})
      e2 = box.error.new({code = 111, reason = "cause of cause"})
      
      e1:set_prev(e2)
      assert(e1.prev == e2) -- true
      ```
      Cycles are not allowed for error lists:
      ```
      e2:set_prev(e1)
      - error: 'builtin/error.lua: Cycles are not allowed'
      ```
      Nil is valid input to :set_prev() method:
      ```
      e1:set_prev(nil)
      assert(e1.prev == nil) -- true
      ```
      Note that error can be 'previous' only to the one error at once:
      ```
      e1:set_prev(e2)
      e3:set_prev(e2)
      assert(e1.prev == nil) -- true
      assert(e3.prev == e2) -- true
      ```
      Setting previous error does not erase its own previous members:
      ```
      -- e1 -> e2 -> e3 -> e4
      e1:set_prev(e2)
      e2:set_prev(e3)
      e3:set_prev(e4)
      e2:set_prev(e5)
      -- Now there are two lists: e1->e2->e5 and e3->e4
      assert(e1.prev == e2) -- true
      assert(e2.prev == e5) -- true
      assert(e3.prev == e4) -- true
      ```
      Alternatively:
      ```
      e1:set_prev(e2)
      e2:set_prev(e3)
      e3:set_prev(e4)
      e5:set_prev(e3)
      -- Now there are two lists: e1->e2 and e5->e3->e4
      assert(e1.prev == e2) -- true
      assert(e2.prev == nil) -- true
      assert(e5.prev == e3) -- true
      assert(e3.prev == e4) -- true
      ``
      Stacked diagnostics is also supported by IProto protocol. Now responses
      containing errors always (even if there's only one error to be returned)
      include new IProto key: IPROTO_ERROR_STACK (0x51). So, body corresponding to
      error response now looks like:
      ```
      MAP{IPROTO_ERROR : string, IPROTO_ERROR_STACK : ARRAY[MAP{ERROR_CODE : uint, ERROR_MESSAGE : string}, MAP{...}, ...]}
      ```
      where IPROTO_ERROR is 0x31 key, IPROTO_ERROR_STACK is 0x52, ERROR_CODE
      is 0x01 and ERROR_MESSAGE is 0x02.
      Instances of older versions (without support of stacked errors in
      protocol) simply ignore unknown keys and still rely only on IPROTO_ERROR
      key.
      4c465312
    • Nikita Pettik's avatar
      box: always promote error created via box.error() to diag · 4bcaf15e
      Nikita Pettik authored
      This patch makes box.error() always promote error to the diagnostic
      area despite of passed arguments.
      
      Closes #4829
      
      @TarantoolBot document
      Title: always promote error created via box.error() to diag
      
      box.error() is able to accept two types of argument: either pair of code
      and reason (box.error{code = 555, reason = 'Arbitrary message'}) or error
      object (box.error(err)). In the first case error is promoted to
      diagnostic area, meanwhile in the latter - it is not:
      ```
      e1 = box.error.new({code = 111, reason = "cause"})
      box.error({code = 111, reason = "err"})
      - error: err
      box.error.last()
      - err
      box.error(e1)
      - error: cause
      box.error.last()
      - err
      ```
      From now box.error(e1) sets error to diagnostic area as well:
      ```
      box.error(e1)
      - error: cause
      box.error.last()
      - cause
      ```
      4bcaf15e
    • Kirill Shcherbatov's avatar
      iproto: refactor error encoding with mpstream · ba7304fb
      Kirill Shcherbatov authored
      Refactor iproto_reply_error and iproto_write_error with a new
      mpstream-based helper mpstream_iproto_encode_error that encodes
      error object for iproto protocol on a given stream object.
      Previously each routine implemented an own error encoding, but
      with the increasing complexity of encode operation with following
      patches we need a uniform way to do it.
      
      The iproto_write_error routine starts using region location
      to use region-based mpstream. It is not a problem itself, because
      errors reporting is not really performance-critical path.
      
      Needed for #1148
      ba7304fb
    • Nikita Pettik's avatar
    • Nikita Pettik's avatar
      box: use stacked diagnostic area for functional indexes · c15cef54
      Nikita Pettik authored
      Since we've introduced stacked diagnostic in previous commit, let's use
      it in the code implementing functional indexes.
      
      Part of #1148
      c15cef54
    • Nikita Pettik's avatar
      box: introduce stacked diagnostic area · 3b887d04
      Nikita Pettik authored
      In terms of implementation, now struct error objects can be organized
      into double-linked lists. To achieve this pointers to the next and
      previous elements (cause and effect correspondingly) have been added to
      struct error. It is worth mentioning that already existing rlist and
      stailq list implementations are not suitable: rlist is cycled list, as a
      result it is impossible to start iteration over the list from random
      list entry and finish it at the logical end of the list; stailq is
      single-linked list leaving no possibility to remove elements from the
      middle of the list.
      
      As a part of C interface, box_error_add() has been introduced. In
      contrast to box_error_set() it does not replace last raised error, but
      instead it adds error to the list of diagnostic errors having already
      been set. If error is to be deleted (its reference counter hits 0 value)
      it is unlinked from the list it belongs to and destroyed. Meanwhile,
      error destruction leads to decrement of reference counter of its
      previous error and so on.
      
      To organize errors into lists in Lua, table representing error object in
      Lua now has .prev field (corresponding to 'previous' error) and method
      :set_prev(e). The latter accepts error object (i.e. created via
      box.error.new() or box.error.last()) and nil value. Both field .prev and
      :set_prev() method are implemented as ffi functions. Also note that
      cycles are not allowed while organizing errors into lists:
      e1 -> e2 -> e3; e3:set_prev(e1) -- would lead to error.
      
      Part of #1148
      3b887d04
    • Tim Curtis's avatar
      build: Fix static build based on Dockerfile · 9706de41
      Tim Curtis authored
      Found that 'openssl-1.1.0h' is EOL and its external link was removed.
      Fixed static build based on Dockerfile with the new link to the external
      openssl version 'openssl-1.1.1f'.
      
      Close #4830
      9706de41
  2. Apr 06, 2020
  3. Apr 02, 2020
    • Leonid's avatar
      luarocks: Add a kludge for option force of luarocks remove · 6ef13c69
      Leonid authored
      @Changelog
      Fixed the tarantoolctl rocks remove flag --force
      
      Forwarding of the --force flag to tarantoolctl rocks module was added.
      (Command: tarantoolctl rocks remove --force)
      
      Fixes: #3632
      6ef13c69
    • Leonid's avatar
      luarocks: Add a kludge for option all of luarocks search · 432ffa97
      Leonid authored
      @Changelog
      Fixed the tarantoolctl rocks search flag --all
      
      Forwarding of the --all flag to tarantoolctl rocks module was added.
      (Command: tarantoolctl rocks search --all)
      
      Fixes: #4529
      432ffa97
    • Alexander V. Tikhonov's avatar
      static build: create new build w/o dockerfile · fc55875e
      Alexander V. Tikhonov authored
      Fixed static build with '-DBUILD_STATIC=ON' option:
      
       - installed liblzma-dev library for libunwind static, due to found that
         static libunwind library uses undefined lzma functions:
           nm -a /usr/lib/x86_64-linux-gnu/libunwind-x86_64.a | grep lzma
                       U lzma_index_buffer_decode
                       U lzma_index_end
                       U lzma_index_size
                       U lzma_index_uncompressed_size
                       U lzma_stream_buffer_decode
                       U lzma_stream_footer_decode
         while dynamic libunwind correctly sees liblzma installed:
           ldd /usr/lib/x86_64-linux-gnu/libunwind-x86_64.so | grep lzma
             liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f8fd1c23000)
         so to fix it the static library of lzma was needed.
      
       - added lzma library to unwind library for Tarantool build at file:
           cmake/compiler.cmake
         due to fail:
           /usr/lib/x86_64-linux-gnu/libunwind-x86_64.a(elf64.o):
             In function `xz_uncompressed_size':
           ./src/elfxx.c:194: undefined reference to `lzma_stream_footer_decode'
           ./src/elfxx.c:201: undefined reference to `lzma_index_buffer_decode'
           ./src/elfxx.c:205: undefined reference to `lzma_index_size'
           ./src/elfxx.c:210: undefined reference to `lzma_index_end'
           ./src/elfxx.c:207: undefined reference to `lzma_index_uncompressed_size'
           ./src/elfxx.c:210: undefined reference to `lzma_index_end'
           /usr/lib/x86_64-linux-gnu/libunwind-x86_64.a(elf64.o):
             In function `_Uelf64_extract_minidebuginfo':
           ./src/elfxx.c:278: undefined reference to `lzma_stream_buffer_decode'
           collect2: error: ld returned 1 exit status
           test/unit/CMakeFiles/luaL_iterator.test.dir/build.make:134:
             recipe for target 'test/unit/luaL_iterator.test' failed
           make[2]: *** [test/unit/luaL_iterator.test] Error 1
      
       - added dl library to gomp library for test/unit tests
         binaries builds at file:
           cmake/BuildMisc.cmake
         due to fail:
           /usr/lib/gcc/x86_64-linux-gnu/7/libgomp.a(target.o):(.text+0x34d):
             more undefined references to `dlsym' follow
           /usr/lib/gcc/x86_64-linux-gnu/7/libgomp.a(target.o): In function
             `gomp_target_init':
           (.text+0x9cc): undefined reference to `dlerror'
           collect2: error: ld returned 1 exit status
      
        - added dl library to icu library for test/unit tests
         binaries builds at file:
           cmake/FindICU.cmake
         due to fail:
           /usr/x86_64-linux-gnu/libicuuc.a(putil.ao):
             In function `uprv_dl_open_60':
           (.text+0x1ce2): undefined reference to `dlopen'
           /usr/x86_64-linux-gnu/libicuuc.a(putil.ao):
             In function `uprv_dlsym_func_60':
           (.text+0x1d3d): undefined reference to `dlsym'
           /usr/x86_64-linux-gnu/libicuuc.a(putil.ao):
             In function `uprv_dl_close_60':
           (.text+0x1d21): undefined reference to `dlclose'
           collect2: error: ld returned 1 exit status
      
      Added static build to gitlab-ci in release check criteria named
      as static_build job. Previously named static_build job renamed to
      static_docker_build, due to it checks the build at Dockerfile.
      
      Also moved static build make targets from .gitlab.mk to .travis.mk
      to store it in common place with the other test/build make targets.
      Moved environement from .gitlab-ci.yml file into make targets to
      make this targets true building in static w/o additional setup.
      
      Close #4551
      fc55875e
    • Alexander V. Tikhonov's avatar
      build: resolve issues with sourceforge.net · 4a9302b8
      Alexander V. Tikhonov authored
      Found that wget may fail on downloading the file from flaky
      available servers with 500 HTTP error, like it was seen on
      icu4c sources downloading, please check the issue:
      https://sourceforge.net/p/forge/site-support/20071/
      
      Found that curl successfully downloads needed files even in
      such situations. Decided to use curl instead of wget tool
      to avoid of such errors feather.
      
      Also found that sourceforge site too often has issues and
      responds with 500 HTTP error. Decided to use the link from
      github instead of sourceforge to download the icu4c sources,
      as suggested on icu4c web site.
      4a9302b8
    • Alexander V. Tikhonov's avatar
      build: added static build subdirectory cleanup · 63640f5a
      Alexander V. Tikhonov authored
      Added the command for 'build' directory cleanup. It purges all
      artefacts produced for curl build, including the old configuration
      in build/curl.
      63640f5a
    • Alexander V. Tikhonov's avatar
      build: cleanup Dockerfile in static build · 4f6862d6
      Alexander V. Tikhonov authored
      Set Dockerfile WORKDIR from the very start of Tarantool sources
      builds to make the Dockerfile code more readable and removed all
      duplicating calls to Tarantool sources directory changes. Check:
        https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#workdir
      4f6862d6
    • Alexander V. Tikhonov's avatar
      build: remove LD_LIBRARY_PATH in static build · abac5fe7
      Alexander V. Tikhonov authored
      Removed LD_LIBRARY_PATH environment from curl build, due to the
      path is empty in the docker image and no other commands from the
      Dockerfile filled it, seemed that it was not needed.
      abac5fe7
    • Alexander V. Tikhonov's avatar
      build: speedup static build · 0b543056
      Alexander V. Tikhonov authored
      Added '-j' option to make for tools buildings, it decreased build time in 4 times.
      0b543056
    • Alexander V. Tikhonov's avatar
      gitlab-ci: add memory leaks detection via LSAN · e737400d
      Alexander V. Tikhonov authored
      The change enables memory leaks detection to existing ASAN testing
      routine and introduces suppression files with the corresponding
      exception list:
       * address sanitizer for compile-time: asan/asan.supp
       * memory leak sanitizer for run-time: asan/lsan.supp
      
      Furthermore, added engine and replication suites for ASAN testing
      routine.
      
      Additionally to the tests blacklisted within #4359,
      'box/on_shutdown.test.lua' is also disabled since it fails the
      introduced leak check. All blacklisted tests have to be enabled
      within #4360.
      
      Close #2058
      e737400d
    • Alexander V. Tikhonov's avatar
      test: use default replication connection timeout · fb892b7a
      Alexander V. Tikhonov authored
      All local connection timeout settings not related to the testing
      scenario are removed within this change. Instead of removed values
      the default one from src/box/lua/load_cfg.lua will be used.
      The approach with a single default value helps to avoid flaky test
      results regarding different timeout values and makes the future
      maintainence easier.
      
      The change is required for LSAN and ASAN testing machinery since it
      introduces a little overhead and cause failures for tests with
      excessively strict time limits.
      
      Needed for #2058
      fb892b7a
    • Alexander V. Tikhonov's avatar
      gitlab-ci: implement OSX 10.14 testing on mac mini · 7faa1abe
      Alexander V. Tikhonov authored
      Switched OSX 10.14 test jobs to MAC mini hosts.
      Renamed Gitlab make rule target 'test_%_no_deps' to 'test_%', which is
      used to pass all appropriate target calls to Travis make rule target.
      7faa1abe
  4. Mar 29, 2020
    • Alexander Turenko's avatar
      test: update test-run · 9933c5d2
      Alexander Turenko authored
      Added popen build-in module to exclusions of pretest_clean script. It'll
      allow to test the module from a test of 'core = tarantool' type.
      
      Part of #4031
      Unverified
      9933c5d2
  5. Mar 27, 2020
    • Nikita Pettik's avatar
      box/error: don't set error created via box.error.new to diag · eaa86088
      Nikita Pettik authored
      To achieve this let's refactor luaT_error_create() to return error
      object instead of setting it via box_error_set().
      luaT_error_create() is used both to handle box.error() and
      box.error.new() invocations, and box.error() is still expected to set
      error to diagnostic area. So, luaT_error_call() which implements
      box.error() processing at the end calls diag_set_error().
      It is worth mentioning that net.box module relied on the fact that
      box.error.new() set error to diagnostic area: otherwise request errors
      don't get to diagnostic area on client side.
      
      Needed for #1148
      Closes #4778
      
      @TarantoolBot document
      Title: Don't promote error created via box.error.new to diagnostic area
      
      Now box.error.new() only creates error object, but doesn't set it to
      Tarantool's diagnostic area:
      ```
      box.error.clear()
      e = box.error.new({code = 111, reason = "cause"})
      assert(box.error.last() == nil)
      ---
      - true
      ...
      ```
      To set error in diagnostic area explicitly box.error.set() has been
      introduced. It accepts error object which is set as last system error
      (i.e. becomes available via box.error.last()).
      Finally, box.error.new() does not longer accept error object as an
      argument (this was undocumented feature).
      Note that patch does not affect box.error(), which still pushes error to
      diagnostic area. This fact is reflected in docs:
      '''
      Emulate a request error, with text based on one of the pre-defined
      Tarantool errors...
      '''
      eaa86088
  6. Mar 26, 2020
    • Vladislav Shpilevoy's avatar
      fio: on close() don't loose fd if internal close fails · 177679a6
      Vladislav Shpilevoy authored
      File descriptor was set to -1 regardless of whether
      the object was closed properly. As a result, in case
      of an error the descriptor would leak.
      
      GC finalizer of a descriptor is left intact not to
      overcomplicate it.
      177679a6
    • Vladislav Shpilevoy's avatar
      swim: use fiber._internal.schedule_task() for GC · f073834b
      Vladislav Shpilevoy authored
      swim object created a new fiber in its GC function, because C
      function swim_delete() yields, and can't be called from an ffi.gc
      hook.
      
      It is not needed since the fiber module has a single worker
      exactly for such cases. The patch uses it.
      
      Follow up #4727
      f073834b
    • Vladislav Shpilevoy's avatar
      fio: close unused descriptors automatically · 3d5b4daa
      Vladislav Shpilevoy authored
      Fio.open() returned a file descriptor, which was not closed
      automatically after all its links were nullified. In other words,
      GC didn't close the descriptor.
      
      This was not really useful, because after fio.open() an exception
      may appear, and user needed to workaround this to manually call
      fio_object:close(). Also this was not consistent with io.open().
      
      Now fio.open() object closes the descriptor automatically when
      GCed.
      
      Closes #4727
      
      @TarantoolBot document
      Title: fio descriptor is closed automatically by GC
      
      fio.open() returns a descriptor which can be closed manually by
      calling :close() method, or it will be closed automatically, when
      it has no references, and GC deletes it.
      
      :close() method existed always, auto GC was added just now.
      
      Keep in mind, that the number of file descriptors is limited, and
      they can end earlier than GC will be triggered to collect not
      used descriptors. It is always better to close them manually as
      soon as possible.
      3d5b4daa
    • Vladislav Shpilevoy's avatar
      fiber: introduce schedule_task() internal function · 8443bd93
      Vladislav Shpilevoy authored
      fiber._internal.schedule_task() is an API for a singleton fiber
      worker object. It serves for not urgent delayed execution of
      functions. Main purpose - schedule execution of a function, which
      is going to yield, from a context, where a yield is not allowed.
      Such as an FFI object's GC callback.
      
      It will be used by SWIM and by fio, whose destruction yields, but
      they need to use GC finalizer, where a yield is not allowed.
      
      Part of #4727
      8443bd93
    • Alexander V. Tikhonov's avatar
      test: fix OSX host setup · a41bef3b
      Alexander V. Tikhonov authored
      Fixed OSX host setup for Tarantool build:
      - set brew installation from Homebrew repository instructions;
      - set in use Python 2 latest commit from tapped local formula,
        since Python 2 is EOL, also removed extra pip installation with
        get-pip script, because tapped formula installs pip itself.
        python@2 was deleted from homebrew/core in commit 028f11f9e:
          python@2: delete (https://github.com/Homebrew/homebrew-core/issues/49796)
          EOL 1 January 2020.
        Tapped formula created from the latest formula before its removal:
          git -C "$(brew --repo homebrew/core)" show 028f11f9e^:Formula/python@2.rb
      - added upgrade packages call to avoid of fails on already
        installed packages, but with previous version;
      - fixed the gitlab-ci configuration for sudo on testing hosts and removed pip
        option '--user' to avoid of use the users paths with special setup for it.
      
      Fixed OSX host setup for Tarantool test:
      - set maximum processes limit value to 2500 for testing process;
      - new Mac machines are going to be added into CI and usernames on them
        are long according to internal policies. It makes a home directory to
        be long too and so a path to a unix socket created during testing can
        be longer then UNIX_PATH_MAX=108 constant which is described as issue
          https://github.com/tarantool/tarantool/issues/4634
        To avoid of it the short working directory for testing set by option:
          --vardir /tmp/tnt
      a41bef3b
    • Nikita Pettik's avatar
      box/error: introduce box.error.set() method · f00945a1
      Nikita Pettik authored
      box.error.set(err) sets err to instance's diagnostics area. Argument err
      is supposed to be instance of error object. This method is required
      since we are going to avoid adding created via box.error.new() errors to
      Tarantool's diagnostic area.
      
      Needed for #1148
      Part of #4778
      f00945a1
    • Nikita Pettik's avatar
      test: move box.error tests to box/error.test.lua · 195d99d0
      Nikita Pettik authored
      We are going to introduce more tests related to error module, so let's
      move all error-related tests from box/misc.test.lua to a separate test
      file (box/error.test.lua).
      
      Needed for #1148
      195d99d0
    • Kirill Shcherbatov's avatar
      box: rename diag_add_error to diag_set_error · 55a39946
      Kirill Shcherbatov authored
      Let's rename diag_add_error() to diag_set_error() because it actually
      replaces an error object in diagnostic area with a new one and this name
      is not representative. Moreover, we are going to introduce a new
      diag_add_error() which will place error at the top of stack diagnostic
      area.
      
      Needed for #1148
      55a39946
    • Kirill Shcherbatov's avatar
      box: rfc for stacked diagnostic area · 61cdd89f
      Kirill Shcherbatov authored
      Part of #1148
      61cdd89f
    • Cyrill Gorcunov's avatar
      test: unit/popen -- provide a child process · 12086678
      Cyrill Gorcunov authored
      
      Testing via plain C interface with a shell is not stable,
      the shell might simply be misconfigured or not found and
      we will simply stuck forever (the signal handling in libev
      is tricky and requires at least idle cycles or similar to
      pass event processing). Thus lets rather run a program we
      know is presenting in the system (popen-child executable).
      
      Fixes #4811
      
      Signed-off-by: default avatarCyrill Gorcunov <gorcunov@gmail.com>
      12086678
    • Cyrill Gorcunov's avatar
      popen: do not require space for shell args · c6b8ed77
      Cyrill Gorcunov authored
      
      In case of direct execute without using a shell there
      is no need to require a caller to allocate redundant
      space, lets pass executable name in first argument.
      
      Since this is yet testing api we're allowed to change
      without breaking aything.
      
      Signed-off-by: default avatarCyrill Gorcunov <gorcunov@gmail.com>
      c6b8ed77
  7. Mar 25, 2020
  8. Mar 23, 2020
    • Timur Safin's avatar
      evio: workaround for wsl1 so_linger assertion · 734bcafc
      Timur Safin authored
      
      SO_LINGER makes no much sense for unix-sockets, and Microsoft WSL
      is returning EINVAL if setsockopts called for SO_LINGER over unix
      sockets:
      
        [004] 2020-03-11 18:42:29.592 [29182] main/102/app sio.c:169 !> SystemError setsockopt(SO_LINGER), called on fd 16, aka
        [004] 2020-03-11 18:42:29.592 [29182] main/102/app F> can't initialize storage: setsockopt(SO_LINGER), called on fd 16,
        [004] 2020-03-11 18:42:29.592 [29182] main/102/app F> can't initialize storage: setsockopt(SO_LINGER), called on fd 16,
      
      And it's sort of correct here, but the problem is Linux is simply
      silently ignoring it, which passes tests.
      
      After much debates we decided to work-around this case via CMAKE
      define.
      
      NB! In a future (April/May 2020), when WSL2 with full Linux kernel
      would be released we should disable this check.
      
      Acked-by: default avatarCyrill Gorcunov <gorcunov@gmail.com>
      734bcafc
  9. Mar 20, 2020
Loading