Skip to content
Snippets Groups Projects
  1. Dec 01, 2021
    • Vladimir Davydov's avatar
      net.box: fix IO error messages · 2c1e0b00
      Vladimir Davydov authored
      1. The net.box state machine is supposed to set E_NO_CONNECTION on
         error, but netbox_transport_connect returns an error returned by
         coio_connect and coio_read as is. Fix this.
      
      2. netbox_transport_communicate sets the error message to
         strerror(errno) on coio_read or coio_write error. This works now,
         because iostream can only be backed by plain sockets, but once we
         implement encryption, errno won't make any sense. We should use
         error::errmsg instead.
      
      3. While we are at it, fix luaT_netbox_transport_make_request: it's
         wrong to pass errmsg directly to box_error_raise - we should use a
         format specifier (%s).
      
      Fixes commit c13b3a31 ("net.box: rewrite
      state machine (transport) in C").
      2c1e0b00
    • Vladimir Davydov's avatar
      error: append strerror(errno) to SystemError and SocketError errmsg · 29026b19
      Vladimir Davydov authored
      Currently, we append strerror(errno) when logging those errors while
      error::errmsg only contains the message passed to the error constructor,
      which usually points at the system call that failed, e.g. sio_connect()
      sets it to
      
        connect, called on fd 50, aka 127.0.0.1:33728
      
      Errno can shed light at the reason of failure so we'd better append it
      to errmsg too:
      
        connect, called on fd 50, aka 127.0.0.1:33728: Connection refused
      
      In particular, this improves error messages returned by the net.box
      state machine, which was recently switched to diag, see commit
      c13b3a31 ("net.box: rewrite state
      machine (transport) in C").
      
      After this patch, we don't need to append strerror(errno) to SystemError
      message manually anymore so this patch also walks over the whole code
      base and removes strerror from diag_set calls - there are just two
      modules that do that, fio and say.
      
      Also, this patch slightly changes logged error messages:
       - A colon (:) is printed after SystemError, because SystemError::log
         doesn't override Exception::log anymore. This makes SystemError
         logging consistent with other error logging.
       - For the same reason we now print the actual error type of errors
         derived from SystemError (SocketError, OutOfMemory and TimedOut)
         instead of SystemError to the log (this should be fine), and don't
         print strerror(errno) for OutOfMemory and TimedOut (it's useless
         anyway).
      29026b19
  2. Nov 30, 2021
  3. Nov 29, 2021
  4. Nov 26, 2021
    • Yaroslav Lobankov's avatar
      ci: run 'integration.yml' WF in pre-commit testing · 0c277b84
      Yaroslav Lobankov authored
      Problem:
      
      For now, the integration testing runs only on a push of changes/tags
      to the 'master' and release branches. This approach is not good enough
      because we become aware of integration issues after the changes were
      merged already to the target branch.
      
      So this patch adds the facility to run the integration testing on a dev
      branch to catch integration issues before the changes are merged to the
      target branch. Now it can be done via naming the branch as `*-full-ci`
      and pushing it to the main repository or setting the 'full-ci' label on
      the pull request.
      0c277b84
    • Alexander Turenko's avatar
      ci: enable CentOS 7 on aarch64 · 65f06ca5
      Alexander Turenko authored
      This commit actually reverts 2.10.0-beta1-136-gaa2a6bc83 ('ci:
      temporarily disable deploy for CentOS 7 for aarch64').
      
      I built packages necessary for testing for CentOS 7 on aarch64 and
      placed them into packpack backports [1].
      
      The source packages are from CentOS 8 [2] and are the following:
      
      * Cython-0.29.14-4.module_el8.3.0+441+3b561464.src.rpm
      * python-gevent-1.2.2-4.el8.src.rpm
      * PyYAML-5.3.1-1.module_el8.3.0+441+3b561464.src.rpm
      * python-six-1.12.0-9.module_el8.3.0+441+3b561464.src.rpm
      
      [1]: https://packagecloud.io/packpack/backports
      [2]: https://vault.centos.org/8.3.2011/AppStream/Source/SPackages/
      
      If you're curious about steps to re-build the packages for CentOS 7,
      read below.
      
      ----
      
      Build steps are the following:
      
      ```sh
      $ ssh <aarch64 machine>
      $ docker run -it -v <...>:/src -v <...>:/res centos:7
      ```
      
      Install tools:
      
      ```sh
      $ yum install yum-utils rpm-build
      ```
      
      (Here I also installed epel-release, but I doubt that it is necessary.)
      
      Build and install Cython:
      
      ```sh
      $ rpmbuild --rebuild /src/Cpython*.src.rpm
      $ vim /root/rpmbuild/SPECS/Cython.spec
      <..fix changelog dates..>
      $ rpmbuild -bb /root/rpmbuild/SPECS/Cython.spec
      <...>
      error: Failed build dependencies:
        gcc is needed by Cython-0.29.14-4.el7.aarch64
        python3-devel is needed by Cython-0.29.14-4.el7.aarch64
        python3-setuptools is needed by Cython-0.29.14-4.el7.aarch64
        python3-rpm-macros is needed by Cython-0.29.14-4.el7.aarch64
      $ yum install gcc python3-devel python3-rpm-macros python3-setuptools
      $ rpmbuild -bb /root/rpmbuild/SPECS/Cython.spec # repeat, yep
      $ ls -1 /root/rpmbuild/RPMS/aarch64/
      Cython-debuginfo-0.29.14-4.el7.aarch64.rpm
      python3-Cython-0.29.14-4.el7.aarch64.rpm
      $ cp /root/rpmbuild/RPMS/aarch64/* /res/
      $ rpm -i /root/rpmbuild/RPMS/aarch64/*
      ```
      
      Build python-gevent:
      
      ```sh
      $ rpmbuild --rebuild /src/python-gevent*.src.rpm
      <...>
      error: Failed build dependencies:
        c-ares-devel is needed by python-gevent-1.2.2-4.el7.aarch64
        libev-devel is needed by python-gevent-1.2.2-4.el7.aarch64
        python2-devel is needed by python-gevent-1.2.2-4.el7.aarch64
      $ yum install c-ares-devel libev-devel # but not python2-devel
      $ vim /root/rpmbuild/SPECS/python-gevent.spec
      <..leave '%bcond_with python2'..>
      $ rpmbuild -bb /root/rpmbuild/SPECS/python-gevent.spec
      $ ls -1 /root/rpmbuild/RPMS/aarch64/*gevent*
      /root/rpmbuild/RPMS/aarch64/python-gevent-debuginfo-1.2.2-4.el7.aarch64.rpm
      /root/rpmbuild/RPMS/aarch64/python3-gevent-1.2.2-4.el7.aarch64.rpm
      $ cp /root/rpmbuild/RPMS/aarch64/*gevent* /res/
      ```
      
      Build PyYAML:
      
      ```sh
      $ rpmbuild --rebuild /src/PyYAML*.src.rpm
      <...>
      error: Failed build dependencies:
          libyaml-devel is needed by PyYAML-5.3.1-1.el7.aarch64
      $ yum install libyaml-devel
      $ rpmbuild --rebuild /src/PyYAML*.src.rpm # again, yep
      $ ls -1 /root/rpmbuild/RPMS/aarch64/*{pyyaml,PyYAML}*
      /root/rpmbuild/RPMS/aarch64/PyYAML-debuginfo-5.3.1-1.el7.aarch64.rpm
      /root/rpmbuild/RPMS/aarch64/python3-pyyaml-5.3.1-1.el7.aarch64.rpm
      $ cp /root/rpmbuild/RPMS/aarch64/*{pyyaml,PyYAML}* /res/
      ```
      
      Build python-six:
      
      ```sh
      $ rpmbuild --rebuild /src/python-six*.src.rpm
      error: Failed build dependencies:
          python3-wheel is needed by python-six-1.12.0-9.el7.noarch
          python3-pytest is needed by python-six-1.12.0-9.el7.noarch
          python3-tkinter is needed by python-six-1.12.0-9.el7.noarch
      $ yum install python3-wheel python3-pytest python3-tkinter
      $ rpmbuild --rebuild /src/python-six*.src.rpm # again, yep
      error: Failed build dependencies:
          python3-pytest is needed by python-six-1.12.0-9.el7.noarch
      $ vim /root/rpmbuild/SPECS/python-six.spec
      <..change '%bcond_without tests' to '%bcond_with tests'..>
      $ rpmbuild -bb /root/rpmbuild/SPECS/python-six.spec
      $ ls -1 /root/rpmbuild/RPMS/noarch/
      python3-six-1.12.0-9.el7.noarch.rpm
      $ cp /root/rpmbuild/RPMS/noarch/* /res/
      ```
      65f06ca5
    • Georgiy Lebedev's avatar
      box: fix 'log.cfg' update on 'box.cfg' error · d30d6cfd
      Georgiy Lebedev authored
      If 'box.cfg' fails with error, 'log.cfg' still gets updated: instead,
      update 'log.cfg' after verifying 'box.cfg', and rollback if update of
      'log.cfg' fails.
      
      Closes #6086
      d30d6cfd
    • Georgiy Lebedev's avatar
      log: fix box_api_cfg_set_log_{level, format} · e18a07b3
      Georgiy Lebedev authored
      box_api_cfg_set_log_{level, format} are part of dynamic_cfg (see
      box/lua/load_cfg.lua) and are expected to throw exceptions (see
      reload_cfg ib.).
      
      Needed for #6086
      e18a07b3
    • Georgiy Lebedev's avatar
      box.xlog: reduce snapshot verbosity · a945a198
      Georgiy Lebedev authored
      On big instances core produces too much logs related to shapshotting,
      which feels redudant: reduce snapshot verbosity.
      
      Closes #6220
      a945a198
    • Georgiy Lebedev's avatar
      core.say: improve say_ratelimit macro and friends · 3d0898a8
      Georgiy Lebedev authored
      Make say_ratelimit_check emit warnings about suppressed messages only if
      the correspoding log level is greater or equal then WARN.
      
      Add say_crit_ratelimited and say_info_ratelimited macros for
      convenience.
      
      Needed for #6220
      3d0898a8
  5. Nov 24, 2021
    • Igor Munkin's avatar
      luajit: bump new version · e8765ba5
      Igor Munkin authored
      * memprof: add info about trace start to symtab
      * memprof: group allocations on traces by traceno
      * memprof: refactor location parsing
      * test: separate memprof Lua API tests into subtests
      
      Closes #5814
      Unverified
      e8765ba5
    • Mergen Imeev's avatar
      sql: introduce field type MAP · 1364629e
      Mergen Imeev authored
      This patch introduces MAP to SQL. After this patch, all SQL operations
      and built-in functions should work correctly with MAP values. However,
      there is currently no way to create MAP values using only SQL tools.
      
      Part of #4763
      
      @TarantoolBot document
      Title: Field type MAP in SQL
      
      Properties of type MAP in SQL:
      1) a value of type MAP can be implicitly and explicitly cast only to
      ANY;
      2) only a value of type ANY with primitive type MAP can be explicitly
      cast to MAP;
      3) a value of any other type cannot be implicitly cast to MAP;
      4) a value of type MAP cannot participate in arithmetic, bitwise,
      comparison, and concatination operations.
      1364629e
    • Mergen Imeev's avatar
      sql: introduce field type ARRAY · e2102a15
      Mergen Imeev authored
      This patch introduces ARRAY to SQL. After this patch, all SQL operations
      and built-in functions should work correctly with ARRAY values. However,
      there is currently no way to create ARRAY values using only SQL tools.
      
      Part of #4762
      
      @TarantoolBot document
      Title: Field type ARRAY in SQL
      
      Properties of type ARRAY in SQL:
      1) a value of type ARRAY can be implicitly and explicitly cast only
         to ANY;
      2) only a value of type ANY with primitive type ARRAY can be explicitly
         cast to ARRAY;
      3) a value of any other type cannot be implicitly cast to ARRAY;
      4) a value of type ARRAY cannot participate in arithmetic, bitwise,
         comparison, and concatination operations.
      e2102a15
    • Mergen Imeev's avatar
      sql: introduce mem_snprintf() · 36668990
      Mergen Imeev authored
      This patch introduces the mem_snprintf() function, which writes the
      string representation of a MEM to buf.
      36668990
    • Mergen Imeev's avatar
      sql: omit quotes for UUID values in errors · 3cb98866
      Mergen Imeev authored
      This patch removes quotes from the representation of UUID values in the
      error description. This is because UUID values are printed without
      quotes elsewhere.
      3cb98866
    • Vladimir Davydov's avatar
      ci: run full ci only on PRs with 'full-ci' label · 6c2e664e
      Vladimir Davydov authored
      After this commit only three workflow are run on pull request or push to
      a developer branch:
       - luacheck
       - release
       - debug_coverage
      
      To run all other tests, one should either name the branch `*-full-ci`
      and push it to the main repository or set the 'full-ci' label on the
      pull request.
      
      It is also possible to disable all tests on push by naming branch as
      `*-notest' or setting the 'notest' label on the pull request.
      
      **Caveats**:
       - Unfortunately, currently it doesn't seem to be possible to run
         workflows automatically when a particular label is set - the best we
         can do is run workflows when *any* label is set. So labeling a PR
         that has the 'full-ci' label set will trigger all workflows!
       - For the same reason, removing the 'notest' label doesn't trigger ci.
         One has to synchronize the PR afterwards. We could trigger ci on the
         'unlabel' event, but this would trigger tests when any label is
         removed, not necessarily 'notest'. Since 'notest' is supposed to be
         used only by developers, who can sync the branch, this should be
         acceptable.
      
      While we are at it:
       - Remove the check disabling certain workflow runs on forks - it's
         pointless, because forks don't have ci. Anyway, we don't bother
         disabling most of our workflows on forks, even those that we run on
         self-hosted machines, so that would only be consistent.
       - Remove the condition from the coverity workflow - coverity doesn't
         run on push or PR so it doesn't make any sense.
       - Remove the condition from the 'source' workflow. Instead trigger it
         only when a tag is pushed. This is needed to avoid showing it as a
         skipped workflow in PRs and commits.
      
      Closes #6605
      6c2e664e
    • Vladimir Davydov's avatar
      net.box: rewrite state machine (transport) in C · c13b3a31
      Vladimir Davydov authored
      Line-by-line (mostly) migration of the net.box state machine
      implementation from Lua to C. The C implementation uses the iostream
      abstration, which will allow us to to easily support complex network
      protocols under IPROTO in future.
      
      Closes #6291
      c13b3a31
    • Vladimir Davydov's avatar
      coio/socket: use same ai_flags in coio_connect and tcp_connect/server · 75364a71
      Vladimir Davydov authored
      Currently,
      
       - socket.tcp_server sets AI_PASSIVE.
       - socket.tcp_connect sets no flags, but passes 'tcp' for protocol.
       - coio_connect sets AI_ADDRCONFIG|AI_PASSIVE|AI_NUMERICSERV.
       - evio_service sets AI_ADDRCONFIG|AI_PASSIVE.
      
      This leads to different getaddrinfo results on some platforms (freebsd)
      and, as a result, test breakages (net.box_connect_timeout_gh-2054) after
      we switch net.box from socket.tcp_connect to coio_connect.
      
      Let's use compatible flags:
       - AI_ADDRCONFIG for clients (coio_connect and tcp_connect).
         (AI_PASSIVE doesn't make sense for connect.)
         (AI_NUMERICSERV isn't really needed in our case.)
       - AI_ADDRCONFIG|AI_PASSIVE for servers (evio_service, tcp_server).
      
      Let's also set ai_protocol to 'tcp' in both tcp_connect and tcp_server
      for consistency, because both these functions are only supposed to work
      with the tcp protocol, judging by their names.
      75364a71
    • Vladimir Davydov's avatar
      test: don't use net.box perform_request in tests · ae522950
      Vladimir Davydov authored
      It's too low-level. Use conn:_request instead.
      
      While we are at it, let's explicitly check the error in
      net.box_long-poll_input_gh-3400.test.lua to make it more
      robust.
      ae522950
    • Vladimir Davydov's avatar
      net.box: refactor netbox_communicate to be iostream-friendly · d50e80e3
      Vladimir Davydov authored
      We will use iostream in net.box when we convert the state machine to C.
      Rework netbox_communicate() so that it's easy to migrate to iostream.
      d50e80e3
    • Vladimir Davydov's avatar
      net.box: never omit password in netbox_encode_auth · bc03a2c7
      Vladimir Davydov authored
      xrow_decode_auth() expects the password to be set so omitting it while
      preparing a request is incorrect. Net.box state machine (in Lua) sets
      the password to an empty string if it's nil so the code handling NULL
      password in netbox_encode_auth() is actually dead. Let's remove it.
      bc03a2c7
    • Vladimir Davydov's avatar
      net.box: keep strong reference to callback in connection object · 1ab6d80d
      Vladimir Davydov authored
      The callback passed to the worker fiber (in create_transport())
      references the connection object. Since a fiber is a GC root, it means
      that the worker fiber pins the connection object. To avoid this, we use
      a weak callback reference. To prevent the callback from being GC-ed
      prematurely, we need to store a strong reference to it somewhere.
      Currently, we have some GC magic in create_transport(): we store a
      strong reference to the callback in the bound parameters of
      transport.stop(). This works fine, but when we migrate the transport
      implementation to C, we won't be able to override transport.stop() so
      let's store a strong reference in the connection object instead.
      
      While we are at it, let's drop create_transport method from net.box
      module exports, because it's not used anywhere in Tarantool code or
      tests, neither is it documented.
      1ab6d80d
    • Vladimir Davydov's avatar
      net.box: fix yield in connection GC callback · ce867aaa
      Vladimir Davydov authored
      A GC callback must not yield.
      
      Fixes commit 72adeda6 ("net.box:
      implement connection closing without dropping requests");
      
      Closes #6617
      ce867aaa
    • Vladimir Davydov's avatar
      net.box: raise if on_event callback failed · 4978f474
      Vladimir Davydov authored
      The callback isn't supposed to normally raise exceptions (it runs client
      code in pcall). If something abnormal happens, like a memory allocation
      error, better break the loop and stop the state machine.
      
      Also, raise an exception if we failed to decode an IPROTO_EVENT packet -
      this means that something is wrong with the other end and we better stop
      the connection (this is what we do if we fail to decode other packets).
      4978f474
    • Vladimir Davydov's avatar
      net.box: pass all state machine errors in exception · ab2e085d
      Vladimir Davydov authored
      Currently, some errors raise exception (e.g. memory allocation, fiber is
      cancelled) while others are handled by the state machine code. Let's
      pass all errors in exceptions and drop error_sm. This simplifies the
      state machine code a little bit.
      ab2e085d
    • Vladimir Davydov's avatar
      net.box: drop transport.wait_state method · 69902512
      Vladimir Davydov authored
      We can implement it in Lua, because the transport invokes the callback
      whenever the state changes. This will simplify conversion of the state
      machine to C.
      69902512
    • Vladimir Davydov's avatar
      net.box: drop id callback · afbaa87a
      Vladimir Davydov authored
      Pass protocol version and features to handshake callback along with
      greeting. The fewer callbacks we have, the easiser it'll be to convert
      the code to C.
      
      While we are at it, add a missing comment for the event callback.
      afbaa87a
    • Vladimir Davydov's avatar
      net.box: pass reconnect_after and connect_timeout to sm explicitly · fa29e8fe
      Vladimir Davydov authored
      There's no point in using the callback for this. It'd only complicate
      moving the state machine to C.
      fa29e8fe
    • Vladimir Davydov's avatar
      net.box: drop will_fetch_schema callback · f379bd1e
      Vladimir Davydov authored
      Not used since console support was dropped. Drop it to simplify
      migration of the net.box state machine to C.
      
      @TarantoolBot document
      Title: Update the net.box state machine diagram
      
      New diagram from the net.box source code:
      
      ```
      initial -> auth -> fetch_schema <-> active
      
      (any state, on error) -> error_reconnect -> auth -> ...
                                               \
                                                -> error
      (any state, but 'error') -> closed
      ```
      f379bd1e
    • Vladimir Davydov's avatar
      net.box: drop console support · 68837841
      Vladimir Davydov authored
      net.box console support was deprecated more than three years ago, in
      commit bd06e32a ("netbox: deprecate
      console support"). Let's drop it, because it'll greatly simply rewriting
      the net.box state machine in C.
      
      @TarantoolBot document
      Title: Delete documentation entry for net.box console
      
      net.box console support, which was deprecated in 1.10, was dropped.
      Now, one has to use `require('console').connect(host, port)` instead
      of `require('net.box').connect(host, port, {console = true})`.
      68837841
    • Vladimir Davydov's avatar
      net.box: drop establish_connection and wrap methods · 1bc6fef2
      Vladimir Davydov authored
      establish_connection is used by console to check greeting: if the
      protocol is 'Binary', it wraps the connected socket in a net.box
      connection; if it's 'Lua console', it wraps the socket in a console
      handler.
      
      Presence of these methods complicates migration of the net.box state
      machine to C. Also, it'll be difficult to support establish_connection
      when we switch net.box to the iostream abstraction, because a greeting
      should be received via iostream, not fd, probably after some
      protocol-specific negoitiation (e.g. TLS), while iostream isn't (and
      won't be) available in Lua and so can't be easily passed around from
      establish_connection to Lua and then back to net.box state machine.
      
      Since Lua console will always use plain sockets (it doesn't require
      encryption as it's supposed to run on the local host), let's first try
      to establish a net.box connection and if it fails because of unsupported
      protocol, try to establish a Lua console connection. It means that the
      socket will be connected twice for Lua console, which is suboptimal, but
      it should be fine, because it's not a hot path.
      1bc6fef2
  6. Nov 23, 2021
  7. Nov 22, 2021
    • Oleg Babin's avatar
      util: fix bloom test build on MacOS · 11e87877
      Oleg Babin authored
      Before this patch test build failed with following error:
      ```
      /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.0.sdk/usr/include/c++/v1/memory:2666:26: error: unexpected type name '_CompressedPair': expected expression
          struct _ALIGNAS_TYPE(_CompressedPair) _Storage {
      ```
      
      This patch seems to be fix this issue.
      The root of the problem is in "trivia/util.h" module. It defines
      alignas macros: `#define alignas(_n)
      __attribute__((aligned(_n)))`. And it seems causes some issues in
      stdlib internals. To fix this issue let's unconditionally include
      stdalign.h for C++. According [1] this header should internally
      define __alignas_is_defined macro.
      
      [1] https://en.cppreference.com/w/cpp/language/alignas
      
      Part of #6576
      11e87877
    • Mergen Imeev's avatar
      sql: define default types for built-in functions · 14aa6a59
      Mergen Imeev authored
      After this patch, all functions that take arguments of one of two or
      more types have defined a default type, which is used when the argument
      type cannot be determined.
      
      Closes #6483
      
      @TarantoolBot document
      Title: Default types of SQL built-in functions
      
      In case a function takes an argument of one of two or more types, this
      function has defined the default type for this argument. This type is
      used when the type of the argument cannot be determined, for example, in
      the case of a bound variable.
      
      Rules for determining default types as follows:
      1) When there is only one possible type, it is default.
      2) When possible types are INTEGER, DOUBLE or DECIMAL, DECIMAL is
         default.
      3) When possible types are STRING or VARBINARY, STRING is default.
      4) When possible data types are any other scalar data type, SCALAR is
         default.
      5) Otherwise, there is no default type.
      14aa6a59
Loading