Skip to content
Snippets Groups Projects
  1. Oct 14, 2021
    • Timur Safin's avatar
      build, lua: built-in module datetime · 43e10ed3
      Timur Safin authored
      Introduce a new builtin Tarantool module `datetime.lua` for timestamp
      and interval types support.
      
      New third_party module - c-dt
      -----------------------------
      
      * Integrated chansen/c-dt parser as 3rd party module to the
        Tarantool cmake build process;
      * We use tarantool/c-dt instead of original chansen/c-dt to
        have an easier cmake build integration, as we have added some
        changes, which provide cmake support, and allow to rename symbols
        if necessary (this symbol renaming is similar to that we see
        with xxhash or icu).
      
      New built-in module `datetime`
      ------------------------------
      
      * created a new Tarantool built-in module `datetime`, which uses
        `struct datetime` data structure for keeping timestamp values;
      * Lua module uses a number of `dt_*` functions from `c-dt` library,
        but they were renamed to `tnt_dt_*` at the moment of exporting
        from executable - to avoid possible name clashes with external
        libraries.
      
      * At the moment we libc `strftime` for formatting of datetime
        values according to flags passed, i.e. `date:format('%FT%T%z')`
        will return something like '1970-01-01T00:00:00+0000', but
        `date:format('%A %d, %B %Y')` will return 'Thursday 01, January 1970'
      
      * if there is no format provided then we use default
        `tnt_datetime_to_string()` function, which converts datetime
        to their default ISO-8601 output format, i.e.
        `tostring(date)` will return string like "1970-01-01T00:00:00Z"
      
      * There are a number of simplified interfaces
        - totable() for exporting table with attributes names as provided
          by `os.date('*t')`
        - set() method provides unified interface to set values using
          the set of attributes as defined above in totable()
      
      Example,
      
      ```
      local dt = datetime.new {
      	nsec      = 123456789,
      
      	sec       = 19,
      	min       = 29,
      	hour      = 18,
      
      	day       = 20,
      	month     = 8,
      	year      = 2021,
      
      	tzoffset  = 180
      }
      
      local t = dt:totable()
      --[[
      {
      	sec = 19,
      	min = 29,
      	wday = 6,
      	day = 20,
      	nsec = 123456789,
      	isdst = false,
      	yday = 232,
      	tzoffset = 180,
      	month = 8,
      	year = 2021,
      	hour = 18
      }
      --]]
      
      dt:format()   -- 2021-08-21T14:53:34.032Z
      dt:format('%Y-%m-%dT%H:%M:%S')   -- 2021-08-21T14:53:34
      
      dt:set {
      	usec      = 123456,
      
      	sec       = 19,
      	min       = 29,
      	hour      = 18,
      
      	day       = 20,
      	month     = 8,
      	year      = 2021,
      
      	tzoffset  = 180,
      
      }
      dt:set {
      	timestamp = 1629476485.124,
      
      	tzoffset  = 180,
      }
      
      ```
      
      Coverage is
      
      File                 Hits Missed Coverage
      -----------------------------------------
      builtin/datetime.lua 299  23     92.86%
      -----------------------------------------
      Total                299  23     92.86%
      
      Part of #5941
      
      @TarantoolBot document
      Title: Introduced a new `datetime` module for timestamp and interval support
      
      Create `datetime` module for timestamp and interval types support.
      It allows to create date and timestamp values using either object interface,
      or via parsing of string values conforming to iso-8601 standard.
      One may manipulate (modify, subtract or add) timestamp and interval values.
      
      Please refer to https://hackmd.io/@Mons/S1Vfc_axK#Datetime-in-Tarantool
      for a more detailed description of module API.
      43e10ed3
  2. Sep 28, 2021
    • VitaliyaIoffe's avatar
      github-ci: use openssl@1.1 · 33830978
      VitaliyaIoffe authored
      OSX workflows use brew for install openssl.
      There was a new release of openssl@3.0 and
      homebrew updated the openssl formula.
      
      Close #6468
      33830978
  3. Sep 01, 2021
    • Kirill Yukhin's avatar
      build: build libcurl with PIC enabled · 01964d4c
      Kirill Yukhin authored
      To be able to configure libcurl for ARM64 CPU
      on Fedora distros with linker hardening it is
      necessary to enable `-fPIC`. It fails with wrong
      relocation type for glibc symbols (e.g. socket)
      otherwise.
      
      Closes #6366
      01964d4c
  4. Aug 18, 2021
    • Egor Elchinov's avatar
      build: disable backtraces on arm · 94925b53
      Egor Elchinov authored
      MacOS has its own libunwind which works not good with fibers
      under M1. Backtraces need to be enabled again after #6060 is fixed.
      Other arm backtraces are temporarily disabled too for #6222.
      
      Closes #6272
      94925b53
  5. Jun 22, 2021
    • Oleg Babin's avatar
      build: cleanup BuildZSTD.cmake · 2e854223
      Oleg Babin authored
      Seems CMakeLists.txt contains the same line as inside
      BuildZSTD.cmake. Seems we don't need to duplicate this logic
      twice let's remove it.
      2e854223
  6. Jun 18, 2021
    • Oleg Babin's avatar
      build: add libxxhash to third_party · ce382f96
      Oleg Babin authored
      This patch is the first step for fixing regression introduced in
      f998ea39 (digest: introduce FFI bindings for xxHash32/64).
      We used xxhash library that is shipped with zstd. However it's
      possible that user doesn't use bundled zstd. In such cases we
      couldn't export xxhash symbols and build failed with following
      error:
      
      ```
      [ 59%] Linking CXX executable tarantool
      /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/tarantool.dir/exports.c.o:(.data.rel+0xd80): undefined reference to `XXH32'
      /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/tarantool.dir/exports.c.o:(.data.rel+0xd88): undefined reference to `XXH32_copyState'
      /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/tarantool.dir/exports.c.o:(.data.rel+0xd90): undefined reference to `XXH32_digest'
      /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/tarantool.dir/exports.c.o:(.data.rel+0xd98): undefined reference to `XXH32_reset'
      /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/tarantool.dir/exports.c.o:(.data.rel+0xda0): undefined reference to `XXH32_update'
      /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/tarantool.dir/exports.c.o:(.data.rel+0xda8): undefined reference to `XXH64'
      /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/tarantool.dir/exports.c.o:(.data.rel+0xdb0): undefined reference to `XXH64_copyState'
      /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/tarantool.dir/exports.c.o:(.data.rel+0xdb8): undefined reference to `XXH64_digest'
      /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/tarantool.dir/exports.c.o:(.data.rel+0xdc0): undefined reference to `XXH64_reset'
      /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/tarantool.dir/exports.c.o:(.data.rel+0xdc8): undefined reference to `XXH64_update'
      collect2: error: ld returned 1 exit status
      ```
      
      To avoid a problem this patch introduces standalone xxhash library
      that will be bundled anyway. It's worth to mention that our
      approach is still related to zstd. We use Cyan4973/xxHash that is
      used in zstd and passes the same compile flags to it. Single
      difference is usage of XXH_NAMESPACE to avoid symbols clashing
      with zstd.
      
      Need for #6135
      ce382f96
  7. Jun 12, 2021
  8. Apr 30, 2021
  9. Apr 14, 2021
  10. Apr 13, 2021
    • Alexander Turenko's avatar
      build: fix configuring using cmake3 command · 820d2be6
      Alexander Turenko authored
      `cmake` command was hardcoded for configuring libcurl, however only
      `cmake3` may be installed in a system. Now we use the same cmake command
      for configuring libcurl as one that is used for configuring tarantool
      itself.
      
      The problem exists since 2.6.0-196-g2b0760192 ('build: enable cmake in
      curl build').
      
      Fixes #5955
      820d2be6
  11. Mar 17, 2021
    • Sergey Kaplun's avatar
      luajit: bump new version · a0ade206
      Sergey Kaplun authored
      LuaJIT submodule is bumped to introduce the following changes:
      * test: disable LuaJIT CLI tests in lua-Harness suite
      * test: set USERNAME env var for lua-Harness suite
      * test: adjust lua-Harness tests that use dofile
      * test: adjust lua-Harness suite to CMake machinery
      * test: add lua-Harness test suite
      
      Within this changeset lua-Harness suite[1] is added to Tarantool
      testing. Considering Tarantool specific changes in runtime the suite
      itself is adjusted in LuaJIT submodule.
      
      However, Tarantool provides and unconditionally loads TAP module
      conflicting with the one used in the new suite. Hence, the Tarantool
      built-in module is "unloaded" in test/luajit-test-init.lua.
      
      Furthermore, Tarantool provides UTF-8 support via another built-in
      module. Its interfaces differ from the ones implemented in Lua5.3 and
      moonjit. At the same time our LuaJIT fork provides no UTF-8 support, so
      lua-Harness UTF-8 detector is simply confused with non-nil utf8 global
      variable. As a result, utf8 is set to nil in test/luajit-test-init.lua.
      
      There are also some tests launching Lua interpreter, so strict need to
      be disabled for their child tests too. Hence `strict.off()` is added to
      `progname` (i.e. arg[-1] considering the way Tarantool parses its CLI
      arguments) command used in these tests.
      
      [1]: https://framagit.org/fperrad/lua-Harness/tree/a74be27/test_lua
      
      
      
      Closes #5844
      Part of #4473
      
      Reviewed-by: default avatarSergey Ostanevich <sergos@tarantool.org>
      Reviewed-by: default avatarIgor Munkin <imun@tarantool.org>
      Signed-off-by: default avatarIgor Munkin <imun@tarantool.org>
      Unverified
      a0ade206
  12. Mar 10, 2021
    • Igor Munkin's avatar
      luajit: bump new version · 1458a2b2
      Igor Munkin authored
      LuaJIT submodule is bumped to introduce the following changes:
      * test: adjust LuaJIT test suite for Tarantool
      * test: change LuaJIT test suite to match b4e6bf0
      * test: change LuaJIT test suite to match 5a61e1a
      * test: change LuaJIT test suite to match c198167
      * test: change LuaJIT test suite to match de5568e
      * test: add LuaJIT-test-cleanup test suite
      * test: fix Lua command in utils.selfrun
      * test: fix luacheck invocation for non-real paths
      
      Within this changeset LuaJIT-test-cleanup suite[1] is added to Tarantool
      testing. Considering Tarantool specific changes in runtime the suite
      itself is adjusted in LuaJIT submodule. However, there is <strict>
      module enabled by default in Debug build, so the testing environment
      need to be tweaked via test/luajit-test-init.lua script to be run prior
      to every LuaJIT suite test is started.
      
      [1]: https://github.com/LuaJIT/LuaJIT-test-cleanup/tree/014708b/test
      
      
      
      Closes #4064
      Part of #4473
      
      Reviewed-by: default avatarSergey Kaplun <skaplun@tarantool.org>
      Reviewed-by: default avatarSergey Ostanevich <sergos@tarantool.org>
      Signed-off-by: default avatarIgor Munkin <imun@tarantool.org>
      Unverified
      1458a2b2
  13. Mar 05, 2021
  14. Feb 28, 2021
    • Igor Munkin's avatar
      build: adjust LuaJIT build system · 07c83aab
      Igor Munkin authored
      LuaJIT submodule is bumped to introduce the following changes:
      * test: run luacheck static analysis via CMake
      * test: fix warnings found with luacheck in misclib*
      * test: run LuaJIT tests via CMake
      * build: replace GNU Make with CMake
      * build: preserve the original build system
      
      Since LuaJIT build system is ported to CMake in scope of the changeset
      mentioned above, the module building the LuaJIT bundled in Tarantool is
      completely reworked. There is no option to build Tarantool against
      another prebuilt LuaJIT due to a91962c0
      ('Until Bug#962848 is fixed, don't try to compile with external
      LuaJIT'), so all redundant options defining the libluajit to be used in
      Tarantool are dropped with the related auxiliary files.
      
      To run LuaJIT related tests or static analysis for Lua files within
      LuaJIT repository, <LuaJIT-test> and <LuaJIT-luacheck> targets are used
      respectively as a dependency of the corresponding Tarantool targets.
      
      As an additional dependency to run LuaJIT tests, prove[1] utility is
      required, so the necessary binary packages are added to the lists with
      build requirements.
      
      [1]: https://metacpan.org/pod/TAP::Harness#prove
      
      
      
      Closes #4862
      Closes #5470
      Closes #5631
      
      Reviewed-by: default avatarSergey Kaplun <skaplun@tarantool.org>
      Reviewed-by: default avatarTimur Safin <tsafin@tarantool.org>
      Signed-off-by: default avatarIgor Munkin <imun@tarantool.org>
      Unverified
      07c83aab
    • Igor Munkin's avatar
      build: fix lua.c file generation · 8484cabc
      Igor Munkin authored
      
      If Lua source path given to <lua_source> function is relative, the
      output file is generated in the binary directory. At the same time if
      the given path to be compiled to *.lua.c is absolute, the output
      file is generated in source directory instead of the binary one. This
      patch fixes the latter case, providing the valid behaviour for out of
      source build type.
      
      Needed for #4862
      
      Reviewed-by: default avatarSergey Kaplun <skaplun@tarantool.org>
      Reviewed-by: default avatarTimur Safin <tsafin@tarantool.org>
      Signed-off-by: default avatarIgor Munkin <imun@tarantool.org>
      Unverified
      8484cabc
  15. Jan 25, 2021
    • Alexander V. Tikhonov's avatar
      build: bump zstd submodule · c7094cd7
      Alexander V. Tikhonov authored
      Updated third_party/zstd submodule from v1.3.3 to v1.4.8 version.
      
      Found issue building on Fedora 33:
      
        third_party/zstd/lib/decompress/zstd_decompress.c: In function ‘ZSTD_findFrameCompressedSize’:
        third_party/zstd/lib/decompress/zstd_decompress.c:1502:18: error: ‘zfh.headerSize’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
         1502 |         ip += zfh.headerSize;
              |                  ^
      
      Also found that later releases of third_party/zstd submodule already
      fixed it. Decided to bump third_party/zstd submodule from v1.3.3 to
      v1.4.8. Added to zstd cmake build rules new files appeared on bumping.
      
      Found that some checking in static-build test exporting symbols like:
      
        ZSTD_free
        ZSTD_malloc
      
      never were public symbols and should not be tested for presence in the
      Tarantool executable, but also these symbols currently outdated and
      broke the testing. To avoid of it these symbols removed from test.
      
      Needed for #5502
      Closes #5697
      c7094cd7
  16. Dec 25, 2020
    • Sergey Bronnikov's avatar
      test: integrate with OSS Fuzz · 7680948f
      Sergey Bronnikov authored
      To run Tarantool fuzzers on OSS Fuzz infrastructure it is needed to pass
      library $LIB_FUZZING_ENGINE to linker and use external CFLAGS and
      CXXFLAGS. Full description how to integrate with OSS Fuzz is in [1] and
      [2].
      
      Patch to OSS Fuzz repository [2] is ready to merge.
      
      We need to pass options with "-fsanitize=fuzzer" two times
      (in cmake/profile.cmake and test/fuzz/CMakeLists.txt) because:
      
      - cmake/profile.cmake is for project source files,
        -fsanitize=fuzzer-no-link option allows to instrument project source
        files for fuzzing, but LibFuzzer will not replace main() in these
        files.
      
      - test/fuzz/CMakeLists.txt uses -fsanitize=fuzzer and not
        -fsanitize=fuzzer-no-link because we want to add automatically
        generated main() for each fuzzer.
      
      1. https://google.github.io/oss-fuzz/getting-started/new-project-guide/
      2. https://google.github.io/oss-fuzz/advanced-topics/ideal-integration/
      3. https://github.com/google/oss-fuzz/pull/4723
      
      Closes #1809
      7680948f
    • Sergey Bronnikov's avatar
      test: add fuzzers and support for fuzzing testing · 2ad7caca
      Sergey Bronnikov authored
      There is a number of bugs related to parsing and encoding/decoding data.
      Examples:
      
      - csv: #2692, #4497, #2692
      - uri: #585
      
      One of the effective method to find such issues is a fuzzing testing.
      Patch introduces a CMake flag to enable building fuzzers (ENABLE_FUZZER)
      and add fuzzers based on LibFuzzer [1] to csv, http_parser and uri
      modules. Note that fuzzers must return 0 exit code only, other exit
      codes are not supported [2].
      
      NOTE: LibFuzzer requires Clang compiler.
      
      1. https://llvm.org/docs/LibFuzzer.html
      2. http://llvm.org/docs/LibFuzzer.html#id22
      
      How-To Use:
      
      $ mkdir build && cd build
      $ cmake -DENABLE_FUZZER=ON \
      	-DENABLE_ASAN=ON \
      	-DCMAKE_BUILD_TYPE=Debug \
      	-DCMAKE_C_COMPILER="/usr/bin/clang" \
      	-DCMAKE_CXX_COMPILER="/usr/bin/clang++" ..
      $ make -j
      $ ./test/fuzz/csv_fuzzer -workers=4 ../test/static/corpus/csv
      
      Part of #1809
      2ad7caca
  17. Oct 22, 2020
    • Sergey Kaplun's avatar
      build: provide missing LuaJIT lmisclib.h header · 08bde945
      Sergey Kaplun authored
      Since LuaJIT provides extended LuaC API introduced in the scope of
      5a61e1ab54b5c66bfebd836db1ac47996611e065 ('misc: add C and Lua API for
      platform metrics') corresponding header should be provided along with
      other Tarantool development files.
      
      Follows up #5187
      08bde945
  18. Oct 16, 2020
    • Alexander V. Tikhonov's avatar
      build: enable cmake in curl build · 2b076019
      Alexander V. Tikhonov authored
      Initially tried to change autoconf tools in Curl build to cmake and
      found the following build issue on:
      
        CentOS 6
        CentOS 7
        Debian 8
        Ubuntu 14.04
      
      Issue found:
      
        CMake Error at CMakeLists.txt:41 (cmake_minimum_required):
          CMake 3.0 or higher is required.  You are running version 2.8.12.2
      
      To fix the issue check is removed of the version from curl sources
      in Tarantool's third party '<root Tarantool sources>/third_party/curl':
      
        12af024bc85606b14ffc415413a7e86e6bbee7eb ('Enable curl build with old cmake')
      
      After this fix completely changed autoconf to cmake in curl build.
      Autoconf part was completely removed and code cleaned up for cmake.
      For curl cmake build all autoconf options were ported to cmake
      configuration call, check the accurate list of the change in [1].
      
      Also the following issues resolved:
      
      1. Found that CURL cmake configuration file:
      
           third_party/curl/lib/CMakeLists.txt
      
         has installation part for built libcurl.a library:
      
           install(TARGETS ${LIB_NAME}
             EXPORT ${TARGETS_EXPORT_NAME}
             ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR}
             LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}
      
         where it changes CMAKE_INSTALL_LIBDIR to appropriate name with
         suffix of the architecture, like:
      
           lib
           lib64
           x86_64
      
         Found that find_library routine from the file:
      
           cmake/FindLibCURL.cmake
      
         returns only 'lib' value and it breaks the building of the depends
         binaries. To avoid of it the CMAKE_INSTALL_LIBDIR option was set to
         cmake call:
      
           -DCMAKE_INSTALL_LIBDIR=lib
      
      2. Found issue with building on CentOS 6:
      
           Linking C executable curl
           build/ares/dest/lib/libcares.a(ares__timeval.c.o): In function `ares__tvnow':
           ares__timeval.c:(.text+0x15): undefined reference to `clock_gettime'
           collect2: error: ld returned 1 exit status
      
         It was fixed with added "-lrt" flag to CMAKE_C_FLAGS and
         CMAKE_CXX_FLAGS build flags, when cmake version is lower
         than 3.0 and RT library had needed function.
      
      3. Found issues with building Tarantool statically on Debian 9 and
         its package build on CentOS 8.
      
         Building statically got the issues with openssl linking, like [2]:
      
           static-build/openssl-prefix/lib/libcrypto.a(threads_pthread.o): In function `CRYPTO_THREAD_lock_new':
           threads_pthread.c:(.text+0x45): undefined reference to `pthread_rwlock_init'
      
         It happened because in openssl radically changed how threads-related
         things were handled before 1.1.0 version. It required the application
         to provide us with the locking functionality in form of callbacks.
         Since 1.1.0, these matters are entirely internal, so libcrypto
         requires the presence of the platform specific thread implementation
         of our choosing, which is pthread on everything.
      
         After '-pthread' added to C compile flags package build on CentOS 8
         failed with the issue [3]:
      
           /build/usr/src/debug/tarantool-2.6.0.141/third_party/curl/lib/warnless.c:101:4: error: #error "SIZEOF_CURL_OFF_T not defined"
            #  error "SIZEOF_CURL_OFF_T not defined"
               ^~~~~
           /build/usr/src/debug/tarantool-2.6.0.141/third_party/curl/lib/warnless.c: In function 'curlx_uztoso':
           /build/usr/src/debug/tarantool-2.6.0.141/third_party/curl/lib/warnless.c:192:40: error: 'CURL_MASK_SCOFFT' undeclared (first use in this function); did you mean 'CURL_MASK_SINT'?
            return (curl_off_t)(uznum & (size_t) CURL_MASK_SCOFFT);
                                                 ^~~~~~~~~~~~~~~~
                                                 CURL_MASK_SINT
      
         To avoid of the issue decided to use '-pthread' flag only for openssl
         when it had not this flag in openssl compilation.
      
      4. Found issue with static build using CentOS 7, where SSL cmake rule
         failed. Building the image got the issue:
      
            [  1%] Performing configure step for 'bundled-libcurl-project'
            CMake Warning at CMakeLists.txt:50 (message):
              the curl cmake build system is poorly maintained.  Be aware
      
            -- curl version=[7.66.0-DEV]
            -- Found c-ares: /tarantool/build/ares/dest/lib/libcares.a
            Found *nroff option: -- -man
            CMake Error at /usr/share/cmake/Modules/FindOpenSSL.cmake:278 (list):
              list GET given empty list
            Call Stack (most recent call first):
              CMakeLists.txt:347 (find_package)
      
         Root cause of the issue that Dockerfile uses globaly
         installed openSSL with:
      
           cmake ... -DOPENSSL_ROOT_DIR=/usr/local ...
      
         Its cmake build file:
      
           /usr/share/cmake/Modules/FindOpenSSL.cmake
      
         fails on parsing the SSL version:
      
         it has:
                REGEX "^#define[\t ]+OPENSSL_VERSION_NUMBER[\t ]+0x([0-9a-fA-F])+.*")
      
         but it should to use:
                REGEX "^#[\t ]*define[\t ]+OPENSSL_VERSION_NUMBER[\t ]+0x([0-9a-fA-F])+.*")
      
         Anyway we want to use the same OpenSSL library for libcurl, as is used
         for Tarantool itself. So the path to it set for cURL build:
      
           list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_MODULE_PATH=${PROJECT_SOURCE_DIR}/cmake")
      
      5. In CMake build CMAKE_USE_LIBSSH2 flag is enabled by default, while
         in autoconf --with-libssh2 was disabled by default. We need to switch
         CMAKE_USE_LIBSSH2 flag off with:
      
           list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_USE_LIBSSH2=OFF")
      
         to avoid of linking issues, like:
      
           ld: libssh2.c:(.text+0x4d8): undefined reference to `libssh2_*...
      
         this issue exists in curl issues [4].
      
         Furthermore the following defaults are also disabled to keep the
         configuration consistent with autoconf one:
      
           list(APPEND LIBCURL_CMAKE_FLAGS "-DPICKY_COMPILER=OFF")
           list(APPEND LIBCURL_CMAKE_FLAGS "-DBUILD_CURL_EXE=OFF")
      
      Closes #4968
      Closes #5019
      Closes #5396
      
      [1] - https://github.com/tarantool/tarantool/issues/4968#issue-617183031
      [2] - https://gitlab.com/tarantool/tarantool/-/jobs/779176133#L6021
      [3] - https://gitlab.com/tarantool/tarantool/-/jobs/778309145#L3060
      [4] - https://github.com/curl/curl/issues/1146
      2b076019
    • Alexander V. Tikhonov's avatar
      build: generate bootstrap.h in CMAKE_BINARY_DIR · cca763ad
      Alexander V. Tikhonov authored
      Prior to these changes bootstrap.h was generated right in the source
      directory even for out of source build. Firstly such approach doesn't
      respect the idea of building outside the source files. Furthermore
      this leads to build failures when the source directory is located on
      read-only file system.
      
      As a result of the patch bootstrap.h is generated within the build
      tree and include directories are adjusted the corresponding way.
      
      Also changed destination build directory from source to binary path.
      
      Part of #4968
      cca763ad
  19. Sep 15, 2020
    • HustonMmmavr's avatar
      build: refactor static build process · 800e5ed6
      HustonMmmavr authored
      
      Refactored static build process to use static-build/CMakeLists.txt
      instead of Dockerfile.staticbuild (this allows to support static
      build on macOS). Following third-party dependencies for static build
      are installed via cmake `ExternalProject_Add`:
        - OpenSSL
        - Zlib
        - Ncurses
        - Readline
        - Unwind
        - ICU
      
      * Added support static build for macOS
      * Fixed `CONFIGURE_COMMAND` while building bundled libcurl for static
        build at file cmake/BuildLibCURL.cmake:
          - disable building shared libcurl libraries (by setting
            `--disable-shared` option)
          - disable hiding libcurl symbols (by setting
            `--disable-symbol-hiding` option)
          - prevent linking libcurl with system libz (by setting
            `--with-zlib=${FOUND_ZLIB_ROOT_DIR}` option)
      * Removed Dockerfile.staticbuild
      * Added new gitlab.ci jobs to test new style static build:
        - static_build_cmake_linux
        - static_build_cmake_osx_15
      * Removed static_docker_build gitlab.ci job
      
      Closes #5095
      
      Co-authored-by: default avatarYaroslav Dynnikov <yaroslav.dynnikov@gmail.com>
      800e5ed6
  20. Jul 08, 2020
  21. Jun 16, 2020
    • Vladislav Shpilevoy's avatar
      cmake: split UB sanitations into separate flags. · 5115d9f3
      Vladislav Shpilevoy authored
      Clang undefined behaviour sanitizer was turned on using
      -fsanitize=undefined flag, which is supposed to turn on all the
      sanitizations, except a few ones. Not needed sanitations were
      turned off explicitly, using -fno-sanitize=<type> flags. However
      appeared it does not work with some flags. For example,
      nullability sanitations can't be turned off when
      -fsanitize=undefined is used.
      
      Nullability sanitations lead to lots of false-positive fails
      such as typeof(*obj) where obj is NULL, or memcpy() with NULL
      destination but 0 size.
      
      The patch splits -fsanitize=undefined into separate flags and
      never turns on nullability checks.
      
      Part of #4609
      5115d9f3
  22. Jun 10, 2020
    • HustonMmmavr's avatar
      static build: fix build on ubuntu · 2e8fa27a
      HustonMmmavr authored
      Fixed static build with '-DBUILD_STATIC=ON' option:
      
      * Added cmake option CMAKE_DL_LIBS to icu library for
        test/unit tests binaries builds at file:
          cmake/FindICU.cmake
        due to fail:
          [ 84%] Linking CXX executable vy_point_lookup.test
          /usr/local/lib/libicuuc.a(putil.ao): In function `uprv_dl_open_62':
          putil.cpp:(.text+0x1a22): undefined reference to `dlopen'
          /usr/local/lib/libicuuc.a(putil.ao): In function `uprv_dlsym_func_62':
          putil.cpp:(.text+0x1a7d): undefined reference to `dlsym'
          /usr/local/lib/libicuuc.a(putil.ao): In function `uprv_dl_close_62':
          putil.cpp:(.text+0x1a61): undefined reference to `dlclose'
          collect2: error: ld returned 1 exit status
      
      * Added cmake option CMAKE_DL_LIBS to gomp library for
        test/unit tests binaries builds at file:
          cmake/BuildMisc.cmake
        due to fail:
          [ 91%] Linking CXX executable bps_tree.test
          /usr/lib/gcc/x86_64-linux-gnu/7/libgomp.a(target.o): In function `gomp_target_init':
          (.text+0x8b): undefined reference to `dlopen'
          (.text+0xa2): undefined reference to `dlsym'
          (.text+0xd9): undefined reference to `dlclose'
          (.text+0x29b): undefined reference to `dlsym'
          (.text+0x2a8): undefined reference to `dlerror'
          (.text+0x2d0): undefined reference to `dlsym'
          (.text+0x2e9): undefined reference to `dlsym'
          (.text+0x300): undefined reference to `dlsym'
          (.text+0x317): undefined reference to `dlsym'
          (.text+0x330): undefined reference to `dlsym'
          /usr/lib/gcc/x86_64-linux-gnu/7/libgomp.a(target.o):(.text+0x34d): more undefined references to `dlsym' follow
          /usr/lib/gcc/x86_64-linux-gnu/7/libgomp.a(target.o): In function `gomp_target_init':
          (.text+0x9cc): undefined reference to `dlerror'
          collect2: error: ld returned 1 exit status
      
      Close #5024
      2e8fa27a
  23. Jun 09, 2020
    • Vladislav Shpilevoy's avatar
      cmake: enable misc types of UB detection in clang · 325b678f
      Vladislav Shpilevoy authored
      Option ENABLE_UB_SANITIZER enables clang undefined behaviour
      sanitizer. So far the only UB to detect was alignment violation.
      This was the biggest problem found by the sanitizer. Now when it
      is fixed, most of the other types of UB are also turned on to fix
      them as well.
      
      There is a few of exceptions - pointer type overflow, vptr check,
      and all types of integer overflow and truncation.
      
      Pointer type overflow detection is disabled because it is abused
      in the source code a lot, by stailq data structure.
      
      Vptr sanitation is a runtime check ensuring that a pointer at a
      non-POD type really points at an object of this type, using RTTI.
      The check false-positively fails in alter.cc when AlterSpaceOp
      class objects are stored in an rlist, and the list is iterated
      using rlist_foreach_entry(). In the cycle there is a condition:
      
          &item->member != head
      
      In the end the 'item' points not at an AlterSpaceOp, but at the
      rlist head - offsetof(typeof(item), member), at an rlist
      structure. Despite 'item' is never dereferenced, clang anyway
      generates vptr check here, which of course fails. Note,
      '&item->member' does not dereference item. It is
      item + offsetof(typeof(item), member). Just another address a few
      bytes after item.
      
      Integer overflow and truncation are disabled because SQL uses
      int64_t variables as a container of values of range [INT64_MIN,
      UINT64_MAX]. This works because there is a flag 'is_neg' near
      each such value which tells how to interpret it - as negative
      int64_t, or as positive uint64_t. As a result, some operations
      lead to a false-positive overflow. For example, consider
      expr_code_int() function. It essentially can do this:
      
          int64_t value;
          ((uint64_t *)&value) = 9223372036854775808;
          value = -value;
      
      9223372036854775808 is -INT64_MIN. It can't be stored in int64_t.
      But the thing is that (uint64_t)9223372036854775808 is stored
      exactly like (int64_t)INT64_MIN, in binary form. So the expression
      "value = -value" looks perfectly valid:
      "value = -9223372036854775808", But in fact it is interpreted as
      "value = -(-9223372036854775808)".
      
      These integer overflow/truncation problems are going to be fixed
      in a separate commit due to big amount of changes needed for that.
      
      Part of #4609
      325b678f
  24. Jun 08, 2020
    • Vladislav Shpilevoy's avatar
      cmake: add option ENABLE_UB_SANITIZER · 366cb668
      Vladislav Shpilevoy authored
      Clang has a built-in sanitizer for undefined behaviour. Such as
      wrong memory alignment, array boundaries violation, 0 division,
      bool values with non standard content, etc.
      
      The sanitizer emits runtime checks which lead to either crash, or
      a trap, or a warning print, depending on what is chosen.
      
      The patch makes it possible to turn the sanitizer on and catch
      UBs. The only supported UB so far is alignment check. Other types
      can be added gradually, along with fixing bugs which they find.
      
      The UB sanitizer is activated for ASAN builds in CI.
      
      Part of #4609
      366cb668
    • Vladislav Shpilevoy's avatar
      cmake: ignore warnings on alignof() and offsetof() · 635b5ad9
      Vladislav Shpilevoy authored
      Warning about invalid offsetof() (used on non-POD types) was set
      for g++, but wasn't for clang++.
      
      Warning about invalid alignof() (when expression is passed to it
      instead of a type) wasn't ignored, but is going to be very
      useful in upcoming unaligned memory access patches. That allows
      to write something like:
      
          struct some_long_type *object = region_aligned_alloc(
                  region, size, alignof(*object));
      
      This will work even if type of 'object' will change in future,
      and so it is safer. And shorter.
      
      Part of #4609
      635b5ad9
  25. May 20, 2020
  26. Apr 02, 2020
    • Alexander V. Tikhonov's avatar
      static build: create new build w/o dockerfile · fc55875e
      Alexander V. Tikhonov authored
      Fixed static build with '-DBUILD_STATIC=ON' option:
      
       - installed liblzma-dev library for libunwind static, due to found that
         static libunwind library uses undefined lzma functions:
           nm -a /usr/lib/x86_64-linux-gnu/libunwind-x86_64.a | grep lzma
                       U lzma_index_buffer_decode
                       U lzma_index_end
                       U lzma_index_size
                       U lzma_index_uncompressed_size
                       U lzma_stream_buffer_decode
                       U lzma_stream_footer_decode
         while dynamic libunwind correctly sees liblzma installed:
           ldd /usr/lib/x86_64-linux-gnu/libunwind-x86_64.so | grep lzma
             liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f8fd1c23000)
         so to fix it the static library of lzma was needed.
      
       - added lzma library to unwind library for Tarantool build at file:
           cmake/compiler.cmake
         due to fail:
           /usr/lib/x86_64-linux-gnu/libunwind-x86_64.a(elf64.o):
             In function `xz_uncompressed_size':
           ./src/elfxx.c:194: undefined reference to `lzma_stream_footer_decode'
           ./src/elfxx.c:201: undefined reference to `lzma_index_buffer_decode'
           ./src/elfxx.c:205: undefined reference to `lzma_index_size'
           ./src/elfxx.c:210: undefined reference to `lzma_index_end'
           ./src/elfxx.c:207: undefined reference to `lzma_index_uncompressed_size'
           ./src/elfxx.c:210: undefined reference to `lzma_index_end'
           /usr/lib/x86_64-linux-gnu/libunwind-x86_64.a(elf64.o):
             In function `_Uelf64_extract_minidebuginfo':
           ./src/elfxx.c:278: undefined reference to `lzma_stream_buffer_decode'
           collect2: error: ld returned 1 exit status
           test/unit/CMakeFiles/luaL_iterator.test.dir/build.make:134:
             recipe for target 'test/unit/luaL_iterator.test' failed
           make[2]: *** [test/unit/luaL_iterator.test] Error 1
      
       - added dl library to gomp library for test/unit tests
         binaries builds at file:
           cmake/BuildMisc.cmake
         due to fail:
           /usr/lib/gcc/x86_64-linux-gnu/7/libgomp.a(target.o):(.text+0x34d):
             more undefined references to `dlsym' follow
           /usr/lib/gcc/x86_64-linux-gnu/7/libgomp.a(target.o): In function
             `gomp_target_init':
           (.text+0x9cc): undefined reference to `dlerror'
           collect2: error: ld returned 1 exit status
      
        - added dl library to icu library for test/unit tests
         binaries builds at file:
           cmake/FindICU.cmake
         due to fail:
           /usr/x86_64-linux-gnu/libicuuc.a(putil.ao):
             In function `uprv_dl_open_60':
           (.text+0x1ce2): undefined reference to `dlopen'
           /usr/x86_64-linux-gnu/libicuuc.a(putil.ao):
             In function `uprv_dlsym_func_60':
           (.text+0x1d3d): undefined reference to `dlsym'
           /usr/x86_64-linux-gnu/libicuuc.a(putil.ao):
             In function `uprv_dl_close_60':
           (.text+0x1d21): undefined reference to `dlclose'
           collect2: error: ld returned 1 exit status
      
      Added static build to gitlab-ci in release check criteria named
      as static_build job. Previously named static_build job renamed to
      static_docker_build, due to it checks the build at Dockerfile.
      
      Also moved static build make targets from .gitlab.mk to .travis.mk
      to store it in common place with the other test/build make targets.
      Moved environement from .gitlab-ci.yml file into make targets to
      make this targets true building in static w/o additional setup.
      
      Close #4551
      fc55875e
    • Alexander V. Tikhonov's avatar
      gitlab-ci: add memory leaks detection via LSAN · e737400d
      Alexander V. Tikhonov authored
      The change enables memory leaks detection to existing ASAN testing
      routine and introduces suppression files with the corresponding
      exception list:
       * address sanitizer for compile-time: asan/asan.supp
       * memory leak sanitizer for run-time: asan/lsan.supp
      
      Furthermore, added engine and replication suites for ASAN testing
      routine.
      
      Additionally to the tests blacklisted within #4359,
      'box/on_shutdown.test.lua' is also disabled since it fails the
      introduced leak check. All blacklisted tests have to be enabled
      within #4360.
      
      Close #2058
      e737400d
  27. Mar 23, 2020
    • Timur Safin's avatar
      evio: workaround for wsl1 so_linger assertion · 734bcafc
      Timur Safin authored
      
      SO_LINGER makes no much sense for unix-sockets, and Microsoft WSL
      is returning EINVAL if setsockopts called for SO_LINGER over unix
      sockets:
      
        [004] 2020-03-11 18:42:29.592 [29182] main/102/app sio.c:169 !> SystemError setsockopt(SO_LINGER), called on fd 16, aka
        [004] 2020-03-11 18:42:29.592 [29182] main/102/app F> can't initialize storage: setsockopt(SO_LINGER), called on fd 16,
        [004] 2020-03-11 18:42:29.592 [29182] main/102/app F> can't initialize storage: setsockopt(SO_LINGER), called on fd 16,
      
      And it's sort of correct here, but the problem is Linux is simply
      silently ignoring it, which passes tests.
      
      After much debates we decided to work-around this case via CMAKE
      define.
      
      NB! In a future (April/May 2020), when WSL2 with full Linux kernel
      would be released we should disable this check.
      
      Acked-by: default avatarCyrill Gorcunov <gorcunov@gmail.com>
      734bcafc
  28. Mar 17, 2020
    • Igor Munkin's avatar
      build: disable LUAJIT_ENABLE_PAIRSMM · 7ec7ced6
      Igor Munkin authored
      
      Since this build flag has been removed as a result of reverting the
      tarantool/luajit@d4e985a, its definition in the corresponding Tarantool
      cmake file is irrelevant.
      
      Furthermore, considering the breakage faced in #4770 the following tests
      are introduced:
      * the check whether space __pairs metamethod is set to space.pairs to
        create a Lua Fun iterator that handles __pairs manually underneath.
      * the check whether pairs builtin behaviour doesn't change when __pairs
        is set e.g. on space object.
      
      Follow-up #4560
      Closes #4770
      
      Reviewed-by: default avatarVladislav Shpilevoy <v.shpilevoy@tarantool.org>
      Signed-off-by: default avatarIgor Munkin <imun@tarantool.org>
      7ec7ced6
  29. Mar 05, 2020
    • Serge Petrenko's avatar
      build: link bundled libcurl with c-ares · 23837076
      Serge Petrenko authored
      libcurl has a built-in threaded resolver used for asynchronous DNS
      requests, however, when DNS server is slow to respond, the request still
      hangs tarantool until it is finished. The reason is that curl calls
      thread_join on the resolving thread internally upon timeout, making the
      calling thread hang until resolution has ended.
      Use c-ares as an asynchronous resolver instead to eliminate the problem.
      
      Closes #4591
      23837076
  30. Dec 05, 2019
  31. Nov 08, 2019
    • Alexander Turenko's avatar
      build: don't pass LDFLAGS from environment to curl · 0bead600
      Alexander Turenko authored
      
      After ea5929db ('build: fix OpenSSL
      linking problems on FreeBSD') we set CFLAGS explicitly (possibly to an
      empty value) when invoking a configure script for curl. When this
      parameter is set the script does not use a value of environment variable
      CFLAGS.
      
      Before this commit LDFLAGS environment variable can affect build of curl
      submodule. This can lead to a problem when a user or a tool set CFLAGS
      and LDFLAGS both and some linker flag assumes that some compilation flag
      is present. Here we set empty LDFLAGS explicitly to avoid using of the
      environment variable.
      
      A distributive build tool such as rpmbuild or emerge usually sets CFLAGS
      and LDFLAGS. The problem with incompatible compiler / linker options has
      been reveal under rpmbuild on CentOS 8 with hardened build enabled
      (which is so when backtraces are disabled).
      
      It is not clear whether we should follow environment variables or values
      determined by CMake for CFLAGS, CPPFLAGS and LDFLAGS when building a
      submodule (such as luajit and curl). Let's decide about this later.
      
      Part of #4543.
      
      Reviewed-by: default avatarAlexander V. Tikhonov <avtikhon@tarantool.org>
      Reviewed-by: default avatarIgor Munkin <imun@tarantool.org>
      Unverified
      0bead600
  32. Oct 28, 2019
    • Alexander Turenko's avatar
      build: pass path to toolchain for luajit and curl · 1eead75e
      Alexander Turenko authored
      This allows to overcome problems when CMake chooses one toolchain to
      build tarantool, but a library (libluajit.a or libcurl.a) is built using
      another (incompatible) toolchain.
      
      Fixes #4587.
      1eead75e
    • Alexander Turenko's avatar
      build: fix OpenSSL linking problems on FreeBSD · ea5929db
      Alexander Turenko authored
      FreeBSD has OpenSSL as part of the base system: libraries are located in
      /usr/lib, headers are in /usr/include. However a user may install the
      library into /usr/local/{lib,include} from ports / pkg. In this case
      tarantool did choose /usr/local version, while libcurl will pick up a
      base system library. This is fixed by passing --with-ssl option with an
      argument (/usr/local or /usr if custom -DOPENSSL_ROOT_DIR=<...> is not
      passed).
      
      Now the behaviour is the following. If -DOPENSSL_ROOT_DIR=<...> is
      passed, then try to use OpenSSL from it. Otherwise find the library in
      /usr/local and then in /usr. This is right as for tarantool's crypto
      module as well as for libcurl submodule.
      
      There is a flaw here: a user is unable to choose a base system library
      if a ports / pkg version of OpenSSL is installed. The reason here is
      that tarantool's crypto module depends on other libraries and
      -I/usr/local/include may be added to build options. I have no good
      solution for that, so `cmake . -DOPENSSL_ROOT_DIR=/usr` will give a
      warning on FreeBSD and `gmake` likely will fail if libraries are of
      different versions (see cmake/os.cmake comments for more information).
      See also a [discussion][1] in FreeBSD community about all those /usr and
      /usr/local problems.
      
      There were two other problems that may fail tarantool build on FreeBSD:
      they are fixed in this commit and described below.
      
      First, libcurl's configure script chooses GCC by default if it exists
      (say, installed from ports / pkg). It is unexpected behaviour when
      tarantool sources itself are built with clang. Now it is fixed by
      passing a compiler explicitly to the libcurl's configure script: the
      library will use base system clang by default or one that a user pass to
      tarantool's cmake.
      
      Side note: GCC has /usr/local/include in its default headers search
      paths; libcurl's configure script chooses GCC as a compiler and OpenSSL
      from a base system by default (when CC and --with-ssl=<...> are not set)
      that leads to OpenSSL header / library mismatch. It is the primary
      reason of the build fail that was fixed in
      1f2338bd ('build: FreeBSD packages
      installation'). It is not much relevant anymore, because we don't try to
      link with a base system OpenSSL if /usr/local one exists (however if it
      is asked explicitly with -DOPENSSL_ROOT_DIR=<...> we'll do, but will
      give a warning). Anyway, it is important to know such details if we'll
      change build scripts in a future.
      
      Second, backtraces are not supported on FreeBSD, but were enabled if
      libunwind headers is found. This leads to an error on cmake stage,
      because of inability to find a right library (this is a bug). Now we
      disable backtraces on FreeBSD by default even if libunwind is found. See
      
      When CC is passed to libcurl's configure script, the new problem opens
      on Mac OS. CMake chooses XCode toolchain by default (at least on a
      particular system where I tried it), which requires -isysroot=<SDK_PATH>
      option to be passed to a preprocessor and a compiler in order to find
      system headers. See [2] for more information.
      
      [1]: https://wiki.freebsd.org/WarnerLosh/UsrLocal
      [2]: https://developer.apple.com/documentation/xcode_release_notes/xcode_10_release_notes#3035623
      
      Follows up #4490.
      ea5929db
Loading