Skip to content
Snippets Groups Projects
  1. Jan 19, 2024
    • Aleksandr Lyapunov's avatar
      box: loose truncate check in case of foreign key · cb0b51d6
      Aleksandr Lyapunov authored
      In #7309 a truncation of a space that was referenced by foreign
      key from some other space was prohibited.
      
      It appeared that this solution is too bothering since a user can't
      truncate a space even if he truncated referring space before that.
      
      Fix it by allowing space truncate if referring spaces are empty.
      Also allow drop of the primary index in the same case with the
      same reason: logically the index along with all space data is
      not needed for consistency if there's no referring data.
      
      Note that by design space truncate is implemented quite similar
      to space drop. Both delete all indexes, from secondary to primary.
      Since this patch allows deletion of the primary index (which is
      the action that actually deletes all data from the space), this
      patch changes the result of space drop too: the space remains
      alive with no indexes, while before this patch it remained alive
      with no secondary indexes but with present primary. In both cases
      the behaviour is quite strange and must be fixed in #4348. To
      make tests pass I had to perform drop in box.atomic manually.
      
      Closes #8946
      
      NO_DOC=bugfix
      
      (cherry picked from commit 983a7ec2)
      cb0b51d6
    • Vladimir Davydov's avatar
      test: increase max fiber slice for box/before_replace · b2bccf7c
      Vladimir Davydov authored
      The test fails on osx_debug quite frequently with the following error:
      
      NO_WRAP
      > box/before_replace.test.lua                                     [ fail ]
      >
      > Test failed! Result content mismatch:
      > --- box/before_replace.result	Mon Aug  7 10:36:06 2023
      > +++ /tmp/t/rejects/box/before_replace.reject	Mon Aug  7 10:42:03 2023
      > @@ -899,6 +899,7 @@
      >  ...
      >  for i = 1,17 do gen_inserts() end
      >  ---
      > +- error: fiber slice is exceeded
      >  ...
      >  test_run:cmd('restart server test')
      >  s = box.space.test_on_schema_init
      > @@ -906,7 +907,7 @@
      >  ...
      >  s:count()
      >  ---
      > -- 1
      > +- 0
      >  ...
      >  -- For this test the number of invocations of the before_replace
      >  -- trigger during recovery multiplied by the amount of return values
      > @@ -921,7 +922,8 @@
      >  -- the value is only increased by 1, and this change is visible only in memory.
      >  s:get{1}[2] == 1 + 16999 + 17000 + 1 or s:get{1}[2]
      >  ---
      > -- true
      > +- error: '[string "return s:get{1}[2] == 1 + 16999 + 17000 + 1 o..."]:1: attempt to
      > +    index a nil value'
      >  ...
      >  test_run:cmd('switch default')
      >  ---
      NO_WRAP
      
      Let's try to increase the max fiber slice up to 15 seconds to avoid
      that, like we do in other potentially long tests. (The default value
      is 1 second.)
      
      NO_DOC=test fix
      NO_CHANGELOG=test fix
      
      (cherry picked from commit 371d4110)
      b2bccf7c
    • Vladimir Davydov's avatar
      memtx: assert that space is not NULL in index_read_view_create · f05128f7
      Vladimir Davydov authored
      This should suppress the following coverity issues:
      
      https://scan7.scan.coverity.com/reports.htm#v39198/p13437/fileInstanceId=146712118&defectInstanceId=18978766&mergedDefectId=1563095
      https://scan7.scan.coverity.com/reports.htm#v39198/p13437/fileInstanceId=146712113&defectInstanceId=18978750&mergedDefectId=1563094
      
      While we are at it, let's use space_by_id instead of space_cache_find
      because read view creation is a rare operation affecting all spaces so
      caching the last space by id doesn't make any sense.
      
      NO_DOC=code health
      NO_TEST=code health
      NO_CHANGELOG=code health
      
      (cherry picked from commit f2886dd0)
      f05128f7
    • Igor Munkin's avatar
      asan: enable ASan and LSan support for LuaJIT back · d4486591
      Igor Munkin authored
      All LuaJIT related LSan warnings were suppressed in the scope of the
      commit 985548e4 ("asan: suppress all
      LSAN warnings related to LuaJIT"), since all compiler flags tweaks were
      enclosed in LuaJIT CMake machinery. As a result of the commit in LuaJIT
      submodule tarantool/luajit@a86e376 ("build: introduce LUAJIT_USE_ASAN
      option") ASan and LSan support has been finally added to LuaJIT runtime,
      so it was decided to remove LSan suppressions for LuaJIT functions.
      Unfortunately, it was not so easy as it looked like.
      
      At first, Lua global state is not closed properly at Tarantool instance
      exit (see <tarantool_free> in src/main.cc and <tarantool_lua_free> in
      src/lua/init.c for more info), so LSan false-positive leaks are detected
      (for more info, see #3071). Hence, the original LSan suppression for
      lj_BC_FUNCC is returned back (temporarily) until the aforementioned
      issue is not resolved.
      
      Furthermore, the internal LuaJIT memory allocator is not instrumented
      yet, so to find any memory faults it's worth building LuaJIT with system
      provided memory allocator (i.e. enable LUAJIT_USE_SYSMALLOC option).
      However, again, since Tarantool doesn't finalize Lua universe the right
      way, so running Tarantool testing routine with LUAJIT_USE_SYSMALLOC
      enabled generates false-positive LSan leaks. Return back here to enable
      LUAJIT_USE_SYSMALLOC, when #3071 is resolved.
      
      Last but not least, the default value of fiber stack size is 512Kb, but
      several tests in test/PUC-Rio-Lua-5.1-test suite in LuaJIT submodule
      (e.g. some cases with deep recursion in errors.lua or pm.lua) have
      already been tweaked according to the limitations mentioned in #5782,
      but the crashes still occur while running LuaJIT tests with ASan support
      enabled. Experiments once again confirm the notorious quote that "640 Kb
      ought to be enough for anybody".
      
      Anyway, LuaJIT tests are added to <test-release-asan> target in .test.mk
      and LUAJIT_TEST_ENV is extended with required ASan and LSan options.
      
      Follows up #5878
      
      NO_DOC=ci
      NO_TEST=ci
      NO_CHANGELOG=ci
      
      (cherry picked from commit bacf4e56)
      d4486591
    • Vladimir Davydov's avatar
      compat: move is_new and is_old option methods to metatable · 01270b2f
      Vladimir Davydov authored
      The is_new and is_old methods are the same for all compat options so
      they should be defined in a metatable. A good thing about this change
      is that it removes is_new and is_old from serialization:
      
      * Before:
      
      NO_WRAP
        tarantool> require('compat').yaml_pretty_multiline
        ---
        - is_new: 'function: 0x4175d6e8'
          is_old: 'function: 0x4175d790'
          brief: |
            Whether to encode in block scalar style all multiline strings or ones
            containing "\n\n" substring. The new behavior makes all multiline string output
            as single text block which is handier for the reader, but may be incompatible
            with some existing applications that rely on the old style.
      
            https://tarantool.io/compat/yaml_pretty_multiline
          current: default
          default: new
        ...
      NO_WRAP
      
      * After:
      
      NO_WRAP
        tarantool> require('compat').yaml_pretty_multiline
        ---
        - current: default
          brief: |
            Whether to encode in block scalar style all multiline strings or ones
            containing "\n\n" substring. The new behavior makes all multiline string output
            as single text block which is handier for the reader, but may be incompatible
            with some existing applications that rely on the old style.
      
            https://tarantool.io/compat/yaml_pretty_multiline
          default: new
        ...
      NO_WRAP
      
      To achieve that, we have to remove the option name from the usage error
      message but it seems to be okay because such errors shouldn't happen in
      practice and the error message is clear enough to figure out what went
      wrong.
      
      Follow-up #8807
      
      NO_DOC=refactoring
      NO_CHANGELOG=refactoring
      
      (cherry picked from commit 75b5fd05)
      01270b2f
    • Serge Petrenko's avatar
      applier: fix use after free · 66dc4e59
      Serge Petrenko authored
      Applier thread uses lsregion to allocate the messages for tx thread. The
      messages are freed upon return to the applier thread using a
      corresponding lsr_id.
      
      Due to a typo, one of the lsregion allocations was made with a postfix
      increment of lsr_id instead of the prefix one. Essentially, part of a
      new message was allocated with an old lsr_id, and might be freed early
      by a return of a previous message.
      
      Fix this.
      
      Closes #8848
      
      NO_DOC=bugfix
      NO_TEST=covered by asan in #8901
      NO_CHANGELOG=bugfix
      
      (cherry picked from commit 0d5bd6b7)
      66dc4e59
    • Nikita Zheleztsov's avatar
      relay: fix unterminated final_join thread · 040cd9e3
      Nikita Zheleztsov authored
      
      Currently if tarantool exits during relay's final join stage,
      corresponding thread isn't terminated. This causes the flakiness
      of the replicaset_ro_mostly.test.lua.
      
      Let's reuse the same relay, in which subscribe cord is running, for
      the final join stage. This way the cord will be cancelled during
      replication_free().
      
      Closes #8082
      
      NO_DOC=not user-visible
      NO_TEST=fix flaky test
      NO_CHANGELOG=not user-visible
      
      Co-authored-by: default avatarSergey Petrenko <sergepetrenko@tarantool.org>
      
      (cherry picked from commit 70a68836)
      040cd9e3
    • Vladimir Davydov's avatar
      git: add diff=cpp attribute for C/C++ source files · 6148e590
      Vladimir Davydov authored
      This improves diff hunk name detection. Needed for checkpatch to
      correctly detect if patched code belongs to a function.
      
      NO_DOC=git
      NO_TEST=git
      NO_CHANGELOG=git
      
      (cherry picked from commit 642584fd)
      6148e590
    • Mergen Imeev's avatar
      sql: fix wrong region allocation · 9fef276e
      Mergen Imeev authored
      This patch fixes an issue in generate_column_metadata(). Prior to this
      patch, the number of variable-only expressions was counted incorrectly
      when temporary memory was allocated on region to store their positions.
      However, although this allocation was incorrect, this did not lead to
      any problems due to the specifics of the region allocations.
      
      This patch fixes this by removing the temporary memory allocation.
      
      Closes #8763
      
      NO_DOC=no user-visible changes
      NO_TEST=no user-visible changes
      NO_CHANGELOG=no user-visible changes
      
      (cherry picked from commit d4f143ad)
      9fef276e
    • Georgy Moiseev's avatar
      msgpack: fix decoding intervals with int64 · aa48086c
      Georgy Moiseev authored
      It is possible for interval to have days, hours, minutes and seconds
      larger than INT_MAX (or less than INT_MIN). Before this patch, msgpack
      decoding had failed to parse intervals with msgpack int64 and uint64.
      int64_t should be enough to store any value allowed for datetime
      intervals.
      
      Closes #8887
      
      NO_DOC=small bug fix
      
      (cherry picked from commit 01c7ae11)
      aa48086c
    • Sergey Kaplun's avatar
      lua: replace `api_check()` with `assert()` · e5a18019
      Sergey Kaplun authored
      `api_check()` is the LuaJIT internal assertion. To prevent inconsistency
      during internal assertion changes (for example, during backporting), use
      glibc's `assert()` instead.
      
      NO_DOC=internal
      NO_TEST=internal
      NO_CHANGELOG=internal
      
      (cherry picked from commit e78afb3b)
      e5a18019
    • Igor Munkin's avatar
      luajit: bump new version · f259e74a
      Igor Munkin authored
      * FFI: Fix recording of union initialization.
      * Fix maxslots when recording BC_VARG, part 2.
      * Fix maxslots when recording BC_VARG.
      * Fix BC_UCLO insertion for returns.
      * ci: update job concurrency group definition
      
      Part of #8825
      
      NO_DOC=LuaJIT submodule bump
      NO_TEST=LuaJIT submodule bump
      f259e74a
    • Georgy Moiseev's avatar
      datetime: allow boundary values for interval.new · c369dd8f
      Georgy Moiseev authored
      Before this patch, one couldn't create new datetime interval with
      boundary value from Lua. At the same time, it was possible to create
      such interval from Lua through addition and subtraction. C range
      verification allow to create boundary value intervals, error message
      also implies that they should be allowed. (See #8878 for more info.)
      
      Closes #8878
      
      NO_DOC=small bug fix
      
      (cherry picked from commit b2a001cc)
      c369dd8f
    • Rimma Tolkacheva's avatar
      test/fuzz: refactor LuaJIT fuzzer · e08a0c7c
      Rimma Tolkacheva authored
      This refactoring will:
      
      1. Move macros from a header to the source file.
      Macros should be used in header only with undef to avoid redefinitions.
      Undef directive is not useful since we want to use these macros in the
      source file.
      
      2. Remove `using namespace lua_grammar` from header.
      https://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines#Rs-using-directive
      
      3. Moving serializer entry point and constant parameters into
      luajit_fuzzer namespace.
      It's a common practice in C++ to avoid name collisions.
      
      4. Move serializer functions into anonymous namespace.
      These functions are not a part of the interface so should have
      static linkage.
      https://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines#Rs-unnamed2
      
      5. Fix ConvertToStringDefault function.
      It was logically wrong so it would generate an identifier `123` from
      `*123`.
      
      NO_CHANGELOG=internal
      NO_DOC=fuzzer fix
      
      (cherry picked from commit 56488e15)
      e08a0c7c
    • klauwier's avatar
      test/fuzz: fix luaJIT fuzzer timeout · 89cbb737
      klauwier authored
      LuaJIT fuzzer used to stop due to timeout caused by infinite cycles and
      recursions. Counters were introduced for every cycle and function to
      address LuaJIT fuzzer timeouts.
      
      The idea is to add unique counters for every cycle and function to
      ensure finite code execution, if it wasn't already. For while, repeat,
      for cycles, local and global named, anonymous functions, counters will
      be initialized before the code generated from protobuf, and checked
      in the first body statement. An entry point for the serializer was
      created to count cycles and functions for counter initialization.
      
      The idea was taken from a paper "Program Reconditioning: Avoiding
      Undefined Behaviour When Finding and Reducing Compiler Bugs" [1].
      
      Here is an example of a change in serialized code made by this commit.
      
      Before:
      ```lua
      while (true) do
          foo = 'bar'
      end
      function bar()
          bar()
      end
      ```
      
      After:
      ```lua
      counter_0 = 0
      counter_1 = 0
      while (true) do
          if counter_0 > 5 then
              break
          end
          counter_0 = counter_0 + 1
          foo = 'bar'
      end
      function bar()
          if counter_1 > 5 then
              return
          end
          counter_1 = counter_1 + 1
          bar()
      end
      ```
      Protobuf structures that reproduce the timeout problem were added to
      the LuaJIT fuzzer corpus.
      
      [1] https://www.doc.ic.ac.uk/~afd/homepages/papers/pdfs/2023/PLDI.pdf
      
      NO_CHANGELOG=internal
      NO_DOC=fuzzer fix
      
      (cherry picked from commit 4d004bbe)
      89cbb737
    • klauwier's avatar
      test/fuzz: add breaks to switch-case · 53bed159
      klauwier authored
      Cases in two switches had no breaks, so they were falling
      through. Breaks were added to solve the problem. Code
      generated by the LuaJIT fuzzer became more various.
      
      NO_CHANGELOG=internal
      NO_DOC=fuzzer fix
      
      (cherry picked from commit 4430cac9)
      53bed159
    • Sergey Bronnikov's avatar
      cmake: propagate CMAKE_BUILD_TYPE for ProtobufMutator · 8cb37b89
      Sergey Bronnikov authored
      Follows-up #4823
      
      NO_CHANGELOG=internal
      NO_DOC=internal
      NO_TEST=internal
      
      (cherry picked from commit 95d62cfc)
      8cb37b89
    • Sergey Bronnikov's avatar
      test/fuzz: fix datetime_strptime fuzzing test · 9304de91
      Sergey Bronnikov authored
      Function `datetime_strptime` decodes string with datetime according to
      specified format, it accepts a datetime struct, buffer with datetime and
      string with format in arguments. Fuzzing test used static string
      "iso8601" as a format and it blocked fuzzing test to cover functions
      used by datetime_strptime under the hood. Fuzz introspector shows that
      code coveraged by a test is quite low.
      
      Patch updates the test to make it more effective: buffer with datetime
      and format string are generated using FDP (Fuzzing Data Provider).
      
      Test file extension was changed to .cc, because FuzzingDataProvider is
      used and we need building it by C++ compiler.
      
      Function `tnt_strptime` uses assert, that triggered by fuzzing tests.
      Therefore it was replaced with to if..then.
      
      1. https://storage.googleapis.com/oss-fuzz-introspector/tarantool/
      
      Fixes #8490
      
      NO_CHANGELOG=fuzzing test
      NO_DOC=fuzzing test
      NO_TEST=fuzzing test
      
      (cherry picked from commit a1bd6e0b)
      9304de91
    • Timur Safin's avatar
      datetime: fix buffer overflow in tnt_strptime · cbc92fe8
      Timur Safin authored
      Fixes #8502
      Needed for #8490
      
      NO_DOC=bugfix
      NO_TEST=covered by fuzzing test
      
      (cherry picked from commit 783a7040)
      cbc92fe8
    • Dmitriy Nesterov's avatar
      test/fuzz: add grammar-based LuaJIT fuzzer · 23d56ddf
      Dmitriy Nesterov authored
      Patch adds a LuaJIT fuzzer based on libprotobuf-mutator and LibFuzzer.
      Grammar is described via messages in protobuf format, serializer is
      applied to convert .proto format to string.
      
      For displaying generated code on the screen during fuzzing set
      the environment variable 'LPM_DUMP_NATIVE_INPUT'.
      
      For displaying error messages from lua functions set
      the environment variable 'LUA_FUZZER_VERBOSE'.
      
      Note: UndefinedBehaviourSanitizer is unsupported by LuaJIT (see #8473),
      so fuzzing test is disabled when CMake option ENABLE_UB_SANITIZER is
      passed.
      
      Closes #4823
      
      NO_DOC=<fuzzing testing of LuaJIT>
      NO_TEST=<fuzzing testing of LuaJIT>
      
      (cherry picked from commit a287c853)
      23d56ddf
    • Dmitriy Nesterov's avatar
      cmake: add dependencies for LuaJIT and SQL fuzzers · 07ee9a4f
      Dmitriy Nesterov authored
      Added Google's 'libprotobuf-mutator' and 'protobuf' libraries
      for developing grammar-based LuaJIT and SQL fuzzers based on
      LibFuzzer.
      
      It is needed to build protobuf module from source because
      by default, the system-installed version of protobuf is used
      by libprotobuf-mutator, and this version can be too old.
      
      Part of #4823
      
      NO_CHANGELOG=<dependencies>
      NO_DOC=<dependencies>
      NO_TEST=<dependencies>
      
      (cherry picked from commit b11072a6)
      07ee9a4f
    • Dmitriy Nesterov's avatar
      test/fuzz: add options for better fuzzing · 481d7c7a
      Dmitriy Nesterov authored
      Added options for fuzzing and for getting more information
      on debugging.
      
      NO_CHANGELOG=<fuzzing options>
      NO_DOC=<fuzzing options>
      NO_TEST=<fuzzing options>
      
      (cherry picked from commit 69f21e25)
      481d7c7a
    • Vladimir Davydov's avatar
      vinyl: fix use-after-free in vy_read_iterator_next · 0a7d6e7f
      Vladimir Davydov authored
      A read source iterator stores statements in a vy_history object using
      vy_history_append_stmt(). If a statement can be referenced, it's
      reference counter is incremented. If it can't, i.e. it belongs to a
      memory source, it's stored in a vy_history object without referencing.
      
      This works fine because memory sources are append-only. A problem arises
      only when we get to scanning disk sources. Since we yield while reading
      disk, a dump task may complete concurrently dropping the memory sources
      and possibly invalidating statements stored in the iterator history.
      Although we drop the history accumulated so far and restart the
      iteration from scratch in this case, there's still an issue that can
      result in a use-after-free bug in vy_read_iterator_next().
      
      The problem is that we access the current candidate for the next
      statement while evaluating a disk source after a disk read. If 'next'
      refers to a referenced statement, it's fine, but if it refers to a
      statement from a memory source, it may cause use-after-free because
      the memory source may be dropped during a disk read.
      
      To fix this issue, let's make vy_history_append_stmt() copy statements
      coming from memory sources. This should be fine performance-wise because
      we copied memory statements eventually in vy_history_apply() anyway,
      before returning them to the user.
      
      Note that we also have to update vy_read_iterator_restore_mem() because
      it implicitly relied on the fact that 'next' coming from a memory source
      can't be freed by vy_mem_iterator_restore(), which cleans up the memory
      source history. Now, it isn't true anymore so we have to temporarily
      take a reference to 'next' explicitly.
      
      Closes #8852
      
      NO_DOC=bug fix
      NO_TEST=tested by ASAN
      
      (cherry picked from commit 0e5a3cc2)
      0a7d6e7f
    • Igor Munkin's avatar
      cmake: introduce FIBER_STACK_SIZE option · a6a52d1b
      Igor Munkin authored
      In scope of the commit 82f4b4a3 ("lib/core/fiber: Increase default
      stack size") the default value of fiber stack size is increased up to
      512 Kb (you can find the reasons in the aforementioned commit message
      and in https://github.com/tarantool/tarantool/issues/3418 description).
      
      Some of the tests in test/PUC-Rio-Lua-5.1-test suite in LuaJIT repo
      (e.g. some cases with deep recursion in errors.lua or pm.lua) have
      already been tweaked according to the limitations mentioned in
      https://github.com/tarantool/tarantool/issues/5782, but the crashes
      still occurs while running LuaJIT tests with ASan support enabled.
      
      To make the testing routine more convenient, FIBER_STACK_SIZE option is
      introduced to Tarantool CMake machinery. One can provide the size either
      by raw digits (i.e. in bytes) or using Kb/Mb suffixes for convenience.
      
      A couple of important nits:
      * If the given value is not a multiple of 4Kb, CMake machinery adjusts
        it up to the nearest one greater than this value.
      * If the adjusted value is less than 512Kb, configuration fails with the
        corresponding CMake fatal error.
      
      Follows up #3418
      Relates to #5782
      
      @TarantoolBot document
      Title: introduce FIBER_STACK_SIZE configuration option
      
      To make managing of the default fiber stack size more convenient, the
      corresponding CMake option is added.
      
      **NB**: The stack size can't be less than 512Kb and if the given value
      is not a multiple of 4Kb, CMake machinery adjusts it up to the nearest
      one greater than this value.
      
      (cherry picked from commit ff57f990)
      a6a52d1b
    • Gleb Kashkin's avatar
      compat: fix err msg in compat.<option>:is_new() · 546e63d3
      Gleb Kashkin authored
      In the original commit 5f6d367c ("compat: add is_new and
      is_old to options") `compat.<option_name>:is_new()` and `:is_old()`
      were introduced, but by mistake they contained different usage
      messages. This patch updates `:is_new()` usage msg to more
      informative one from `:is_old()`.
      
      Follows up #8807
      
      NO_CHANGELOG=changelog from 5f6d367c is valid
      NO_DOC=doc from 5f6d367c is valid
      
      (cherry picked from commit f590ec22)
      546e63d3
    • Gleb Kashkin's avatar
      compat: add is_new and is_old to options · 05eb81e2
      Gleb Kashkin authored
      It used to be somewhat complicated to check the effective value
      of a compat option, because `<option_name>.current` could contain
      'default' state.
      This patch introduces helper functions that take care of that.
      
      The following alternatives were considered:
      * `compat.<option_name>.effective` - it is excessive in the presence
        if `current` and `default`, and is visible in serialization
      * `compat.<option_name>.get()` - while it is a function, it does only
        half of the work required, user still has to compare result to 'new'
      
      Closes #8807
      
      @TarantoolBot document
      Title: Add `:is_new/old()` helpers to tarantool.compat options
      
      `compat.<option_name>.current` can be 'new', 'old' or 'default',
      thus when it is default there must be an additional check if
      `compat.<option_name>.default` is 'new'. It is handier to have a
      helper to deal with that instead of complicated `if`:
      * check if effective value is 'new' before the patch:
        ```lua
        if compat.<option_name>.current == 'new' or
                (compat.<option_name>.current == 'default' and
                 compat.<option_name>.default == 'new') then
            ...
        end
        ```
      * after the patch:
        ```lua
        if compat.<option_name>:is_new() then
            ...
        end
        ```
      
      Please update [tutorial on using compat], maybe add an example to
      [Listing options details].
      
      [tutorial on using compat]: https://www.tarantool.io/en/doc/latest/reference/reference_lua/compat/compat_tutorial/
      [Listing options details]: https://www.tarantool.io/en/doc/latest/reference/reference_lua/compat/compat_tutorial/#listing-options-details
      
      (cherry picked from commit 5f6d367c)
      05eb81e2
    • Magomed Kostoev's avatar
      box: compare and hash msgpack value of double key field as double · 71f5992f
      Magomed Kostoev authored
      1. Make double-formatted fields accept integer and float values.
      2. Make indexes compare the values as double if the field key type
         is FIELD_TYPE_DOUBLE.
      3. Make hashers cast double key field to double before hashing, so
         we are able to insert and select any int, uint, float or double
         if their value casted to double is equal (for double keys).
      
      Notes about tuple_compare.cc:
      
      Since now `mp_compare_double` casts any value placed in field to
      double it was renamed to `mp_compare_as_double` to not semantically
      conflict with existing `mp_compare_double_*` functions.
      
      Notes about tuple_hash.cc:
      
      The hashee cast result is encoded in MP_DOUBLE and hashed for
      backward compatibility reasons.
      
      Since now the field hashing function (tuple_hash_field) requires
      field type to hash the field correctly, a new parameter has been
      introduced.
      
      By the way added assertions to the generic `field_hash` to prevent
      invalid hashing for new precompiled hashers and made
      `key_hash_slowpath` static cause it's only used in this file.
      
      Closes #7483
      Closes #5933
      Unblocks tarantool/crud#298
      
      @TarantoolBot document
      Title: It's not required to ffi-cast integral floating point to
      double anymore.
      
      The page describing tarantool data model states that:
      
      > In Lua, fields of the double type can only contain non-integer
      > numeric values...
      
      If the patch is merged this isn't the case anymore, so this
      statement and the code snippet below it should be updated.
      
      Link to the document: [Data storage](https://www.tarantool.io/en/doc/latest/concepts/data_model/value_store/#field-type-details).
      
      Affected segments:
      
      > double. The double field type exists mainly to be equivalent
      > to Tarantool/SQL’s DOUBLE data type. In msgpuck.h (Tarantool’s
      > interface to MsgPack), the storage type is MP_DOUBLE and the
      > size of the encoded value is always 9 bytes. In Lua, fields of
      > the double type can only contain non-integer numeric values and
      > cdata values with double floating-point numbers. Examples: 1.234,
      > -44, 1.447e+44.
      >
      > To avoid using the wrong kind of values inadvertently, use
      > ffi.cast() when searching or changing double fields. For example,
      > instead of space_object:insert{value} use ffi = require('ffi')
      > ... space_object:insert({ffi.cast('double',value)}). Example:
      >
      > ```
      > s = box.schema.space.create('s', {format = {{'d', 'double'}}})
      > s:create_index('ii')
      > s:insert({1.1})
      > ffi = require('ffi')
      > s:insert({ffi.cast('double', 1)})
      > s:insert({ffi.cast('double', tonumber('123'))})
      > s:select(1.1)
      > s:select({ffi.cast('double', 1)})
      > ```
      
      (cherry picked from commit 51af059c)
      71f5992f
    • Magomed Kostoev's avatar
      Bump msgpuck submodule · e8bbe9db
      Magomed Kostoev authored
      This update pulls the following commits:
      * Add mp_read_double_lossy without direct convertibility checks
      * Fix mp_read_double_lossy tests freebsd build
      
      These commits introduce a function required to compare and hash
      msgpack values of double key fields as double.
      
      Need for #7483, #5933
      
      NO_DOC=see the next commit
      NO_CHANGELOG=see the next commit
      
      (cherry picked from commit be47f8fb)
      e8bbe9db
    • Sergey Bronnikov's avatar
      test: fix tarantool process teardown · d3f7e18f
      Sergey Bronnikov authored
      Test uses a popen module that starts tarantool process in background
      mode. Tarantool process started in background mode forks a new process
      and closes a parent, after that popen loses a PID of the started process
      and `ph:kill()` and `ph:terminate()` doesn't work anymore. It leads to
      non-terminated tarantool processes after running the test.
      
      Patch fixes that by running `kill` using os.execute with a PID of
      tarantool process written to a pid file.
      
      Follows up #6128
      
      NO_CHANGELOG=fix test
      NO_DOC=fix test
      
      (cherry picked from commit 88686227)
      d3f7e18f
    • Ilya Grishnov's avatar
      box: fix shared lang between connected clients · ab3f317d
      Ilya Grishnov authored
      Fixed the implementation of the box console.
      Before this fix, result of `\set language` is shared between clients
      via `console.connect`, despite the fact that clients have different
      `box.session.id`. Now the parameter of the selected language is stored
      by each client in his own `box.session.storage`.
      
      Fixes #8817
      
      NO_DOC=bugfix
      
      (cherry picked from commit e4fda4b7)
      ab3f317d
    • Igor Munkin's avatar
      luajit: bump new version · c27cdcb2
      Igor Munkin authored
      * test: fix flaky <unit-jit-parse.test.lua>
      * Fix use-def analysis for vararg functions.
      * Fix use-def analysis for BC_VARG.
      * Fix TNEW load forwarding with instable types.
      * Fix memory probing allocator to check for valid end address, too.
      * Another fix for lua_yield() from C hook.
      * Fix lua_yield() from C hook.
      * Fix saved bytecode encapsulated in ELF objects.
      * x64: Fix 64 bit shift code generation.
      * Fix canonicalization of +-0.0 keys for IR_NEWREF.
      * test: add utility for parsing `jit.dump`
      * test: split utils.lua into several modules
      * test: rewrite lj-49-bad-lightuserdata test in C
      * test: rewrite misclib-sysprof-capi test in C
      * test: rewrite misclib-getmetrics-capi test in C
      * test: introduce utils.h helper for C tests
      * test: introduce module for C tests
      * test: fix setting of {DY}LD_LIBRARY_PATH variables
      * build: fix build with LUAJIT_USE_GDBJIT enabled
      * ci: update the branch name for Tarantool 2.11
      
      Closes #8718
      Part of #7900
      Part of #8516
      
      NO_DOC=LuaJIT submodule bump
      NO_TEST=LuaJIT submodule bump
      c27cdcb2
    • Sergey Bronnikov's avatar
      test: fix flakiness in gh_6128_background_mode_test · 2ce71928
      Sergey Bronnikov authored
      Previous attempt to fix flakiness in commit 6a2c73f8 ("test: fix
      flakiness in gh_6128_background_mode_test") used a constant buffer size
      in check_err_msg function. Tarantool 2.10 has a bit bigger log before a
      desired message that other versions of Tarantool and it leads to a this
      resulted in a truncated message ("entering the even" instead of
      "entering the event loop"). Patch replaces check_err_msg()
      implementation to grep_log used in luatest, it reads the whole log.
      
      Also patch renames check_err_msg to check_msg, because "entering the
      event loop" is not an error message.
      
      Follows up #6128
      
      NO_CHANGELOG=fix test
      NO_DOC=fix test
      
      (cherry picked from commit 1c8e7124)
      2ce71928
    • Magomed Kostoev's avatar
      box: check permissions on constraint functions on creation · 357629d2
      Magomed Kostoev authored
      Function execution permissions should only be checked on constraint
      creation.
      
      So when the function is used to check a tuple access rights don't
      have to be checked on each call for the current user.
      
      Closes #7873
      
      NO_DOC=bugfix
      
      (cherry picked from commit 6b8f2c5f)
      357629d2
    • Magomed Kostoev's avatar
      box: check permissions on functional index functions on creation · ba9698af
      Magomed Kostoev authored
      Function execution permissions should only be checked on functional
      index creation and on functional index function set.
      
      So when the function is used by key_list_iterator its rights don't
      have to be checked on each call for the current user.
      
      Part of #7873
      
      NO_DOC=bugfix
      NO_CHANGELOG=see the next commit
      
      (cherry picked from commit ddbdb77a)
      ba9698af
    • Georgiy Lebedev's avatar
      memtx: fix heap-use-after-free of tuple stories caused by space alter · 5070f252
      Georgiy Lebedev authored
      When a space is altered, we abort all in-progress transactions and delete
      all stories related to that space: the problem is we don't delete the
      stories' read gaps, which are also linked to the stories' transactions,
      which get cleaned up on transaction destruction — this, in turn, results in
      heap-use-after-free. To fix this, clean up stories' read gap in
      `memtx_on_space_delete` — we don't do this in `memtx_tx_story_delete` since
      it expects the story to not have any read gaps (see
      `memtx_tx_story_gc_step`).
      
      Tested this patch manually against Nick Shirokovskiy's experimental
      small-ASAN integration branch.
      
      Closes #8781
      
      NO_DOC=bugfix
      NO_TEST=<already covered by existing tests, but was not detectable by ASAN>
      
      (cherry picked from commit e1ed31bb)
      5070f252
    • Sergey Bronnikov's avatar
      test: fix flakiness in gh_6128_background_mode_test · 791ff79a
      Sergey Bronnikov authored
      Test runs an external process with tarantool that writes to a log file.
      Then test reads that log file and searches a string with required
      message in it (see function check_err_msg). Test was flaky on macOS and
      I suspect it was happening due to a high log level - timeout was not
      enough to wait message in the log file.
      
      Patch decreases a log level to a default value and replaces io
      functions with the similar alternatives in a fio module. Using
      fio functions allows to not block fibers.
      
      NO_CHANGELOG=test fix
      NO_DOC=test fix
      
      (cherry picked from commit 47380bb7)
      791ff79a
    • Vladimir Davydov's avatar
      lua/xlog: don't ignore unknown header fields · d1064aca
      Vladimir Davydov authored
      The xlog reader Lua module uses the xlog_cursor_next_row, which decodes
      the row header with xrow_header_decode. The latter silently ignores any
      unknown fields, which complicates catching bugs when garbage is written
      to a row header by mistake, for example, see #8783.
      
      Let's parse a row header without using xrow_header_decode in the xlog
      reader module, like we parse a row body, and output all unknown/invalid
      keys as is.
      
      To do that, we have to extend the xlog cursor API with the new method
      xlog_cursor_next_row_raw that returns a pointer to the position in the
      tx buffer where the next xrow is stored without advancing it. To avoid
      a memory leak in case the caller fails to parse an xrow returned by this
      function, we also have to move the call to xlog_tx_cursor_destroy from
      xlog_tx_cursor_next_row to xlog_cursor_next_tx.
      
      While we are at it,
       - Don't raise an error if a key type encountered in a row body is
         invalid (not an integer). Instead, silently ignore such keys.
       - Remove the useless body MsgPack validness check because we already
         check it after decoding the header.
       - Add error injection based tests to check all the corner cases.
      
      NO_DOC=bug fix
      
      (cherry picked from commit 8a25d170)
      d1064aca
    • Vladimir Davydov's avatar
      txn: reset stream_id row header field · e29782da
      Vladimir Davydov authored
      To avoid garbage written to xlog.
      
      Closes #8783
      
      NO_DOC=bug fix
      NO_TEST=next commit
      
      (cherry picked from commit f058cee7)
      e29782da
    • Ilya Verbin's avatar
      core: strip the PAC out of IP during backtrace on AArch64 macOS · bda1e6c3
      Ilya Verbin authored
      Apple's libunwind for AArch64 returns the Instruction Pointer with the
      Pointer Authentication Codes (bits 47-63) even though Tarantool is compiled
      for arm64 (not arm64e) architecture, so we have to strip them out [1].
      Although there is the ptrauth_strip macro for this purpose, it works only
      if compilation target is arm64e (not arm64) [2].
      
      1. https://developer.apple.com/documentation/security/preparing_your_app_to_work_with_pointer_authentication#3042105
      2. https://github.com/dotnet/runtime/issues/42955#issuecomment-886910180
      
      Closes #8074
      Closes tarantool/tarantool-qa#308
      Closes tarantool/tarantool-qa#309
      
      NO_DOC=bugfix
      
      (cherry picked from commit 88990e2f)
      bda1e6c3
    • Igor Munkin's avatar
      box: reduce luaT_pushtuple Lua GC memory usage · 42563c8d
      Igor Munkin authored
      
      <luaT_pushtuple> function created a new GCfunc object for a tuple __gc
      handler (i.e. <lbox_tuple_gc>), having no upvalues, on each call when
      tuple object for Lua world is created.
      
      The change introduces the reference to the single GCfunc object created
      on Tarantool startup and stored to Lua registry. Using this approach in
      scope of <luaT_pushtuple> is aimed to reduce Lua GC memory usage.
      
      Part of #5201
      
      NO_DOC=refactoring
      NO_TEST=refactoring
      NO_CHANGELOG=refactoring
      
      Reviewed-by: default avatarVladimir Davydov <vdavydov@tarantool.org>
      Reviewed-by: default avatarSergey Kaplun <skaplun@tarantool.org>
      Signed-off-by: default avatarIgor Munkin <imun@tarantool.org>
      (cherry picked from commit 8d7d1460)
      42563c8d
Loading