asan: enable ASan and LSan support for LuaJIT back

All LuaJIT related LSan warnings were suppressed in the scope of the
commit 985548e4 ("asan: suppress all
LSAN warnings related to LuaJIT"), since all compiler flags tweaks were
enclosed in LuaJIT CMake machinery. As a result of the commit in LuaJIT
submodule tarantool/luajit@a86e376 ("build: introduce LUAJIT_USE_ASAN
option") ASan and LSan support has been finally added to LuaJIT runtime,
so it was decided to remove LSan suppressions for LuaJIT functions.
Unfortunately, it was not so easy as it looked like.

At first, Lua global state is not closed properly at Tarantool instance
exit (see <tarantool_free> in src/main.cc and <tarantool_lua_free> in
src/lua/init.c for more info), so LSan false-positive leaks are detected
(for more info, see #3071). Hence, the original LSan suppression for
lj_BC_FUNCC is returned back (temporarily) until the aforementioned
issue is not resolved.

Furthermore, the internal LuaJIT memory allocator is not instrumented
yet, so to find any memory faults it's worth building LuaJIT with system
provided memory allocator (i.e. enable LUAJIT_USE_SYSMALLOC option).
However, again, since Tarantool doesn't finalize Lua universe the right
way, so running Tarantool testing routine with LUAJIT_USE_SYSMALLOC
enabled generates false-positive LSan leaks. Return back here to enable
LUAJIT_USE_SYSMALLOC, when #3071 is resolved.

Last but not least, the default value of fiber stack size is 512Kb, but
several tests in test/PUC-Rio-Lua-5.1-test suite in LuaJIT submodule
(e.g. some cases with deep recursion in errors.lua or pm.lua) have
already been tweaked according to the limitations mentioned in #5782,
but the crashes still occur while running LuaJIT tests with ASan support
enabled. Experiments once again confirm the notorious quote that "640 Kb
ought to be enough for anybody".

Anyway, LuaJIT tests are added to <test-release-asan> target in .test.mk
and LUAJIT_TEST_ENV is extended with required ASan and LSan options.

Follows up #5878

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

(cherry picked from commit bacf4e56)

.test.mk

@@ -70,10 +70,19 @@ test-release: build run-luajit-test run-test
 
 .PHONY: test-release-asan
 test-release-asan: CMAKE_ENV = CC=clang-11 CXX=clang++-11
+# FIBER_STACK_SIZE=640Kb: The default value of fiber stack size
+# is 512Kb, but several tests in test/PUC-Rio-Lua-5.1-test suite
+# in the LuaJIT repo (e.g. some cases with deep recursion in
+# errors.lua or pm.lua) have already been tweaked according to the
+# limitations mentioned in #5782, but the crashes still occur
+# while running LuaJIT tests with ASan support enabled.
+# Experiments once again confirm the notorious quote that "640 Kb
+# ought to be enough for anybody".
 test-release-asan: CMAKE_PARAMS = -DCMAKE_BUILD_TYPE=RelWithDebInfo \
                                   -DENABLE_WERROR=ON \
                                   -DENABLE_ASAN=ON \
                                   -DENABLE_UB_SANITIZER=ON \
+                                  -DFIBER_STACK_SIZE=640Kb \
                                   -DENABLE_FUZZER=ON
 # Some checks are temporary suppressed in the scope of the issue
 # https://github.com/tarantool/tarantool/issues/4360:
@@ -88,7 +97,15 @@ test-release-asan: TEST_RUN_ENV = ASAN=ON \
                                                detect_invalid_pointer_pairs=1:symbolize=1:$\
                                                detect_leaks=1:dump_instruction_bytes=1:$\
                                                print_suppressions=0
-test-release-asan: build run-test
+test-release-asan: LUAJIT_TEST_ENV = LSAN_OPTIONS=suppressions=${PWD}/asan/lsan.supp \
+                                     ASAN_OPTIONS=detect_invalid_pointer_pairs=1:$\
+                                                  detect_leaks=1:$\
+                                                  dump_instruction_bytes=1:$\
+                                                  heap_profile=0:$\
+                                                  print_suppressions=0:$\
+                                                  symbolize=1:$\
+                                                  unmap_shadow_on_exit=1
+test-release-asan: build run-luajit-test run-test
 
 # Debug build
 

asan/lsan.supp

@@ -31,12 +31,17 @@ leak:Curl_resolver_init
 # source: /usr/lib/x86_64-linux-gnu/gconv/UTF-16.so
 leak:gconv_init
 
-# XXX: All warnings reported for LuaJIT runtime are considered
-# false positive until sanitizers support is not introduced in
-# LuaJIT. See the issue below for more info.
-# https://github.com/tarantool/tarantool/issues/5878
-# source: third_party/luajit
-leak:lj_*
+# XXX: As a result of introducing ASan support in LuaJIT, in scope
+# of https://github.com/tarantool/tarantool/issues/5878 all lj_*
+# suppressions can be removed. Unfortunately, Lua global state
+# is not closed properly at Tarantool instance exit, so the
+# suppression for lj_BC_FUNCC is returned back (it was removed in
+# the scope of the commit 985548e4815a4cd73beb33acc60cf354e6eadd01
+# ("asan: suppress all LSAN warnings related to LuaJIT"). For more
+# info, see https://github.com/tarantool/tarantool/issues/3071.
+# source: src/main.cc (see <tarantool_free>)
+# source: src/lua/init.c (see <tarantool_lua_free>)
+leak:lj_BC_FUNCC
 
 # test: box/access.test.lua
 # test: box/access_bin.test.lua

cmake/luajit.cmake

@@ -64,13 +64,19 @@ if(ENABLE_VALGRIND)
         "Valgrind support" FORCE)
 endif()
 
-# FIXME: ASAN support is badly implemented in LuaJIT and there is
-# not a specific build options for this. At the same time there
-# are several places wrapped with LUAJIT_USE_ASAN define.
-# Just enable it here if needed and patiently wait until ASAN
-# support is implemented properly in LuaJIT.
+# Enable LuaJIT ASan support. The internal LuaJIT memory allocator
+# is not instrumented yet unfortunately, so to find any memory
+# faults it's worth building LuaJIT with system provided memory
+# allocator (i.e. enable LUAJIT_USE_SYSMALLOC option). However,
+# Tarantool doesn't finalize Lua universe the right way (see the
+# comments near <tarantool_lua_free>), so running Tarantool
+# testing routine with LUAJIT_USE_SYSMALLOC enabled generates
+# false-positive LSan leaks. Return back here to enable
+# LUAJIT_USE_SYSMALLOC, when the issue below is resolved.
+# https://github.com/tarantool/tarantool/issues/3071
 if(ENABLE_ASAN)
-    add_definitions(-DLUAJIT_USE_ASAN=1)
+    set(LUAJIT_USE_ASAN ON CACHE BOOL
+        "Build LuaJIT with AddressSanitizer" FORCE)
 endif()
 
 if(TARGET_OS_DARWIN AND NOT LUAJIT_ENABLE_GC64)