Alexey Protsenko authored 2 years ago and Дмитрий Кольцов committed 1 year ago

Add to .gitlab.ci.yml test_linux, test_debian_docker_luacheck, coverage
from .travis.mk. Also sign package on build
Add checkpatch linter.
Add docker image build. Image copies original tarantool/tarantool from
Dockerhub

NO_DOC=ci change
NO_TEST=ci change
NO_CHANGELOG=ci change

The ability to support backtraces is checked in cmake/compiler.cmake,
it makes no sense to duplicate the check in rpm/tarantool.spec. Also do
not enable backtraces unconditionally in apk/APKBUILD and static-build.

Part of #6998

NO_DOC=build
NO_TEST=build
NO_CHANGELOG=build

(cherry picked from commit f7c4a34a)

Part of tarantool/tt#286

NO_TEST=third-party dependency
NO_DOC=third-party dependency

(cherry picked from commit 10f7109c)

The dependency is redundant, because the library is now bundled into
tarantool (see PR #6877).

There is a practical reason to remove it: this way we'll verify that
presence of system libunwind headers is not necessary to build
tarantool. In other words, we'll verify that #8025 will not appear
again.

Follows up #8025
Follows up #6877
Part of #6998

NO_DOC=No user visible changes, just some validation for a build fix.
NO_TEST=See NO_DOC.
NO_CHANGELOG=See a changelog entry in the previous commit.

Support for OpenSUSE 15.1/15.2 has been recently dropped.
There is no sense to keep the dead code. So removing it.

Follows up #8021

NO_DOC=cd
NO_TEST=cd
NO_CHANGELOG=cd

Now we base on some unreleased state of libunwind. This is by itself not
a good practice. Yet the main motivation is that in the current version of
libunwind fast path for backtrace detection on x86_64 does not work.
I guess this is because of libunwind/libunwind@400b3f819ad4 (" x86_64: Stop
aliasing RSP and CFA"). See libunwind/libunwind#440. Fortunately this
commit is not present it the latest release.

Using fast or slow path has a great impact on performance of debug build
where collecting fiber gc allocation backtrace is turned on by default.
It surely depends on frequency and pattern of allocation. Test
sql/delete3 depends on backtrace performance most dramatically. On some
installations collecting backtraces slowed down the test up to 10 times.

If fast path is available then collecting backtrace does not affect
performance in a noticeable way.

I propose not to keep autotools products in the libunwind fork repo as
it is done previously. LOG_CONFIGURE is removed because it somehow
incompatible with using && in CONFIGURE_COMMAND command and not critical
to the build.

Also add autotools build dependencies for packpack package specs.
Currently not all packpack images have autotools preinstalled so this is
required for build to success. Anyway we'b better have precise build
requirements in build specs.

Follow-up #5665

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

Introduce cmake option ENABLE_HARDENING, which is TRUE by default for
non-debug regular and static builds, excluding AArch64 and FreeBSD.
It passess compiler flags that harden Tarantool (including the bundled
libraries) against memory corruption attacks. The following flags are
passed:

* -Wformat - Check calls to printf and scanf, etc., to make sure that
  the arguments supplied have types appropriate to the format string
  specified.

* -Wformat-security -Werror=format-security - Warn about uses of format
  functions that represent possible security problems. And make the
  warning into an error.

* -fstack-protector-strong - Emit extra code to check for buffer
  overflows, such as stack smashing attacks.

* -fPIC -pie - Generate position-independent code (PIC). It allows to
  take advantage of the Address Space Layout Randomization (ASLR).

* -z relro -z now - Resolve all dynamically linked functions at the
  beginning of the execution, and then make the GOT read-only.

Also do not disable hardening for Debian and RPM-based Linux distros.

Closes #5372
Closes #7536

NO_DOC=build
NO_TEST=build

Sometimes we need to disable testing while building deb/rpm packages to
speed up the build process. Now it is possible via `MAKE_CHECK` env var.
By default, testing is on, but if one defines `MAKE_CHECK=false`, tests
will be off.

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

This patch adds the tzdata package as a dependency for DEB/RPM tarantool
package since some tarantool datetime functionality needs this.

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

Now the test-run dependencies (pyyaml, gevent) have the corresponding
deb packages installable via the 'apt' package manager and finally it's
time to enable running tests in the package build process.

Closes #1341

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

This reverts commit 9d1f9f0e.

Follow-up #6477

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

In Fedora 36 the default version of the `openssl-devel` package is 3.
But for now Tarantool doesn't work with this version. So this patch
adds a workaround with installation of the `openssl1.1-devel` package
to build rpms and exporting env variable `OPENSSL_CONF=/dev/null` to
pass tests.

See for details: https://github.com/tarantool/tarantool/issues/6477.

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

This change contains the following improvements:

1. Prettify view in the job list.

   Before:
     - centos_7 / centos_7 (GC64=OFF)
     - centos_7 / centos_7 (GC64=ON)

   After:
     - centos_7 / centos_7
     - centos_7 / centos_7 (gc64)

2. Change possible values for the ${GC64} env variable to
   `true/false` instead of `ON/OFF` which is more traditional.

3. Use `-DLUAJIT_ENABLE_GC64=ON` flag in the `debian/rules` and
   `rpm/tarantool.spec` files directly instead of providing it
   in the workflow files.

Follows-up tarantool/tarantool-qa#159
Follows-up tarantool/tarantool-qa#161

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

It looks like the simplest solution to deliver gc64 tarantool builds
is to have a separate repo for it. So this patch removes old stuff
related to the 'tarantool-gc64' package and adds the new logic to
store gc64 packages in a separate repo.

Closes: tarantool/tarantool-qa#161

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

Co-authored-by: Yaroslav Lobankov <y.lobankov@tarantool.org>

To make tarantool-gc64 available in repository we need to provide
a product name in a package itself and in the repository.

To build Tarantool with gc64 enabled, the flag -DLUAJIT_ENABLE_GC64
needs to be passed to cmake flags. For this purposes GC64 env var
was added to all workflows for Debian, Ubuntu, Centos and Fedora.

This variable is processed to rename PRODUCT_NAME. If GC64 is
eanbled, PRODUCT_NAME is `tarantool-gc64`, otherwise it stays
`tarantool`, which goes to repository name and the package itself.
Thus, Tarantool with gc64 enabled can be installed by
`<packaen_manager> install -y tarantool-gc64`
in the one single repository.

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

Closes: tarantool/tarantool-qa#161

After fixing libunwind issues in 7dc9fe44, libcoro fibers issues in
761053f0 and c8ad49f0, we can now re-enable backtrace feature on
AARCH64.

For some not investigated reason `unw_backtrace` does not collect any
ips when "-mbranch-protection" compile feature is enabled: provide a
workaround for this case.

Closes #6060

NO_DOC=build system
NO_TEST=build system

Investigation of GNU libunwind problems on the aarch64-linux-gnu
platform drive us to the conclusion that libunwind-1.2.1 provided by
major distribution packages is broken. Not to mention that its test
suite fails with SEGFAULTs.

Last but not least, some distributions, e.g. CentOS 8 (see #4611) do
not provide a libunwind package.

Hence, bundle libunwind: bundling is enabled by default on all
platforms, except for macOS — a system package can be used if its
version is greater or equal than 1.3.0 (minimal version that does not
seem to be broken on aarch64-linux-gnu).

* Add new submodule: bump it to current master.
* Refactor libunwind package search logic out of compiler.cmake.
* Add CMake script for building bundled libunwind.
* Add CMake script for extracting version of libunwind.
* Re-enable backtrace for all RHEL distributions by default.
* Remove libunwind from static build.

Needed for #4002
Closes #4611

NO_DOC=build system
NO_TEST=build system

Before this patch Tarantool http client did not support HTTP/2.
The reasons to enable HTTP/2 support are efficiency offered by
the protocol and potential ability to interact with a GRPC server.
The CMake version requirement is updated from 3.2 to 3.3, because
we need generator expressions in cmake for setting multiple paths
in CMAKE_FIND_ROOT_PATH for nghttp2 support.

Closes #5771

@TarantoolBot document
Title: Now we require CMake 3.3 to build tarantool

In tarantool/doc#2065 we requested to update
the CMake minimum version to 3.2. Now it is time for 3.3.
See details in the linked commit.

Actualize changelog based on GitHub release pages changelogs. Some
entries were skipped since they are not relevant to modern packages.
Versions 1.8.1 and 2.0.4 changelogs were merged to 2.1.1 changelog.

Closes #6397

Except for backtrace related stuff, see gh-6060 for details.

This partially reverts commit 24c065d6.

Tests were turned off from arm workflows according to instability.

Needed for #6222

The reason of the update is to protect us against possible MITM attack
from a malicious HTTPS proxy server with trusted certificate when TLS
1.3 is used (CVE-2021-22890, [1]). libcurl versions prior to 7.76.0 can
skip a TLS handshake with a target host in this circumstances.

Other vulnerabilities fixed in the (7.71.1; 7.76.0] version range do not
look relevant to our built-in http client. See [2] for the full list.

The CMake version requirement is updated from 3.1 to 3.2, because curl's
CMakeLists.txt has the following clause at beginning:

 | cmake_minimum_required(VERSION 3.2...3.16 FATAL_ERROR)

(It was there in vanilla curl 7.71.1 too and we had to remove it in
order to support CMake 2. Now we don't support CMake 2, so it is good
time to get rid of the extra patch upward vanilla curl repository.)

According to the CMake versions table in
8a7702b1 ('github-ci: purge Debian
Jessie from CI'), CMake 3.2+ is available on all supported OSes.

[1]: https://curl.se/docs/CVE-2021-22890.html
[2]: https://curl.se/docs/vulnerabilities.html

@TarantoolBot document
Title: Now we require CMake 3.2 to build tarantool

In https://github.com/tarantool/doc/issues/1780 we requested to update
the CMake minimum version to 3.1. Now it is time for 3.2. See details in
the linked commit.

Please, update the 'Building from source' manual.

Ship libcurl headers to system path "${PREFIX}/include/tarantool"
in the case of libcurl included as bundled library or static
build. It is needed to use SMTP client with tarantool's libcurl
instead of system libcurl.

See related issue: https://github.com/tarantool/smtp/issues/24

Closes #4559

Fix package names with Python modules on Centos 7, OpenSUSE, Fedora.
All supported RPM-based distributives (Fedora 33, Fedora 32, OpenSUSE 15.2,
OpenSUSE 15.1, CentOS 8) contains python3, python3-six and python3-gevent
packages except CentOS 7 and CentOS 6.
We don't run tests when make packages for CentOS 6. For CentOS 7 we create
our own RPM packages for Python3 dependencies.

Patch unblocks regression testing on Fedora 33.

Part of #5652
Closes tarantool/tarantool-qa#17

LuaJIT submodule is bumped to introduce the following changes:
* test: run luacheck static analysis via CMake
* test: fix warnings found with luacheck in misclib*
* test: run LuaJIT tests via CMake
* build: replace GNU Make with CMake
* build: preserve the original build system

Since LuaJIT build system is ported to CMake in scope of the changeset
mentioned above, the module building the LuaJIT bundled in Tarantool is
completely reworked. There is no option to build Tarantool against
another prebuilt LuaJIT due to a91962c0
('Until Bug#962848 is fixed, don't try to compile with external
LuaJIT'), so all redundant options defining the libluajit to be used in
Tarantool are dropped with the related auxiliary files.

To run LuaJIT related tests or static analysis for Lua files within
LuaJIT repository, <LuaJIT-test> and <LuaJIT-luacheck> targets are used
respectively as a dependency of the corresponding Tarantool targets.

As an additional dependency to run LuaJIT tests, prove[1] utility is
required, so the necessary binary packages are added to the lists with
build requirements.

[1]: https://metacpan.org/pod/TAP::Harness#prove



Closes #4862
Closes #5470
Closes #5631

Reviewed-by: Sergey Kaplun <skaplun@tarantool.org>
Reviewed-by: Timur Safin <tsafin@tarantool.org>
Signed-off-by: Igor Munkin <imun@tarantool.org>

In scope of implementing self-sufficient testing environment LuaJIT
build system is partially ported to CMake. While integrating the new
build system with Tarantool it was found (on practice of course) that
using generator expressions[1] in target DEPENDS section is introduced
only in CMake 3.1[2]. This CMake feature is used to pass so called
"LuaJIT binary" to be used in LuaJIT tests, so one can run the same
testing machinery with both Tarantool and LuaJIT with no changes in it.

[1]: https://cmake.org/cmake/help/v3.1/manual/cmake-generator-expressions.7.html
[2]: https://cmake.org/cmake/help/latest/release/3.1.html#commands



Relates to #4862

Reviewed-by: Alexander V. Tikhonov <avtikhon@tarantool.org>
Signed-off-by: Igor Munkin <imun@tarantool.org>

@TarantoolBot document
Title: Update CMake minimum required version in build documentation

This commit updates the CMake minimum required version to be used in
Tarantool build infrastructure. However, one can build Tarantool from
sources and the required toolchain list should be updated. CMake minimum
required version is 3.1 since this commit. See the commit message for
more info.

Since Python 2 was deprecated then new OS do not have repositories
with its packages. It is not good way to create all these packages
manually and store it as backported. Let's wait when Python 3 will
be used for testing and for now the testing disabled for Fedora 33
packaging.

Also found that %cmake macros used in build spec was changed - new
option was added:

  -B x86_64-redhat-linux-gnu

It changed the build path and broke the build, to fix it these
options was additionally locally reset to:

  -B .

Found that LTO was set by default in Fedora 33, check

  /usr/lib/rpm/redhat/macros

    # LTO is the default in Fedora.
    #   "%define _lto_cflags %{nil}"  to opt out
    #
    # We currently have -ffat-lto-objects turned on out of an abundance of
    # caution.  To remove it we need to do a check of the installed .o/.a files
    # to verify they have real sections/symbols after LTO stripping.  That
    # way we can detect installing an unusable .o/.a file.  This is on the TODO
    # list for F34.
    %_gcc_lto_cflags        -flto=auto -ffat-lto-objects
    %_clang_lto_cflags      -flto
    %_lto_cflags            %{expand:%%{_%{toolchain}_lto_cflags}}

    %_general_options       -O2 %{?_lto_cflags} -fexceptions -g -grecord-gcc-switches -pipe

and build issues occured:

  /build/usr/src/debug/tarantool-2.8.0.2/src/lib/core/reflection.h:124:33: warning: type of ‘METHODS_SENTINEL’ does not match original declaration [-Wlto-type-mismatch]
    124 | extern const struct method_info METHODS_SENTINEL;
        |                                 ^
  /build/usr/src/debug/tarantool-2.8.0.2/src/lib/core/reflection.c:35:26: note: ‘METHODS_SENTINEL’ was previously declared here
     35 | const struct method_info METHODS_SENTINEL = {
        |                          ^

it was fixed as was suggested in 1st way of [1][2]. This warning was
disabled in commit:

  f9e28ce4 ("Add LTO support")

with cmake flag:

  -D_ENABLE_LTO=ON

so added this flag to cmake RPM spec build flags for Fedora 33 and its
later versions.

Needed for #5502
Needed for #5697

[1] - https://github.com/tarantool/tarantool/issues/5697#issuecomment-759409465
[2] - https://github.com/tarantool/tarantool/pull/5707#discussion_r561072044

Removed CentOS 6 from packages build jobs, due to it reached
end of lifetime [1].

[1] - https://wiki.centos.org/About/Product

After cURL was changed to use cmake in commit

  2b076019 ('build: enable cmake in curl build')

there is no need to set dependencies to autotools. So removed it from:

  - rpm build spec for packages builds
  - deb build control for packages builds
  - OSX build targets
  - static OSX build target
  - freebsd VBOX VM image setup

Enabled replication/ suite in packaging testing. After major number of
issues were resolved in test-run tool and in frequently failing flaky
tests replication/ suite can be tested within packaging testing.

Closes #4798

Removed obvious part in rpm spec for Travis-CI, due to it is no
longer in use.

---- Comments from @Totktonada ----

This change is a kind of revertion of the commit
d48406d5 ('test: add more tests to
packaging testing'), which did close #4599.

Here I described the story, why the change was made and why it is
reverted now.

We run testing during an RPM package build: it may catch some
distribution specific problem. We had reduced quantity of tests and
single thread tests execution to keep the testing stable and don't break
packages build and deployment due to known fragile tests.

Our CI had to use Travis CI, but we were in transition to GitLab CI to
use our own machines and don't reach Travis CI limit with five jobs
running in parallel.

We moved package builds to GitLab CI, but kept build+deploy jobs on
Travis CI for a while: GitLab CI was the new for us and we wanted to do
this transition smoothly for users of our APT / YUM repositories.

After enabling packages building on GitLab CI, we wanted to enable more
tests (to catch more problems) and wanted to enable parallel execution
of tests to speed up testing (and reduce amount of time a developer wait
for results).

We observed that if we'll enable more tests and parallel execution on
Travis CI, the testing results will become much less stable and so we'll
often have holes in deployed packages and red CI.

So, we decided to keep the old way testing on Travis CI and perform all
changes (more tests, more parallelism) only for GitLab CI.

We had a guess that we have enough machine resources and will able to do
some load balancing to overcome flaky fails on our own machines, but in
fact we picked up another approach later (see below).

That's all story behind #4599. What changes from those days?

We moved deployment jobs to GitLab CI[^1] and now we completely disabled
Travis CI (see #4410 and #4894). All jobs were moved either to GitLab CI
or right to GitHub Actions[^2].

We revisited our approach to improve stability of testing. Attemps to do
some load balancing together with attempts to keep not-so-large
execution time were failed. We should increase parallelism for speed,
but decrease it for stability at the same time. There is no optimal
balance.

So we decided to track flaky fails in the issue tracker and restart a
test after a known fail (see details in [1]). This way we don't need to
exclude tests and disable parallelism in order to get the stable and
fast testing[^3]. At least in theory. We're on the way to verify this
guess, but hopefully we'll stick with some adequate defaults that will
work everywhere[^4].

To sum up, there are several reasons to remove the old workaround, which
was implemented in the scope of #4599: no Travis CI, no foreseeable
reasons to exclude tests and reduce parallelism depending on a CI
provider.

Footnotes:

[^1]: This is simplification. Travis CI deployment jobs were not moved
      as is. GitLab CI jobs push packages to the new repositories
      backend (#3380). Travis CI jobs were disabled later (as part of
      #4947), after proofs that the new infrastructure works fine.
      However this is the another story.

[^2]: Now we're going to use GitHub Actions for all jobs, mainly because
      GitLab CI is poorly integrated with GitHub pull requests (when
      source branch is in a forked repository).

[^3]: Some work toward this direction still to be done:

      First, 'replication' test suite still excluded from the testing
      under RPM package build. It seems, we should just enable it back,
      it is tracked by #4798.

      Second, there is the issue [2] to get rid of ancient traces of the
      old attempts to keep the testing stable (from test-run side).
      It'll give us more parallelism in testing.

[^4]: Of course, we perform investigations of flaky fails and fix code
      and testing problems it feeds to us. However it appears to be the
      long activity.

References:

[1]: https://github.com/tarantool/test-run/pull/217
[2]: https://github.com/tarantool/test-run/issues/251

Added ENABLE_WERROR flag to build options to enable Werror.

Part of #4941

Since LuaJIT provides extended LuaC API introduced in the scope of
5a61e1ab54b5c66bfebd836db1ac47996611e065 ('misc: add C and Lua API for
platform metrics') corresponding header should be provided along with
other Tarantool development files.

Follows up #5187

Implemented openSUSE packages build with testing for images:
opensuse-leap:15.[0-2]

Added %{sle_version} checks in Tarantool spec file according to
https://en.opensuse.org/openSUSE:Packaging_for_Leap#RPM_Distro_Version_Macros

Added opensuse-leap of 15.1 and 15.2 versions to Gitlab-CI packages
building/deploing jobs.

Closes #4562

Changed number of tests to packaging testing jobs from
suites: "unit/ app/ app-tap/ box/ box-tap/ engine/ vinyl/",
to all suites except 'replication/' suite which has special
issue for it's enabling:
  https://github.com/tarantool/tarantool/issues/4798
Also added testing to the all available packing jobs except
'CentOS 6'.

Removed test-run option that run testing inline to be
able to run it in parallel. Also changed 'test-run' to
'make test-force' command.

Closes #4599

Found that issues #1227 and #1322 were closed, returned
the tests blocked by it into the testing.

Part of #4599

Tarantool packages build scenario at rpm/tarantool.spec
installs Python packages for Tarantool testing. In some
latest OS like Fedora 32 old Python packages based on
Python 2 version were deprecated and changed to Python 3.
Currently Tarantool testing based on Python 2 which
packages stored at local backport repository and it
should be used from there calling it as python2-* packages.

Issue:
...
No matching package to install: 'python-yaml >= 3.0.9'
...
Not all dependencies satisfied
Error: Some packages could not be found.

Needed for #4599
Closes #4965

Added build + test jobs in GitLab-CI and build + test + deploy jobs on
Travis-CI for Fedora 31.

Updated testing dependencies in the RPM spec to follow the new Python 2
package naming scheme that was introduced in Fedora 31: it uses
python2-' prefix rather then 'python-'.

Fedora 31 does not provide python2-gevent and python2-greenlet packages,
so they were pushed to https://packagecloud.io/packpack/backports
repository. This repository is enabled in our build image
(packpack/packpack:fedora-31) by default. Those dependencies are build-time,
so nothing was changed for a user. The source RPM packages were gathered
from https://rpms.remirepo.net/rpmphp/

.

Closes #4612

Reviewed-by: Alexander Turenko <alexander.turenko@tarantool.org>

Added build + test jobs in GitLab-CI and build + test + deploy jobs on
Travis-CI for CentOS 8.

Updated testing dependencies in the RPM spec to follow the new Python 2
package naming scheme that was introduced in CentOS 8: it uses
'python2-' prefix rather then 'python-'.

CentOS 8 does not provide python2-gevent and python2-greenlet packages,
so they were pushed to https://packagecloud.io/packpack/backports
repository. This repository is enabled in our build image
(packpack/packpack:el-8) by default. Those dependencies are build-time,
so nothing was changed for a user. The source RPM packages were gathered
from https://cbs.centos.org

.

Disabled app-tap/pwd.test.lua on CentOS 8 due to systemd-nss issue,
which was not worked around properly. Filed #4592 to resolved it in the
future.

Eliminated libunwind runtime dependency (and libunwind-devel build
dependency) on CentOS 8, because the base system does not provide it.
fiber.info() backtraces and printing of a backtrace after a crash will
not be available on this system. Hopefully we'll fix it in the future,
filed #4611 on this.

Closes #4543

Reviewed-by: Alexander Turenko <alexander.turenko@tarantool.org>
Reviewed-by: Igor Munkin <imun@tarantool.org>

After we started using bundled version of libyaml by default (see commit
47b91e90), we can remove it from
building dependencies for RPM and DEB packages.

Closes #4442

Reviewed-by: Alexander Turenko <alexander.turenko@tarantool.org>