This patch extends the 'integration.yml' workflow and adds a new
workflow call for running tests to verify integration between tarantool
and the smtp module.

Part of #5265
Part of #6056
Closes #6523

The idea of these changes is to run testing workflows on self-hosted
ubuntu-20.04 machines instead of GitHub ubuntu-20.04 runners. It is
planned to use GitHub ubuntu-20.04 runners only for integration testing
of tarantool with a module/connector where ephemeral environments will
allow us avoiding various cleanup pains on self-hosted runners like

    - if: always()
      run: sudo apt-get -y purge 'tarantool*'

or

    - if: always()
      run: rm -rf .rocks

or

    - if: always()
      run: rm -rf pytest-venv

Also, switch workflows running on GitHub ubuntu-18.04 runners to
self-hosted ubuntu-20.04 machines.

For self-host runners, where the workspace with sources is saving
between different workflow runs, it is needed to be sure that all file
permissions are correct for running user to be sure that later call to
sources checkout action by the same user won't fail on sources cleanup.

Follows up tarantool/tarantool-qa#102

This patch extends the 'integration.yml' workflow and adds a new
workflow call for running tests to verify integration between tarantool
and the cartridge module.

Part of #5265
Part of #6056
Closes #6512

Imagine the following situation. One pushed a commit to the master
branch, the 'integration.yml' workflow is triggered and starts to work.
The first step of this workflow is to build tarantool from the pushed
commit. Sometimes we have to wait for a free runner for some period of
time because our resources are limited. One pushed a commit to the
master branch one more time, but tarantool build for the previous
commit hasn't started yet. When tarantool build from the previous
commit starts, actually tarantool will be built from the last commit
because ${{ github.ref }} is passed to the 'reusable_build.yml'
workflow as an input parameter. So we need to pass ${{ github.sha }}.

For now, there is no testing for the tarantool project verifying its
integration with different modules and connectors from the ecosystem.
This is a quite huge gap in our CI system that is going to be covered
by these changes.

This patch introduces a couple of new workflow files that are named
'integration.yml' and 'reusable_build.yml'. The main workflow is
'integration.yml' that will run automatically per push to master and
release branches. Also, this workflow can be run manually against a
development branch. The 'reusable_build.yml' workflow is called by the
main workflow and builds needed tarantool packages (at this moment for
Ubuntu Focal Fossa only), then stores them as a build artifact. After
that, the main workflow calls the 'reusable_testing.yml' workflow from
a module project that tarantool is going to verify integration with.
The testing workflow should download the tarantool build artifact and
run tests against it.

The basic scheme describing the verification process is represented
below:

      integration.yml
             |
             |
             v
     reusable_build.yml --> <artifact>
             |                  |
             |                  |
             v                  |
    reusable_testing.yml <------+
             |
             |
             v
          <result>

For now, we start only with the vshard module. In the future, we are
going to extend the module list incrementally.

Part of #5265
Part of #6056
Closes #4972

Reasons, in short:

* It reaches end of the standard support.
* It looks barely usable with old ca-certificates due to Let's Encrypt
  certificate expire problem.
* It has problems that block updating compat level for our *.deb
  packages.

All details and links can be found in the linked issue.

Fixes #6502

Was added 'export DEBIAN_FRONTEND=noninteractive', because there are
packages, which are noninteractive by default. So, CI raises
'Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?'.

Were added workflows with *.rpm or *.deb packaging for arm architecture.
Those workflows only verify package builds and don't deploy them to our
repositories for now.

Part of #6222

It requires update_repo.sh changes, which are not trivial in
verification. We want to be sure that all apt/yum metainformation is
generated correctly, so we'll postpone the automatic deployment task.

Part of #6222

Follows up #6185

New release policy defines pre-release tags: *-alpha*, *-beta*, *-rc*.
Packages with these tags must be published to pre-release folder in
the S3 repo. Was added PRERELEASE_REPO_S3_DIR path to all workflows with
packaging *.rpm and *.deb and fixed conditions for updating the S3 repo.

Needs for: #6185

MacOS has its own libunwind which works not good with fibers
under M1. Backtraces need to be enabled again after #6060 is fixed.
Other arm backtraces are temporarily disabled too for #6222.

Closes #6272

The ability to select an allocator for memtx has been added to
tarantool. To test a new type of allocator, all tests must also
be run with it. Implemented new option, which allows to set allocator
for memtx. If you wan't to choose allocator type for tests, run test-run.py
with --memtx-allocator="small" or --memtx-allocator="system". Allocator type
is passed via MEMTX_ALLOCATOR environment variable to the test.

Found that Github actions/checkout@v2, when Github Actions triggers
a job upon the tag push, it makes the following calls [1]:

  git fetch origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
  git fetch origin +$HASH:refs/tags/$TAG

While the first call is fine, the second one spoils the tag and makes
it non-annotated.

Later, Tarantool calls git describe without --tags flag, and the
spoiled tag is ignored. This patch handles it by fetching all the
tags once again explicitly.

Also explained the next step, where we remove the tag. The Github
Actions may runs twice the same job on branch push and on new tag
if it was created on this push. To avoid of fails on saving the
commonly named packages, we drop a tag that points to a current
commit (if any) on a job triggered by pushing to a branch (it will
produce the package based on previous tag) and leave the new tag
in job triggered by tag.

To check this change was used commands to create dump commit and tag
the new version. Also made twice push with extra flag for tags:

  git commit -m"Dump commit" --allow-empty
  git tag -a 2.4.8 -m ''
  git push && git push --tags

In this way would be run 2 jobs for each package:

  1. Build package on testing branch. It will produce package with
     name 2.4.7.x (because of the exception mentioned above).
  2. Build package on tag 2.4.8. It will produce package with name
     2.4.8.x.

Closes tarantool/tarantool-qa#119

[1]: https://github.com/actions/checkout/issues/290

Fedora 28 unsupported since 2019-05-28.
Fedora 29 unsupported since 2019-11-26.

Part of: #6074

Add fedora-34 workflow, which runs on push and pull-requests.

Part of: #6074

We use `-notest` postfix when wanna share
the code only, without running tests. And
for FreeBSD template the snippet has been
missed. Add it.

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
Reviewed-by: Alexander Turenko <alexander.turenko@tarantool.org>

Odroid is GNU/Linux ARM64 platform. In scope of this commit new GitHub
Actions workflows for testing Tarantool on Odroid hosts are added:

  Release: .github/workflows/odroid_arm64.yml
  Debug: .github/workflows/odroid_debug_arm64.yml

Introduced new targets in .travis.mk Makefile:

  deps_odroid: Installs required dependencies.

  build_odroid: Builds Tarantool with the following flags set
    in env of .github/workflows/odroid_debug_arm64.yml file:
      1. to avoid the issue #6142:
         -DENABLE_BACKTRACE=OFF
      2. to avoid the issue #6143:
         -DCMAKE_C_FLAGS="-Wno-type-limits "
         -DCMAKE_BUILD_TYPE=Debug

  test_odroid: Builds and tests `LuaJIT-test` suite on Odroid.

Also v1 version of GitHub checkout action is used, because action
version v2 was introduced in git version 2.18.0 [1]. The latest
available version on Odroid is the following:

  git is already the newest version (1:2.17.1-1ubuntu0.8).

[1]: https://github.com/actions/checkout#readme

Closes tarantool/tarantool-qa#121

Add ubuntu-hirsute workflow, which runs on push and pull-requests.
Fix lintian globbing-patterns-out-of-order warnings.

Part of: #5825

Found that after change:

  474eda49 ('github-ci: use vardir option in tests runs')

where 'vardir' was changed. It was forgot to update the Github Actions
workflows with the same change to be able to collect artifacts. This
patch fixes it.

Closes tarantool/tarantool-qa#125

Add ubuntu-groovy workflow, which runs on push and pull-requests.
Fix lintian globbing-patterns-out-of-order warnings.

Part of: #5824

In scope of this commit new GitHub Actions workflows for testing
Tarantool on M1 hosts are added:

  Release: .github/workflows/osx_arm64_11_2.yml
  Debug: .github/workflows/osx_debug_arm64_11_2.yml

Since GitHub Actions uses x86_64 environment by default on M1 targets,
'arch -arm64' prefix is specified in GitHub Actions workflow to make all
commands in .travis.mk run in ARM64 environment.

Introduced a new temporary target in .travis.mk Makefile to run only
specific LuaJIT test suites on M1. Now it runs only the following LuaJIT
test targets:
* PUC-Rio-Lua-5.1-tests
* lua-Harness-tests
* tarantool-tests

Python 3.9 is installed by default on M1 hosts, but gevent is required
for Tarantool tests, which installation fails with the following error:

  Using cached gevent-21.1.2.tar.gz (5.9 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... error
    ERROR: Command errored out with exit status 1:
     command: /opt/homebrew/opt/python@3.9/bin/python3.9 /opt/homebrew/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /var/folders/b0/1vlv5rvn77x2rn6zbl2p4tqr0000gp/T/tmpyy59ae2p
         cwd: /private/var/folders/b0/1vlv5rvn77x2rn6zbl2p4tqr0000gp/T/pip-install-msbf7_vz/gevent_c2956687bb0d4de9bfb5f0660da759ee
    Complete output (42 lines):
      ...
      File "/private/var/folders/b0/1vlv5rvn77x2rn6zbl2p4tqr0000gp/T/pip-build-env-1lesbbxi/overlay/lib/python3.9/site-packages/cffi/api.py", line 48, in __init__
        import _cffi_backend as backend
    ImportError: dlopen(/private/var/folders/b0/1vlv5rvn77x2rn6zbl2p4tqr0000gp/T/pip-build-env-1lesbbxi/overlay/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so, 2): no suitable image found.  Did find:
    	/private/var/folders/b0/1vlv5rvn77x2rn6zbl2p4tqr0000gp/T/pip-build-env-1lesbbxi/overlay/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so: mach-o, but wrong architecture
    	/private/var/folders/b0/1vlv5rvn77x2rn6zbl2p4tqr0000gp/T/pip-build-env-1lesbbxi/overlay/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so: mach-o, but wrong architecture

This issue is described in gevent/gevent#1721. Fortunately, gevent can
be successfully installed via Python 3.8, hence to avoid this failure,
python3 is pinned to the specific version (i.e. python@3.8) until the
mentioned issue is resolved.

Closes tarantool/tarantool-qa#120
Relates to #6068

To avoid targets duplication in the later use of .travis.mk file, it was
decided to parameterize OSX jobs and move all fine tuning manipulations
related to these pipelines from .travis.mk to the corresponding GitHub
Actions workflows.

Found that commit message may consists of special characters which can
be used be Markdown as commands, like '`' or '```'. To avoid of it these
characters must be changed to some predefined names like for:

'\' - BACKSLASH
'`' - BACKTICK

Also added filter block to avoid of other not known symbols which we
could miss. This block converts commit message to HTML and then takes
only text from it.

Found that on self-hosted runners where CentOS 7 is the base OS,
'send-telegram-notify' action creates message with syntax error:

  --------------'\n't'\n'```'\n'')) ; \
  ^
  SyntaxError: unexpected character after line continuation character

It happened because of extra quotes at '\n' while it had to be \n.
To avoid of it the same message changes must be done as for OSX
hosts are doing. These changes should be done when self-hosted
runners uses RHEL or Fedora as base OS.

After commit:

  58fe0fcb ('github-ci: avoid of use container tags in actions')

We began to use not the docker containers, but native github hosts.
To avoid of permissions fails on native github actions runners apt
command must run using sudo. Added flag '-n|--non-interactive' to
sudo command to avoid prompting the user for input of any kind which
could hang it. Added '-y' flag to apt update command to accept changes.

For now python3 is used as the default python on all OS and it is
needed to enable it in send-telegram-notify action.

Found issue:

  Traceback (most recent call last):
    File "<string>", line 3, in <module>
  AttributeError: module 'urllib' has no attribute 'quote_plus'

In Python 3 quote_plus included into urllib.parse.

Check documentaion [1]:

  Note The urllib module has been split into parts and renamed in Python 3 to urllib.request, urllib.parse, and urllib.error.

Check the same issue [2].

This patch changes use of all needed routines just from 'urllib'.

Closes tarantool/tarantool-qa#112

[1]: https://docs.python.org/2/library/urllib.html
[2]: https://github.com/web2py/web2py/issues/1822

Workflow 'debug_coverage' produces and uploads results to 'coveralls.io'
web site. Message about it can be shown in PR within each run was done.
This patch adds the ability to send message in available PR otherwise it
is skipped. Also added 'coverage.info' file to artifacts list.

Found that used 'coverallsapp/github-action' action checks if the
trigger was 'pull_request' used to run the workflow [1]. And only in
this way it writes results message to PR. Previously 'pull_request'
trigger was blocked to avoid duplication with 'push' trigger. To make
able to run workflow by any single trigger and to send message to PR
if it exits, it was implemented the following logic:

  - run workflows on both triggers 'push' and 'pull_request';
  - for workflow with 'push' trigger check if PR exists then skip all
    jobs otherwise continue running;
  - for workflow with 'pull_request' trigger continue running;

To avoid of issue coverallsapp/github-action#55 [2], sources checkout
should be based on 2nd version and later.

Closes #5644

[1]: https://github.com/coverallsapp/github-action/blob/master/src/run.ts#L38
[2]: https://github.com/coverallsapp/github-action/issues/55#issuecomment-644927165

Found that 'actions/checkout' does not remove all temporary files from
previous runs in submodules [1], it runs only 'git clean --xffd' [2]:

  libgcov profiling error:/home/ubuntu/actions-runner/_work/tarantool/tarantool/src/lib/small/CMakeFiles/small.dir/small/small_class.c.gcda:overwriting an existing profile data with a different timestamp

To avoid of it added:

  git submodules foreach --recursive 'git clean -xffd'

to 'actions/environment' which is run after each 'actions/checkout'.

Part of #5986

[1]: https://github.com/tarantool/tarantool/runs/2318244478?check_suite_focus=true#step:5:4012
[2]: https://github.com/actions/checkout/issues/358

For self-hosted runners run w/o restart may need to kill hanged
processes that could be leaved from the previous workflows runs.
This patch implements in cleanup with kill for such processes in
'environment' actions which is called for all workflows and runs
before main steps. Cleanup searches for two names patterns of
running processes:

  - ' tarantool '
  - 'test-run.py '

Closes tarantool/tarantool-qa#114

Enabling ubuntu-20.04 hosts for packaging workflows found that DEB
package Github Actions workflows do not need to install createrepo
tool. Also found that createrepo is not ready for ubuntu-20.04 as
described in (till ubuntu-21.04 where it is available as the new
version of this tool named as 'createrepo_c' as DEB package):

  3a7c2102 ('github-ci: ubuntu-20.04 misses createrepo package')

To fix it added 'createrepo_c' build and installation from sources
and changed in update_repo tool 'createrepo' tool to 'createrepo_c'.
This patch is needed to use these workflows on self-hosted runners
which run under ubuntu-20.04 by default for now.

Also checking the patch on ubuntu-20.04 hosts got the following issue:

  Regenerated DEB file: pool/xenial/main/t/tarantool/tarantool-common_2.8.0.185.g4c3e0eb-1_all.deb

  <botocore.awsrequest.AWSRequest object at 0x7f7998a4ca90>

  <botocore.awsrequest.AWSRequest object at 0x7f627d965070>
  make: *** [.gitlab.mk:131: deploy] Error 255
  Error: Process completed with exit code 2.

Found that there is already issue exists in Github Actions on it [1].
Provided solution to setup AWS region in environment helped to
workaround the issue [2].

Closes tarantool/tarantool-qa#110
Closes tarantool/tarantool-qa#111

[1]: https://github.com/aws/aws-cli/issues/5234
[2]: https://github.com/aws/aws-cli/issues/5234#issuecomment-635459464

Created local composite action 'pack_and_deploy' for Tarantool packages
creation and deployment. It was created using local scripts in packages
workflows. It let to change common parts of packages creations and
deployment in each packaging workflow to call for this action. It helped
to consolidate all the instructions on packages creation and deployment
in a single place.

For self-host runners, where work path with sources is saving between
different workflows runs, it is needed to be sure that all permissions
correct for running user to be sure that later call to sources checkout
by 'actions/checkout' action run by the same user won't fail on sources
cleanup.

Closes tarantool/tarantool-qa#102

Workflow 'coverity' produces and uploads results to 'coverity.com' web
site. Message about it can be shown in PR within each run was done.
This patch adds the ability to send message in available PR otherwise
it is skipped.

Part of #5644

We should remove a tag after fetching of a remote repository.
Ported the following commits:

  0f575e01 ('gitlab-ci: fix tag removal for a branch push job')
  0f564f34 ('gitlab-ci: remove tag from pushed branch commit')

Drop a tag that points to a current commit (if any) on a job triggered
by pushing to a branch (as against of pushing a tag). Otherwise we may
get two jobs for the same x.y.z-0-gxxxxxxxxx build: one is run by
pushing a branch and another by pushing a tag. The idea is to hide the
new tag from the branch job as if a tag would be pushed strictly after
all branch jobs for the same commit.

Closes tarantool/tarantool-qa#103

Changed the following workflows:

  luacheck
  debug_coverage
  release*
  static_build
  static_build_cmake_linux

It was changed the OS in which the test run from debian to ubuntu.
Also changed the way how this OS was booted - before the change it
was booted as docker container using Github Actions tag from inside
the worklfows. And it caused all the workflow steps to be run inside
it. After the change no container run anymore on the running host.
Github Actions host uses for now with its native OS set in 'runs'
tag. It was decided to use the latest one OS `ubuntu-20.04` which is
already the default for 'ubuntu-latest' tag.

This change gave us the abilities to:
 - Remove extra container step in workflow.
 - Switch off swap using 'swapoff' command.
 - Use the same OS as Github Actions uses by default.
 - Setup our local hosts using Github Actions image snapshot.
 - Enable use of actions/checkout@v2.3.4 which is better than v1.
 - Light bootstrap of packages in local .*.mk makefile for:
     build: libreadline-dev libunwind-dev
     tests: pip install -r test-run/requirements.txt

Closes tarantool/tarantool-qa#101

Github Actions provides hosts for Linux base runners in the following
configurations:

  2 Cores
  7 Gb memory
  4 Gb swap memory

To avoid of issues with hanging/slowing tests on high memory use
like [1], hosts configurations must avoid of swap memory use. All
of the tests workflows run inside dockers containers. This patch
sets in docker run configurations memory limits based on current
github actions hosts - 7Gb memory w/o swap memory increase.

Checked 10 full runs (29 workflows in each run used the change) and
got single failed test on gevent() routine in test-run. This result much
better than w/o this patch when 3-4 of workflows fail on each full run.

It could happen because swappiness set to default value:

  cat /sys/fs/cgroup/memory/memory.swappiness
  60

From documentation on swappiness [2]:

  This control is used to define the rough relative IO cost of swapping
  and filesystem paging, as a value between 0 and 200. At 100, the VM
  assumes equal IO cost and will thus apply memory pressure to the page
  cache and swap-backed pages equally; lower values signify more
  expensive swap IO, higher values indicates cheaper.
  Keep in mind that filesystem IO patterns under memory pressure tend to
  be more efficient than swap's random IO. An optimal value will require
  experimentation and will also be workload-dependent.

We may try to tune how often anonymous pages are swapped using the
swappiness parameter, but our goal is to stabilize timings (and make
them as predictable as possible), so the best option is to disable swap
at all and work on descreasing memory consumption for huge tests.

For Github Actions host configurations with 7Gb RAM it means that after
2.8Gb RAM was used swap began to use. But in testing we have some tests
that use 2.5Gb of RAM like 'box/net_msg_max.test.lua' and memory
fragmentation could cause after the test run swap use [3].

Also found that disk cache could use some RAM and it also was the cause
of fast memory use and start swapping. It can be periodically dropped
from memory [4] using 'drop_cache' system value setup, but it won't fix
the overall issue with swap use.

After freed cached pages in RAM another system kernel option can be
tuned [5][6] 'vfs_cache_pressure'. This percentage value controls the
tendency of the kernel to reclaim the memory which is used for caching
of directory and inode objects. Increasing it significantly beyond
default value of 100 may have negative performance impact. Reclaim code
needs to take various locks to find freeable directory and inode
objects. With 'vfs_cache_pressure=1000', it will look for ten times more
freeable objects than there are. This patch won't do this change, but
it can be done as the next change.

To fix the issue there were made changes:

 - For jobs that run tests and use actions/environment and don't use
   Github Actions container tag, it was set 'sudo swapoff -a' command
   in actions/environment action.

 - For jobs that run tests and use Github Actions container tag the
   previous solution doesn't work. It was decided to hard-code the
   memory value based on found on Github Actions hosts memory size
   7Gb. It was set for Github container tag as additional options:
     options: '--init --memory=7G --memory-swap=7G'
   This changes were made temporary till these containers tags will
   be removed within resolving tarantool/tarantool-qa#101 issue for
   workflows:
     debug_coverage
     release
     release_asan_clang11
     release_clang
     release_lto
     release_lto_clang11
     static_build
     static_build_cmake_linux

 - For VMware VMs like with FreeBSD added 'sudo swapoff -a' command
   before build commands.

 - For OSX on Github actions hosts swapping already disabled:
     sysctl vm.swapusage
     vm.swapusage: total = 0.00M  used = 0.00M  free = 0.00M  (encrypted)
   Also manual switching off swap currently not possible due to do
   System Integrity Protection (SIP) must be disabled [7], but we
   don't have such access on Github Actions hosts. For local hosts
   it must be done manually with [8]:
     sudo nvram boot-args="vm_compressor=2"
   Added swap status control to be sure that host correctly configured:
     sysctl vm.swapusage

Closes tarantool/tarantool-qa#99

[1]: https://github.com/tarantool/tarantool-qa/issues/93
[2]: https://github.com/torvalds/linux/blob/1e43c377a79f9189fea8f2711b399d4e8b4e609b/Documentation/admin-guide/sysctl/vm.rst#swappiness
[3]: https://unix.stackexchange.com/questions/2658/why-use-swap-when-there-is-more-than-enough-free-space-in-ram
[4]: https://kubuntu.ru/node/13082
[5]: https://www.kernel.org/doc/Documentation/sysctl/vm.txt
[6]: http://devhead.ru/read/uskorenie-raboty-linux
[7]: https://osxdaily.com/2010/10/08/mac-virtual-memory-swap/
[8]: https://gist.github.com/dan-palmer/3082266#gistcomment-3667471

Added manual and backend triggers to run test workflows. It will
give the ability to run missed/needed workflows in Github Actions
and to use standalone backend scripts to run test workflows.

Found that ubuntu-latest that switched in Github Actions from
ubuntu-18.04 to ubuntu-20.04, doesn't have createrepo package
available in Ubuntu repositories to be installed. It happened
because createrepo written on python2, which is not supported
by ubuntu-20.04. Anyway createrepo tool was rewritten on C to
make it available for the later distributions, its new naming
is createrepo_c. It is already avaliable in Ubuntu 21.04, but
still not backported to 20.04 version. So we need to wait its
avalaibility in Ubuntu 22.04 either its backported version in
20.04 [1]. As for now we need to use ubuntu-18.04 in Actions.

[1]: https://answers.launchpad.net/createrepo/+question/690448