Saving cache isn't instantaneous. Usually it takes about a minute.
If another job tries to restore cache at the same time, it fails
with the error "tar: Unexpected EOF in archive".

Now tar writes to a different filename with a random suffix, and then
renames it atomically.

CI on master branch always reuses the same archive. As a result, its
size gradually grows. It already takes more than 6GB, while it should be
about 3GB large. It makes restoring the cache slower and slower.

With this patch the cache is cleared every time in master branch.

IDE plugins (e.g. vscode-yaml) complains about invalid `gitlab-ci.yml`
schema because yaml anchors don't flatten lists.

This patch re-organizes anchors to praise schema validation and to make
it easier to understand (or at least it tries to).

The previous commit did compare changes to the master branch and always
skipped the `push-base-image` stage. In this commit it's refactored and
explained once more.

The whole CI is refactored. The following problems are solved:

1. Couldn't test base image locally without overriding the old one.

The testing stage is run in two ways depending on whether some
of the docker-related files were changed.

If they didn't (most likely), testing stage is run on the `latest`
docker image as it used to.

Otherwise, the docker image is rebuilt locally and tested. This is done
by using `image:pull_policy` option, see [1].

It also makes a problem of using protected credentials irrelevant.

2. Gitlab caching used to be slow

Stupid gitlab caching (see [2]) wasted on (un)zipping cache 4 minutes
(of 6 total). Now it's replaced with a manual one: an archive (tar) is
placed in a docker volume on a self-hosted gitlab runner with no
compression.

3. Speed up fetching git.

The tarantool-sys submodule requires a tag to be built correctly, but
using shallow clones prevented that. We used to work-around it by
specifying `GIT_DEPTH: 0`, but that's sub-optimal too. This patch
introduces an elegant yet reliable solution:

```
until git describe; do git fetch --deepen 100; done
```

4. No fine-grained time tracking was available

Gitlab only shows the total time of `step_script` without splitting it
into particular commands. This problem is solved by incorporating
collapsible log sections feature, see [3]. It makes logs fancy too.

Refs:

- [1] https://docs.gitlab.com/ee/ci/yaml/#imagepull_policy
- [2] https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3523
- [3] https://docs.gitlab.com/ee/ci/jobs/#custom-collapsible-sections

Alexey Protsenko authored 2 years ago and Yaroslav Dynnikov committed 2 years ago

1. Signed rpm packages used to be overridden by unsigned
   ones due to the artifacts clash. This bug is fixed by
   specifying explicit stage dependencies.
2. Add stage for testing installed packages.

Dmitry Travyan authored 2 years ago and Valentin Syrovatskiy committed 2 years ago

run image build only on push to master branch

1. Review `Pipfile`:
  - Remove unused `filelock`;
  - Install `mypy` - static type checker for Python.
2. Add new command `pipenv run lint`.
3. Enable `mypy` in CI. Fix reported errors in `test_basics.py`.
4. Renew readme.

Remove prctl dependency that is not available on mac.

+ tarantool-sys submodule
+ tarantool-patches directory
+ build.rs build script to patch and build tarantool
+ static linking with tarantool
+ refactoring for cli arguments

Close https://gitlab.com/picodata/picodata/picodata/-/issues/23