Saving cache isn't instantaneous. Usually it takes about a minute.
If another job tries to restore cache at the same time, it fails
with the error "tar: Unexpected EOF in archive".

Now tar writes to a different filename with a random suffix, and then
renames it atomically.

CI on master branch always reuses the same archive. As a result, its
size gradually grows. It already takes more than 6GB, while it should be
about 3GB large. It makes restoring the cache slower and slower.

With this patch the cache is cleared every time in master branch.

Yaroslav Dynnikov authored 2 years ago and Yaroslav Dynnikov committed 2 years ago

Retain it in tests only

Georgy Moshkin authored 2 years ago and Yaroslav Dynnikov committed 2 years ago

Tarantool synchronous replication is broken in such a way, that local
& asynchronous transactions get rolled back upon synchronous
transaction rollback. This is critical for us, because we use local
spaces for the raft log, which should be orthogonal to synchronous
business spaces, but is not due to this behavior. Until it is fixed, we
will not use synchronous replication.

See https://github.com/tarantool/tarantool/issues/7903

Rewrite it from scratch to satisfy tests and to represent governor
logics in general.

No assumptions should be made on `traft::node::Status.leader_id`. It can
be set to `None` at any time due to a re-election.

Lots of tests are marked "fixme" with the desired value commented.
They'll be fixed in the next commit.

This scenario results in quorum loss and not much useful.

This somehow broke the build for raft-proto, until I added `feature =
["bytes"]` for protobuf.

Make it simpler.

IDE plugins (e.g. vscode-yaml) complains about invalid `gitlab-ci.yml`
schema because yaml anchors don't flatten lists.

This patch re-organizes anchors to praise schema validation and to make
it easier to understand (or at least it tries to).

The previous commit did compare changes to the master branch and always
skipped the `push-base-image` stage. In this commit it's refactored and
explained once more.

The whole CI is refactored. The following problems are solved:

1. Couldn't test base image locally without overriding the old one.

The testing stage is run in two ways depending on whether some
of the docker-related files were changed.

If they didn't (most likely), testing stage is run on the `latest`
docker image as it used to.

Otherwise, the docker image is rebuilt locally and tested. This is done
by using `image:pull_policy` option, see [1].

It also makes a problem of using protected credentials irrelevant.

2. Gitlab caching used to be slow

Stupid gitlab caching (see [2]) wasted on (un)zipping cache 4 minutes
(of 6 total). Now it's replaced with a manual one: an archive (tar) is
placed in a docker volume on a self-hosted gitlab runner with no
compression.

3. Speed up fetching git.

The tarantool-sys submodule requires a tag to be built correctly, but
using shallow clones prevented that. We used to work-around it by
specifying `GIT_DEPTH: 0`, but that's sub-optimal too. This patch
introduces an elegant yet reliable solution:

```
until git describe; do git fetch --deepen 100; done
```

4. No fine-grained time tracking was available

Gitlab only shows the total time of `step_script` without splitting it
into particular commands. This problem is solved by incorporating
collapsible log sections feature, see [3]. It makes logs fancy too.

Refs:

- [1] https://docs.gitlab.com/ee/ci/yaml/#imagepull_policy
- [2] https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3523
- [3] https://docs.gitlab.com/ee/ci/jobs/#custom-collapsible-sections

Alexey Protsenko authored 2 years ago and Yaroslav Dynnikov committed 2 years ago

Move build base on Ubuntu 22.04