Add stage to run stress tests in downstream pipeline for each commit

godzie44 authored 11 months ago and Yaroslav Dynnikov committed 11 months ago

Introduce a plugin system for picodata cluster.
Plugin system contains:
- `picoplugin` crate with plugin interfaces.
This crate exposes plugin interfaces for user implementation and API
that used by picodata to interact with plugin as with a code from dynamic library.
- Support for plugin work cycle in picodata instance work cycle.
Picodata instance interacts with running plugin by `PluginManager`
component.
Typically, plugins react to events that occur in instance or whole cluster.
- New lua API with plugin CRUD actions.
    - pico.create_plugin - load a new plugin into picodata cluster
    - pico.remove_plugin - remove plugin from picodata cluster
    - pico.update_plugin_config - update plugin service configuration

Feodor Alexandrov authored 1 year ago and Dmitry Rodionov committed 11 months ago

This MR introduces many more patches needed to fix almost all known
issues with gamayn and svace (with exception of libdecnumber).

Additionally it does some improvements to workflow, now helper script
has `restore` command to clear dirty state when patches were applied to
simplify testing.

For places where asserts where needed they're extended with panic calls
because for static analyzer we need to sometimes prove that condition
holds even in release build where usual asserts are disabled.

Also this patch bumps webui repo to accomodate fixes made there.

Co-Authored-By: Dmitry Rodionov <d.rodionov@picodata.io>

This patch combines patches from these two picodata/tarantool fork branches:
https://git.picodata.io/picodata/tarantool/-/tree/dynamic-linking-from-system?ref_type=heads
https://git.picodata.io/picodata/picodata/picodata/-/tree/svace-fixes?ref_type=heads

Previously patches for tarantool were made for 2.11.0 so for 2.11.2 (version we currently use)
some of them had to be updated.

Patches from b124e8f01fafb98bc3005f77b8852d1f699f8666 initially were committed
to fork repo but for manageability it was decided that its easier to support
those via patch files. So these changes are added to existing patches in
certification_patches dir.

Then follow changes from 6b851b86
and 2be2aab5. Patch application is
currently concentrated in gitlab-ci.yml. 6b851b moved svace patches
application to tarantool's makefile. For now its easier to keep it
in one place. Svace patches from previous commit are moved into
subdirectory. Patches that touch supplementary files or simply remove
them are removed and the list of the files is kept for later to form a
dedicated list of files for deletion.

Conflicting were picked from f98334ea.
Redundant patch for curl libssh was removed because the same change
appeared upstream https://github.com/curl/curl/pull/11088/files



Luagdb patch was updated.
OSCP response patch as well as icu_common_exec patch were removed,
since original files are not needed in the build process and thus are
deleted from the repo.

Lastly patch application is moved to dedicated python script for
ease of sharing with vm builder repo.

Co-authored-by: Feodor Alexandrov <feodor.alex.alexandrov@gmail.com>

When using git describe fetch depth for shallow clones needs to be deep
enough to reach exiting tag. We used to only deepen our submodules for
tarantool build but now we reached the same problem for picodata. So
fetch tags for it as well.

allow_failure in test-helm-docker-image makes CI properly fail and
prevent merges when there are failures. Without it it didnt go past
"manual action required" icon and allowed to merge CI without fixing
other test jobs

Having dynamic build as a default was dictated by certification and now
when we merged certification patches back to master it started to cause
friction. So this patch makes static build the default and introduces
dynamic_build cargo feature instead of static_build one.

New feature `static_build` is added to Cargo.toml. By default dynamic
build is being used. build.rs is changed to respect it.

Use `cargo build --features static_build` to build static binary.

Previous attempt directly modified static-build cmake files which
is less convenient and doesnt allow us to support both build types
at the same time

Replaces https://git.picodata.io/picodata/tarantool/-/merge_requests/143 and https://git.picodata.io/picodata/picodata/picodata/-/merge_requests/817

By adding a comment here in ce9ddc26
I made the whole command to be ignored because for gitlab > means single
command that is split across lines, so by starting it with # on the
first line I made the whole command a comment.

See https://docs.gitlab.com/ee/ci/yaml/script.html#split-long-commands
for details.

pytest supports providing full diff between dictionaries when higher
verbosity is specified. For details see
https://docs.pytest.org/en/7.1.x/how-to/output.html#verbosity.

Make use of max verbosity level in CI (-vv)

Notice: gitlab doesn't support variables interpolation in
`rules:changes:compare_to`, so we interpolate it with yaml anchors.

This is a follow-up for ac2671e3

Web UI feature is already enabled in CI `test` stage. This patch adds it
to the `pack` stage.

To do that, a path to a webui buldle can be passed explicitly, and it
does in CI which now builds it in a separate stage.

In default Rust features `webui` remains disabled by default to keep
minimal development environment truly minimal.

Dmitry Rodionov authored 1 year ago and Yaroslav Dynnikov committed 1 year ago

After seeing failed checkout here
https://git.picodata.io/picodata/picodata/picodata/-/jobs/106805
I've found that gitlab supports specifying number of retries per job
stage. Use 3 attempts before giving up on fetching sources. See
https://docs.gitlab.com/ee/ci/runners/configure_runners.html#job-stages-attempts
for details

Having two different fixtures `binary_path` and
`path_to_binary_with_webui` results in weird failures in CI:

```
FAILED test/int/test_couple.py::test_exit_after_apply - conftest.TarantoolError:
    ('ER_NO_SUCH_PROC', "Procedure 'pico._inject_error' is not defined")
```

The root cause is that one `cargo build` overrides another with
different features enabled. To resolve the issue this patch
significantly refactors the test flow:

1. Web UI tests are skipped by default. One can enable it explicitly
   with `pytest --with-webui`

2. New fixture `cargo_build` behavior depends on environment:
   - In local dev environment it rebuilds the project with required features
   - In CI it skips rebuilding and runs tests against the existing build

3. The `path_to_binary_with_webui` fixture is removed.

See also:

- https://stackoverflow.com/a/55769818/1182787
- https://docs.pytest.org/en/7.4.x/how-to/writing_hook_functions.html
- https://docs.pytest.org/en/7.4.x/how-to/mark.html
- https://docs.pytest.org/en/7.4.x/example/simple.html#control-skipping-of-tests-according-to-command-line-option

The most obvious benefit is link checking. This is especially useful with rust-analyzer, because it allows to jump to definition of the item referenced in the doc comment.

The patch also makes CI to use `make lint` to avoid duplication.