godzie44 authored 8 months ago and Maksim Kaitmazian committed 7 months ago

Closes #584

Dmitry Rodionov authored 8 months ago and Dmitry Ivanov committed 8 months ago

> cargo tree -i either
either v1.13.0
├── itertools v0.10.5
│   └── sbroad-core v0.1.0 (picodata/sbroad/sbroad-core)
│       └── picodata v24.5.0 (picodata/picodata)

feature(plugin migration): use md5 from migration files meta for to confirm consistency of migrations between versions

Closes #652

The problem appeared in form of a failed debian package build. aws-lc-rs
build script was failing with:

thread 'main' panicked at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/aws-lc-sys-0.19.0/builder/cc_builder.rs:244:13:
Your compiler (cc) is not supported due to a memcmp related bug
reported in https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95189.We
strongly recommend against using this compiler.

Since we use system openssl we dont need to rely on these backends.

Error message is probably correct, so upgrading the compiler is still a
good idea.

See: https://git.picodata.io/picodata/picodata/picodata/-/issues/630

godzie44 authored 11 months ago and Georgy Moshkin committed 10 months ago

- add description field to `_pico_plugin` and `_pico_service`
- plugin loading: filter files by its extensions
- plugin loading: check possible service collisions
- plugin loading: now plugin searching at {plugin_dir}/{plugin_name} path
- now all errors between plugin and picodata wrapping with `BoxError` (this protects plugin caller from UAF)

We have recently replaced String with SmolStr in sbroad.
As a result, we need to make corresponding changes in
picodata as it uses sbroad-core for cluster SQL.

godzie44 authored 1 year ago and Yaroslav Dynnikov committed 11 months ago

Introduce a plugin system for picodata cluster.
Plugin system contains:
- `picoplugin` crate with plugin interfaces.
This crate exposes plugin interfaces for user implementation and API
that used by picodata to interact with plugin as with a code from dynamic library.
- Support for plugin work cycle in picodata instance work cycle.
Picodata instance interacts with running plugin by `PluginManager`
component.
Typically, plugins react to events that occur in instance or whole cluster.
- New lua API with plugin CRUD actions.
    - pico.create_plugin - load a new plugin into picodata cluster
    - pico.remove_plugin - remove plugin from picodata cluster
    - pico.update_plugin_config - update plugin service configuration

- Update sbroad submodule to commit,
with refactored tracing api: tracer
can be provided by library user (us).
- Use tracer that creates traces under
admin, because traces are created randomly
for 1% of the queries and the user may not
have the privileges to create/modify tracing
tables
- Move tracing related logic to picodata repo.
In particular, now all tables storing tracing
- Add sql tests that were removed from sbroad
during refactoring
- added extra parameter 'query_id' to pico.sql
lua api. 'query_id' is used to identify
each query in tracing tables, if not specified
hash function on query text is used.
- changed signature of pico.sql to:
pico.sql(pattern[, params, options)
options={'traceable'=boolean,'query_id'=string}

Having dynamic build as a default was dictated by certification and now
when we merged certification patches back to master it started to cause
friction. So this patch makes static build the default and introduces
dynamic_build cargo feature instead of static_build one.

1. Add a new option picodata run --shredding to enable shredding upon xlog_remove_file_impl.
2. For test purposes additionally use the inject_errors feature to keep files after shred operation
and allow the test code to look inside and make sure that content was really shredded.

New feature `static_build` is added to Cargo.toml. By default dynamic
build is being used. build.rs is changed to respect it.

Use `cargo build --features static_build` to build static binary.

Previous attempt directly modified static-build cmake files which
is less convenient and doesnt allow us to support both build types
at the same time

Replaces https://git.picodata.io/picodata/tarantool/-/merge_requests/143 and https://git.picodata.io/picodata/picodata/picodata/-/merge_requests/817

Now the records look like this:

```
{
  "message": "instance is starting",
  "severity": "low",
  "time": "2023-11-22T22:10:24.211+0300",
  "title": "local_startup"
}
{
  "message": "target grade of instance `i1` changed to Online(7)",
  "severity": "low",
  "time": "2023-11-22T22:10:24.334+0300",
  "title": "change_target_grade"
}
{
  "message": "current grade of instance `i1` changed to Replicated(7)",
  "severity": "medium",
  "time": "2023-11-22T22:10:24.436+0300",
  "title": "change_current_grade"
}
```