We have recently replaced String with SmolStr in sbroad.
As a result, we need to make corresponding changes in
picodata as it uses sbroad-core for cluster SQL.

godzie44 authored 11 months ago and Yaroslav Dynnikov committed 11 months ago

Introduce a plugin system for picodata cluster.
Plugin system contains:
- `picoplugin` crate with plugin interfaces.
This crate exposes plugin interfaces for user implementation and API
that used by picodata to interact with plugin as with a code from dynamic library.
- Support for plugin work cycle in picodata instance work cycle.
Picodata instance interacts with running plugin by `PluginManager`
component.
Typically, plugins react to events that occur in instance or whole cluster.
- New lua API with plugin CRUD actions.
    - pico.create_plugin - load a new plugin into picodata cluster
    - pico.remove_plugin - remove plugin from picodata cluster
    - pico.update_plugin_config - update plugin service configuration

- Update sbroad submodule to commit,
with refactored tracing api: tracer
can be provided by library user (us).
- Use tracer that creates traces under
admin, because traces are created randomly
for 1% of the queries and the user may not
have the privileges to create/modify tracing
tables
- Move tracing related logic to picodata repo.
In particular, now all tables storing tracing
- Add sql tests that were removed from sbroad
during refactoring
- added extra parameter 'query_id' to pico.sql
lua api. 'query_id' is used to identify
each query in tracing tables, if not specified
hash function on query text is used.
- changed signature of pico.sql to:
pico.sql(pattern[, params, options)
options={'traceable'=boolean,'query_id'=string}

Having dynamic build as a default was dictated by certification and now
when we merged certification patches back to master it started to cause
friction. So this patch makes static build the default and introduces
dynamic_build cargo feature instead of static_build one.

1. Add a new option picodata run --shredding to enable shredding upon xlog_remove_file_impl.
2. For test purposes additionally use the inject_errors feature to keep files after shred operation
and allow the test code to look inside and make sure that content was really shredded.

New feature `static_build` is added to Cargo.toml. By default dynamic
build is being used. build.rs is changed to respect it.

Use `cargo build --features static_build` to build static binary.

Previous attempt directly modified static-build cmake files which
is less convenient and doesnt allow us to support both build types
at the same time

Replaces https://git.picodata.io/picodata/tarantool/-/merge_requests/143 and https://git.picodata.io/picodata/picodata/picodata/-/merge_requests/817

Now the records look like this:

```
{
  "message": "instance is starting",
  "severity": "low",
  "time": "2023-11-22T22:10:24.211+0300",
  "title": "local_startup"
}
{
  "message": "target grade of instance `i1` changed to Online(7)",
  "severity": "low",
  "time": "2023-11-22T22:10:24.334+0300",
  "title": "change_target_grade"
}
{
  "message": "current grade of instance `i1` changed to Replicated(7)",
  "severity": "medium",
  "time": "2023-11-22T22:10:24.436+0300",
  "title": "change_current_grade"
}
```

Transferred replication factor from Properties table to new Tier table

Instance and Replicaset belongs to tier via new field tier in _pico_instance and _pico_replicaset

New cli option init-cfg with path to configuration file in yaml format. For now contains only info about tiers.

BREAKING CHANGE!:
- pico.trace() function was removed. Use pico.sql() for tracing;
- opentelemetry tables __SBROAD_STAT and __SBROAD_QUERY were renamed
  into _SQL_STAT and _SQL_QUERY tables.

Introduce enrypoints for extended PG protocol:

- pg_bind: bind parameters to portal (IR) by descriptor;
- pg_close: remove portal from the storage by descriptor;
- pg_describe: get the portal metadata and type by descriptor;
- pg_execute: execute portal by descriptor;
- pg_parse: parse an SQL pattern into IR (reusing LRU),
  save IR into portal storage and return the descriptor;
- pg_portals: get a list of portal descriptors visible
  to the current user;

Co-authored-by: Kaitmazian Maksim <m.kaitmazian@picodata.io>

Before this change we were building protobuf twice with different set of
features being enabled.

Part of cargo build --timings output before this change:

protobuf v2.28.0	3.3s	1.5s (46%)	bytes
protobuf v2.28.0	2.6s	0.7s (27%)	bytes, with-bytes

With this patch duplication is removed

The problem lies in nix package which references memfd_create libc symbol 
which doesnt exist in older libc versions available in centos and astra.

The solution is to patch nix to the state of previous version which uses
raw syscall.The offending commit that broke nix for us is:
https://github.com/nix-rust/nix/commit/cf15c2bb5505231bc3128ef61522336e315e2169#diff-17f9119515eead05b2e65afe0b31095ec95425112c31fe3c529a02f46528b970R56

Another instance of the problem in the wild:
https://github.com/lucab/libsystemd-rs/pull/144

The patch uses nix version 26.4 while the latest one is 27 because rustyline
didnt update to 27 yet and keeping two patched versions doesnt make sense.

So this is our fork with the patch: https://git.picodata.io/picodata/picodata/nix/-/tree/fix-memfd-create-for-older-libc-backport-26-4
And upstream PR: https://github.com/nix-rust/nix/pull/2146

In the long run nix doesnt look like a stable option. There is no libc version
compatibility policy at the moment. The way symbols are used makes them
required even if we're not using them. The alternative, rustix uses weak
symbols or falls back to raw syscalls:
https://github.com/bytecodealliance/rustix/blob/b08519f046787b93121214622aac015b68733c8f/src/backend/libc/fs/syscalls.rs#L1631
I believe with this solution we wouldnt have had this problem in the first
place. There is an open issue in rustyline with suggestion to use rustix
instead of nix: https://github.com/kkawakam/rustyline/issues/733

This patch adds new command `picodata sql`. Arguments are the same as `picodata connect`. The idea would be to eventually replace picodata connect with this version.

Simple prerequisites for testing with auth:

```lua
pico.create_user("Yoda", "12341234")
pico.grant_privilege("Yoda", "execute", "role", "super")
```
Super grant wont be needed in the future when access control will be tuned properly (access to system spaces, context: https://git.picodata.io/picodata/picodata/picodata/-/merge_requests/663).

Then run `target/debug/picodata run` in one terminal and `target/debug/picodata sql Yoda@localhost:3301` in another one

Features:
- Usual lib readline goodies, backward/forward search etc. Provided by [rustyline](https://lib.rs/crates/rustyline)
- Simple file based history (~/.picodata_history by default)
- Displays result sets in ascii tables. Styling can be customized if needed. Here we're using [comfy-table](https://lib.rs/crates/comfy-table) library
- If you type `\e` command your `$EDITOR` will be opened so you can write the query there. Once you close your editor the query will be executed (suggested by @funbringer)

There are a lot of things we can add. Completion, highlighting, more special commands.

Close #363 

Also fix some fresh clippy warnings

The reason of this refactoring is that we need Picodata specific
API rather then the API provided by tarantool module. For example,
table creation needs access to the clusterwide API for sharded and
global spaces.

Yaroslav Dynnikov authored 1 year ago and Yaroslav Dynnikov committed 1 year ago

Don't update blake3 version, otherwise it results in

```
Updating constant_time_eq v0.2.5 -> v0.3.0

error: package `constant_time_eq v0.3.0` cannot be built because it
requires rustc 1.66.0 or newer, while the currently active rustc version
is 1.65.0
```

Egor Ivkov authored 1 year ago and Yaroslav Dynnikov committed 1 year ago

This commit introduces new Lua function `pico.help()` along with
description of other public functions.

Current commit introduces distributed SQL in picodata (we use
'sbroad' library). Now it is possible to execute cluster-wide
SQL with 'pico.sql([[query]], {params})' lua method.

Current limitations:
1. DDL is still not implemented in picodata, so space sharding
   is mocked by '_pico_space' space (we'll remove it later).
2. We still have duplicating symbols of the Tarantool module in
   sbroad and in picodata rlibs.
3. We need to reuse piocdata's sharding functions and handle the
   case when different spaces in the query have different sharding
   functions.

Co-authored-by: Georgy Moshkin <gmoshkin@picodata.io>