godzie44 authored 11 months ago and Yaroslav Dynnikov committed 11 months ago

Introduce a plugin system for picodata cluster.
Plugin system contains:
- `picoplugin` crate with plugin interfaces.
This crate exposes plugin interfaces for user implementation and API
that used by picodata to interact with plugin as with a code from dynamic library.
- Support for plugin work cycle in picodata instance work cycle.
Picodata instance interacts with running plugin by `PluginManager`
component.
Typically, plugins react to events that occur in instance or whole cluster.
- New lua API with plugin CRUD actions.
    - pico.create_plugin - load a new plugin into picodata cluster
    - pico.remove_plugin - remove plugin from picodata cluster
    - pico.update_plugin_config - update plugin service configuration

Arseniy Volynets authored 1 year ago and Georgy Moshkin committed 1 year ago

- Add new DML op: Batch, which allows
using multiple dml operations in one
CAS request
- If request has too many dml operations,
the batch will be splitted into multiple
batches, so that each batch can be stored
in raft log

Also here we refactor all the places where we checked command line
parameters, because now PicodataConfig is the main source of this
information.

- Update sbroad submodule to commit,
with refactored tracing api: tracer
can be provided by library user (us).
- Use tracer that creates traces under
admin, because traces are created randomly
for 1% of the queries and the user may not
have the privileges to create/modify tracing
tables
- Move tracing related logic to picodata repo.
In particular, now all tables storing tracing
- Add sql tests that were removed from sbroad
during refactoring
- added extra parameter 'query_id' to pico.sql
lua api. 'query_id' is used to identify
each query in tracing tables, if not specified
hash function on query text is used.
- changed signature of pico.sql to:
pico.sql(pattern[, params, options)
options={'traceable'=boolean,'query_id'=string}

- remove logical clock from raft entry context, (term, index) is used
  for synchronization instead
- this allows us to remove the Notify/Notifier boilerplate
- NodeImpl::notifications -> NodeImpl::read_state_wakers
- this and the recently refactored proc_join finally allows us to remove
  OpResult
- ContextCoercion & CoercionError are also removed for the sake of
  simplification

Dmitry Rodionov authored 1 year ago and Yaroslav Dynnikov committed 1 year ago

Before this patch we've had ADMIN_ID in schema.rs and ADMIN_USER_ID
in lib.rs. This patch removes ADMIN_USER_ID in favor of ADMIN_ID

Dmitry Rodionov authored 1 year ago and Yaroslav Dynnikov committed 1 year ago

This patch adds initiator field to most of the events. Initiator is
added to members of Op enum which do not contain it already (e g owner
in CreateUser serves as initiator). To me it is reasonable to remove
as_user from CaS/ACL checks because it matches initiator. But I decided
not to do this just yet.

Additionally I've mande small cleanup to UserMetadata to avoid need for
intermediate step in decoding when we first fetch tuple and then decode
it into model struct.

Close https://git.picodata.io/picodata/picodata/picodata/-/issues/433

This change allows for creators of corresponding objects to consequently have all privileges on them.

Kurdakov Alexander authored 1 year ago and Yaroslav Dynnikov committed 1 year ago

pico.whoami() returns new field tier

Transferred replication factor from Properties table to new Tier table

Instance and Replicaset belongs to tier via new field tier in _pico_instance and _pico_replicaset

New cli option init-cfg with path to configuration file in yaml format. For now contains only info about tiers.

BREAKING CHANGE!:
- pico.trace() function was removed. Use pico.sql() for tracing;
- opentelemetry tables __SBROAD_STAT and __SBROAD_QUERY were renamed
  into _SQL_STAT and _SQL_QUERY tables.

Introduce enrypoints for extended PG protocol:

- pg_bind: bind parameters to portal (IR) by descriptor;
- pg_close: remove portal from the storage by descriptor;
- pg_describe: get the portal metadata and type by descriptor;
- pg_execute: execute portal by descriptor;
- pg_parse: parse an SQL pattern into IR (reusing LRU),
  save IR into portal storage and return the descriptor;
- pg_portals: get a list of portal descriptors visible
  to the current user;

Co-authored-by: Kaitmazian Maksim <m.kaitmazian@picodata.io>