Dmitry Rodionov authored 7 months ago and Yaroslav Dynnikov committed 7 months ago

Previously function was ~700 loc. The patch splits out the logic of
conversion from ir node to operation. Previously there were two stages,
initial validation that constructed intermediate structure and then
intermediate structure was transformed into Op potentially several times
in case of retries. The patch removes intermediate conversion by
direct mapping from ir node to op. It makes reading more convenient
since there is no need to jump back and forth between initial checks and
actual Op construction. Another benefit is that some checks need to be
repeated on each retry since retries originate from conflicts and
conflicts change system state. So in newer state operation may no longer
make sense. By coupling both operations together there is no way a check
can be missed in the retry phase.

We still do some cloning here and there. We can probably improve on that
by constructing Op once and then adjusting it in place for conflicts but
since retries should be rare in practice it shouldnt be a problem.

Additionally we do a lot of conversion between SmolStr and String. We
need to use SmolStr in more placese inside picodata so we dont need to
do that.

godzie44 authored 8 months ago and Maksim Kaitmazian committed 7 months ago

Closes #651

godzie44 authored 8 months ago and Maksim Kaitmazian committed 7 months ago

Closes #584

godzie44 authored 8 months ago and Maksim Kaitmazian committed 7 months ago

Closes #682

feature(plugin migration): use md5 from migration files meta for to confirm consistency of migrations between versions

Closes #652

- add `migration_up` and `migration_down` lua API functions
- now the migration mechanism in plugins takes into account the plugin version and, accordingly, previous applied migrations
- add _pico_plugin_migration system table

Closes #683

- use RouteBuilder to register rpc endpoints (input and ouptut is raw
  bytes for now)
- use RequestBuilder to send RPC requests to other instances which are
  running a specified plugin.service (guarded by version)
    - supported routing by
        - explicit instance_id
        - replicaset_id
        - bucket_id
        - any available instance with given service
- use context::Context to extract information about request or add
  custom fields at runtime (middleware not yet implemented)

1. make it non-nullable
2. fix tests

godzie44 authored 11 months ago and Georgy Moshkin committed 10 months ago

- add description field to `_pico_plugin` and `_pico_service`
- plugin loading: filter files by its extensions
- plugin loading: check possible service collisions
- plugin loading: now plugin searching at {plugin_dir}/{plugin_name} path
- now all errors between plugin and picodata wrapping with `BoxError` (this protects plugin caller from UAF)

godzie44 authored 1 year ago and Yaroslav Dynnikov committed 11 months ago

Now for installation and enable plugin, there are two steps instead of one
(`pico.load_plugin` previously).
1. Install plugin (`pico.install_plugin`):
validate plugin, create system tables.
2. Enable plugin (`pico.enable_plugin`):
loads plugin into instances memory, call `on_start` callback.

Same for a plugin removing process (`pico.remove_plugin` previously):
1. Disable plugin (`pico.disable_plugin`): call `on_stop` callbacks, drop
plugin from instances.
2. Remove plugin (`pico.remove_plugin`):
remove plugin meta from system tables.

BREAKING CHANGE: "name" index in the _pico_index has changed:
previously it guarantees uniqueness for a pair of (space_id,
index_name). Now index name must be unique among the cluster.
So, all the global indexes for _pico tables were renamed.

BREAKING CHANGE: _pico_index structure has changed.

The reason for refactoring is the request for the cluster wide
indexes with all possible options. The old structure didn't allow
us to support them all.

godzie44 authored 1 year ago and Yaroslav Dynnikov committed 11 months ago

Introduce a plugin system for picodata cluster.
Plugin system contains:
- `picoplugin` crate with plugin interfaces.
This crate exposes plugin interfaces for user implementation and API
that used by picodata to interact with plugin as with a code from dynamic library.
- Support for plugin work cycle in picodata instance work cycle.
Picodata instance interacts with running plugin by `PluginManager`
component.
Typically, plugins react to events that occur in instance or whole cluster.
- New lua API with plugin CRUD actions.
    - pico.create_plugin - load a new plugin into picodata cluster
    - pico.remove_plugin - remove plugin from picodata cluster
    - pico.update_plugin_config - update plugin service configuration

Make field 'auth' nullable

- Update sbroad submodule to commit with
grant/revoke procedure support
- Use existing mechanisms for grant/revoke
ACL opcodes for grant/revoke procedure
implementation
- Add integration tests
- Valid combinations:

all procedures: grant/revoke create/execute/drop
specific procedure: grant/revoke execute/drop

- Syntax:

grant create procedure to alice option(timeout=3)
grant execute procedure on "spam"(int) to "alex"
grant drop procedure on foo to bob
revoke drop procedure on foo from bob

- add new ddl opcode for renaming procedure
- update sbroad submodule to get sql support
- syntax:

alter procedure "foo" rename to "bar" option(timeout=3)
alter procedure "foo"(int, int) rename to "bar"

Maksim Kaitmazian authored 1 year ago and Denis Smirnov committed 1 year ago

Without this commit creation of a function for non-admin users fail with
`AccessDenied: Read access to space \'_pico_routine\' is denied` error.