TopologyCache is a collection of deserialized structures with
information about cluster topology. This currently includes data from
_pico_instance, _pico_replicaset, _pico_tier & _pico_service_route_table.
The info is automatically kept up to date with the corresponding system
tables. The TopologyCache also caches the immutable info related to the
current instance, like instance name, replicaset uuid, etc.

From now on we should be reading this data from TopologyCache whenever
possible instead of going directly to system tables as we were doing
previosly.

At the moment only the plugin RPC module has transitioned to using
TopologyCache but other modules should be refactored as well.
Especially governor.

Add new files:
- storage/snapshot.rs: stuff related to raft snapshot
- storage/schema.rs: stuff related to DDL logic

Code is only moved, no changes to logic

Add alter system cluster_wal_max_size and cluster_wal_max_count parameters
which control the auto compaction of _raft_log system space based on
the size in bytes and number of tuples respectively.

picoplugin -> picodata-plugin
picoplugin_gen -> picodata-plugin-proc-macro

Note that after this merge request, we no longer validate whether the format
of the created object matches with the schema. Consequently, we do not
suppress errors if formats match anymore.

Before:
```sql
CREATE USER u WITH PASSWORD 'Passw0rd'
1
CREATE USER u WITH PASSWORD 'Passw0rd'
0
CREATE USER u WITH PASSWORD 'DifferentPassw0rd'
error: user already exists with different auth method
```

Now:
```sql
CREATE USER u WITH PASSWORD 'Passw0rd'
1
CREATE USER u WITH PASSWORD 'Passw0rd'
error: user u already exists
CREATE USER u WITH PASSWORD 'DifferentPassw0rd'
error: user u already exists
CREATE USER IF NOT EXISTS u WITH PASSWORD 'DifferentPassw0rd'
0
```

CAS predicates should always contain the current raft term, because this
is what's explicitly checked in proc_cas.

Note that this will sometimes result in a EntryTermMismatch error
because the latest applied entry may have a different term in case the
election has started but not finished yet.

We could add this check to all the client cas precondition checks, but
it's not a big deal, because we handle the situation correctly anyways.

- Simplify code
- Check plugin exists on client side

- Move all preconditions checking to the client
- Return errors if preconditions fail instead of silently ignoring them

Dmitry Rodionov authored 7 months ago and Yaroslav Dynnikov committed 7 months ago

Previously function was ~700 loc. The patch splits out the logic of
conversion from ir node to operation. Previously there were two stages,
initial validation that constructed intermediate structure and then
intermediate structure was transformed into Op potentially several times
in case of retries. The patch removes intermediate conversion by
direct mapping from ir node to op. It makes reading more convenient
since there is no need to jump back and forth between initial checks and
actual Op construction. Another benefit is that some checks need to be
repeated on each retry since retries originate from conflicts and
conflicts change system state. So in newer state operation may no longer
make sense. By coupling both operations together there is no way a check
can be missed in the retry phase.

We still do some cloning here and there. We can probably improve on that
by constructing Op once and then adjusting it in place for conflicts but
since retries should be rare in practice it shouldnt be a problem.

Additionally we do a lot of conversion between SmolStr and String. We
need to use SmolStr in more placese inside picodata so we dont need to
do that.

godzie44 authored 8 months ago and Maksim Kaitmazian committed 7 months ago

Closes #651

godzie44 authored 8 months ago and Maksim Kaitmazian committed 7 months ago

Closes #584

godzie44 authored 8 months ago and Maksim Kaitmazian committed 7 months ago

Closes #682

feature(plugin migration): use md5 from migration files meta for to confirm consistency of migrations between versions

Closes #652

- add `migration_up` and `migration_down` lua API functions
- now the migration mechanism in plugins takes into account the plugin version and, accordingly, previous applied migrations
- add _pico_plugin_migration system table

Closes #683

- use RouteBuilder to register rpc endpoints (input and ouptut is raw
  bytes for now)
- use RequestBuilder to send RPC requests to other instances which are
  running a specified plugin.service (guarded by version)
    - supported routing by
        - explicit instance_id
        - replicaset_id
        - bucket_id
        - any available instance with given service
- use context::Context to extract information about request or add
  custom fields at runtime (middleware not yet implemented)

1. make it non-nullable
2. fix tests