Add new files:
- storage/snapshot.rs: stuff related to raft snapshot
- storage/schema.rs: stuff related to DDL logic

Code is only moved, no changes to logic

Add alter system cluster_wal_max_size and cluster_wal_max_count parameters
which control the auto compaction of _raft_log system space based on
the size in bytes and number of tuples respectively.

It is now possible to write in migration `@_plugin_config.foo` where foo is
variable from plugin configuration. Note that configuration typically
belongs to individual services, in catalog this field is called entity. For
migration special values there is special entity called `migration_context`.
So corresponding alter looks this way:

ALTER PLUGIN p 0.1.0 SET migration_context.foo = '"nondefault"'

(extra quotes are temporary, and should be dealt with separately)

Closes https://git.picodata.io/picodata/picodata/picodata/-/issues/659

godzie44 authored 6 months ago and Georgy Moshkin committed 5 months ago

Closes #664

Move vshard related fields from _pico_property to _pico_tier
and adapt it in governor algorithm

Maksim Kaitmazian authored 6 months ago and Georgy Moshkin committed 6 months ago

A global lock for schema changes that solves the following issue:
An expensive index creation can cause RPC timeouts, resulting in re-sending of the
same RPC even though the operation is still in progress on the master. The second RPC
attempts to create the same index, but it realizes that the index already exists
even if the operation has not been completed yet. To handle this scenario, a global
lock was added in `proc_apply_schema_change` to prevent concurrent schema changes.

- Move all preconditions checking to the client
- Return errors if preconditions fail instead of silently ignoring them

- Don't ignore operation if plugin is already enabled,
  the client must take care of this via the CaS ranges

- Don't propose mutliple Raft operations during one step as this may
  lead to inconsistencies in case of crashes

godzie44 authored 8 months ago and Maksim Kaitmazian committed 7 months ago

Closes #651