Add new files:
- storage/snapshot.rs: stuff related to raft snapshot
- storage/schema.rs: stuff related to DDL logic

Code is only moved, no changes to logic

Add alter system cluster_wal_max_size and cluster_wal_max_count parameters
which control the auto compaction of _raft_log system space based on
the size in bytes and number of tuples respectively.

Close #764

It is now possible to write in migration `@_plugin_config.foo` where foo is
variable from plugin configuration. Note that configuration typically
belongs to individual services, in catalog this field is called entity. For
migration special values there is special entity called `migration_context`.
So corresponding alter looks this way:

ALTER PLUGIN p 0.1.0 SET migration_context.foo = '"nondefault"'

(extra quotes are temporary, and should be dealt with separately)

Closes https://git.picodata.io/picodata/picodata/picodata/-/issues/659

Up to this patch `instance_uuid` and `replicaset_uuid` were generated
using name-based UUID v3. So all instances named `i1` in all clusters
had the same UUID.

With this patch UUIDs become random v4. Particular UUID values are
generated in two locations:

1. `start_boot` (src/lib.rs)
2. `handle_join_request_and_wait` -> `build_instance` (src/rpc/join.rs)

Function `Instance::new()` is removed in favor of explicit struct
construction.

godzie44 authored 6 months ago and Georgy Moshkin committed 5 months ago

Closes #664

Move vshard related fields from _pico_property to _pico_tier
and adapt it in governor algorithm

There's still some raft operations which we can't switch to
compare_and_swap yet unfortunately.

Maksim Kaitmazian authored 6 months ago and Georgy Moshkin committed 6 months ago

A global lock for schema changes that solves the following issue:
An expensive index creation can cause RPC timeouts, resulting in re-sending of the
same RPC even though the operation is still in progress on the master. The second RPC
attempts to create the same index, but it realizes that the index already exists
even if the operation has not been completed yet. To handle this scenario, a global
lock was added in `proc_apply_schema_change` to prevent concurrent schema changes.