Remove the corresponding calls, add module-level doc-comments with
explanations on what the hell is going on.

This also fixes some flaky tests including #1294

Turns out blocking inside proc_replication_demote is not safe, as it may
lead to deadlocking if a replicaset master switchover happens during
a replicaset catch-up via raft snapshot. The symptom of the problem was
the flakiness of the tests (for example issue #1294).

Closes #1294

This reverts commit ec2b760a.

Added a `wait_index` parameter to `proc_replication_demote` to improve
synchronization during demotion operations.

Diana Tikhonova authored 3 months ago and Diana Tikhonova committed 2 months ago

Key updates include:
Adding the term field to relevant RPC requests (ConfigureReplicationRequest, ReplicationSyncRequest, and DemoteRequest).
Utilizing node.status().check_term(req.term) in critical sections of the replication logic.

This patch changes when we update current_master_name during consistent
master switchover. Now it is updated as soon as we determine that the
new master has synchronized it's vclock with the old master.

As a result we can now block expel of a non-last replica in cases where
other replicas are offline, which is crucial because instances can
easily become offline due to temporary network issues.

Nobody cares about the lsn

godzie44 authored 11 months ago and Yaroslav Dynnikov committed 11 months ago

Introduce a plugin system for picodata cluster.
Plugin system contains:
- `picoplugin` crate with plugin interfaces.
This crate exposes plugin interfaces for user implementation and API
that used by picodata to interact with plugin as with a code from dynamic library.
- Support for plugin work cycle in picodata instance work cycle.
Picodata instance interacts with running plugin by `PluginManager`
component.
Typically, plugins react to events that occur in instance or whole cluster.
- New lua API with plugin CRUD actions.
    - pico.create_plugin - load a new plugin into picodata cluster
    - pico.remove_plugin - remove plugin from picodata cluster
    - pico.update_plugin_config - update plugin service configuration

Georgy Moshkin authored 1 year ago and Yaroslav Dynnikov committed 1 year ago

This used to cause flakyness of test_bucket_rebalancing_respects_replication_factor
like this:

- governor proposes entry at index N with weight updates,
  and waits till it's applied

- commit index is not updated yet, but governor calls rpc::sharding to
  update the weights in the vshard subsystem on instances and passes the
  lagging commit index for syncing

- peers sync up to the lagging commit index and then reconfigure vshard
  to the outdated weights

- governor thinks everything is in order and waits for updates

- the test fails because buckets don't get rebalanced to the storages
  with weights set to 0

Georgy Moshkin authored 2 years ago and Yaroslav Dynnikov committed 2 years ago

Tarantool synchronous replication is broken in such a way, that local
& asynchronous transactions get rolled back upon synchronous
transaction rollback. This is critical for us, because we use local
spaces for the raft log, which should be orthogonal to synchronous
business spaces, but is not due to this behavior. Until it is fixed, we
will not use synchronous replication.

See https://github.com/tarantool/tarantool/issues/7903

Cases covered:
- when someone goes offline
- whenever vshard gets reconfigured