Yaroslav Dynnikov authored 2 years ago and Yaroslav Dynnikov committed 2 years ago

Bootstrapping the replication was implemented in 2dac77c5. But it was
configured on the new instance only. The old instance (that joined
earlier) couldn't update `box.cfg({replication})` until now.

Close https://git.picodata.io/picodata/picodata/picodata/-/issues/52

Just supply it with a default value "demo". No new tests are necessary,
we already have `test/int/test_joining.py::test_cluster_id_mismatch`.

Close https://git.picodata.io/picodata/picodata/picodata/-/issues/96

If proc_discover is invoked after raft node was initialized but before
raft leader was elected, it would return an error before this commit.
Because of that it was impossible to restart the whole cluster at once.

This commit change proc_discover such that in case leader_id is not
ready, the normal discovery algorithm takes place.

Closes #93

* Make `--cluster-id` CLI mandatory.
* Handle cluster_id mismatch in raft_join.
  When an instance attempts to join the cluster and the instances's
  `--instance-id` parameter mismatches the cluster_id of the cluster
  an error is raised inside the raft_join handler.

Previously the discovery algorithm would try to reach each known peer
sequentially requiring each consequent request to succeed until the next
one can be attempted. This would not work in some cases (see test in
previous commit).

So the new algorithm instead makes a single attempt to reach each peer
within a round, and if some failed they're retried in the next round of
requests. This allows overall discovery to succeed in cases when some
of the initial peers never respond.

Closes #54

The `peer_address` parameter is an inbound address used for
communication with the peer. It shouldn't be confused with the listen
address. The persisted `peer_address` may become obsolete due to
circumstances beyond picodata control (e.g. DNS or IP changes). Thus
there's no point in its prior validation, including the uniqueness
check.

There's also no such task as getting peer by peer_address.

To sum up, an index over `peer_address` is useless. It only creates
problems and causes panics.

Close https://git.picodata.io/picodata/picodata/picodata/-/issues/88

Picodata already assigns `replicaset_id` to an instance when it joins,
but it wasn't used in Tarantool `box.cfg` yet. Now it is.

It's also important to set up listen port in `start_join` immediately.
Without it Tarantool will stuck waiting for connection to self.

Part of https://git.picodata.io/picodata/picodata/picodata/-/issues/52

It's already formatted in conformity to usual `cargo test`.
Also, remove unused command-line arguments from `picodata test` command.

Close https://git.picodata.io/picodata/picodata/picodata/-/issues/61

When bootstrapping an instance, there're two possible execution paths -
`start_boot` and `start_join`. While `start_join` takes all uuids from
JoinResponse, `start_boot` already deals with a bootstrapped `box.cfg`
(it's done in `start_discover`, refer to [1]). In order to make uuids
consistent across `box.cfg` and topology module, `start_boot` stage is
preceded with rebootstrap.

This case is also covered with a pytest.

- [1] doc/clustering.md

- Add corresponding field to the Peer struct.
- Generate it in the topology module.
- Use it in `box.cfg`.

Close https://git.picodata.io/picodata/picodata/picodata/-/issues/51

Address `replication_factor` when choosing `relicaset_id` for a new
instance. It dosn't consider `failure_domain` yet, but takes into
account the number of instances.

Close https://git.picodata.io/picodata/picodata/picodata/-/issues/68

- Choose it in the topology module if it's not provided in a
  `JoinRequest`.
- Persist in `raft_group` space.
- Respond with an error if `JoinRequest` contains different
  `replicaset_id`.
- In `JoinResponse` it's transferred automatically.

Part of https://git.picodata.io/picodata/picodata/picodata/-/issues/51

- Generate it in the topology module
- Persist it in `raft_group` space
- Transfer it in `JoinResponse`
- Use it in `box.cfg`

Close https://git.picodata.io/picodata/picodata/picodata/-/issues/50

It's necessary to incapsulate topology management logics away from main.

It encapsulates the logics of a JoinRequest batch processing.

Topology module will be quite important in picodata. This first version
misses a lot of features, but a few commits later it's going to
implement quite a lot of logics.

When a new instance is joined - there's one complex thing: raft leader
has to decide where this new instance is going to be emplaced, i.e. what
replicaset should it join. There're many different parameters have an
influence - `repliction_factor`, `failure-domain`, and of course the
existing topology. So, this new `topology` module must make the decision.

This patch only refactors the current Picodata behavior, and doesn't
bring new features for its users. Instead, it opens the door to a future
development.

Also, this patch provides a unit-testing basis for the future features.

Both JoinRequest and JoinResponse are going to be used in other modules.
Move them one level above from `traft::node::*` to `traft::*`.

One of the most tricky Raft cases is a so-called ABA problem [1].
In that case it's important to protect a batch of join requests with a
term number. Since the whole batch is supplied with atomicity-sensitive
uuids, applying it on a different term may cause an inconsistency, which
is very, veeeery bad.

[1] https://en.wikipedia.org/wiki/ABA_problem

Yaroslav Dynnikov authored 2 years ago and Yaroslav Dynnikov committed 2 years ago

This patch covers one more case when discovery request is handled
by an instance that has the discovery module unitialized.

By default cargo runs tests in parallel in multiple threads.  Both
`test_log_level` and `test_parse` access environment variables which are
shared across threads. Consequently, their concurrent modification
results in the test failure.

This patch unites these two tests making it linear.

Close https://git.picodata.io/picodata/picodata/picodata/-/issues/28

1. Lower log level of connection errors in `netork.rs`.
2. Give raft fibers a name.

There were some problems with join requests synchronization. Raft
forbids proposing a configuration change if there's another one
uncommitted (see [1]). In that case, it replaces an `EntryConfChange`
with an `EntryNormal`. It could happen at any time even without bugs in
code due to the network partitioning, and its the repsonsibility of
the picodata product to handle it properly.

Earlier, there was no way to wait when raft leaves the joint state. It
used to slow down cluster assembling and made it race-prone. The waiting
for the cluster readiness is also important in tests. Some operations
(the most important amongst them is leader switching) are impossible
until instance finishes promotion to a voter. For instance, raft rejects
`MsgTimeoutNow` unless the node is promotable (see [2]). It makes some
testing scenarios flaky.

This patch introduces new synchronization primitive - `JointStateLatch`.
The latch is held on the leader and is locked upon
`raw_node.propose_conf_change()`. It's unlocked only when the second
(implicit) conf change that represents leaving joint state is committed.
The latch also tracks the index of the corresponding `EntryConfChange`.
Even if raft ignores it for any reason, the latch is still unlocked as
soon as the committed index exceeds the one of the latch.

[1] https://github.com/tikv/raft-rs/blob/v0.6.0/src/raft.rs#L2014-L2026
[2] https://github.com/tikv/raft-rs/blob/v0.6.0/src/raft.rs#L2314

Close https://git.picodata.io/picodata/picodata/picodata/-/issues/47
Close https://git.picodata.io/picodata/picodata/picodata/-/issues/53

Waiting for a valid `leader_id` on a node isn't enough. It may already
have one, but still be a Learner. Instead, the fixture should wait until
the node is promoted to voter.