Since commit d87dd4ca `leader_id` became an `Option`, so the `None`
value isn't rendered in the `picolib.raft_status` response:

```python
status={'is_ready': False, 'raft_state': 'Follower', 'id': 1}
```

It makes pytest complain about missing argument:

```
cluster2 = Cluster("127.0.0.1:3300", n=2)

    def test_restart_leader(cluster2: Cluster):
        i1, i2 = cluster2.instances
        i1.assert_raft_status('Leader')
        i2.assert_raft_status('Follower')

        i1.restart()
>       i1.wait_ready()

test/int/test_joining.py:209:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../.local/share/virtualenvs/picodata-6sv6l6y-/lib/python3.10/site-packages/funcy/decorators.py:45:
in wrapper
    return deco(call, *dargs, **dkwargs)
../../.local/share/virtualenvs/picodata-6sv6l6y-/lib/python3.10/site-packages/funcy/flow.py:127:
in retry
    return call()
../../.local/share/virtualenvs/picodata-6sv6l6y-/lib/python3.10/site-packages/funcy/decorators.py:66:
in __call__
    return self._func(*self._args, **self._kwargs)
test/int/conftest.py:305: in wait_ready
    status = self._raft_status()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = Instance(i1, listen=127.0.0.1:3301)

    def _raft_status(self) -> RaftStatus:
        status = self.call("picolib.raft_status")
        assert isinstance(status, dict)
        eprint(f"{status=}")
>       return RaftStatus(**status)
E       TypeError: RaftStatus.__init__() missing 1 required positional
argument: 'leader_id'

test/int/conftest.py:280: TypeError
```

This patch fixes the failure message:

```
self = Instance(i1, listen=127.0.0.1:3301)

    @funcy.retry(tries=20, timeout=0.1)
    def wait_ready(self):
        status = self._raft_status()
>       assert status.is_ready
E       AssertionError: assert False
E        +  where False = RaftStatus(id=1, raft_state='Follower',
is_ready=False, leader_id=None).is_ready

test/int/conftest.py:306: AssertionError
```

* Make `--cluster-id` CLI mandatory.
* Handle cluster_id mismatch in raft_join.
  When an instance attempts to join the cluster and the instances's
  `--instance-id` parameter mismatches the cluster_id of the cluster
  an error is raised inside the raft_join handler.

Previously the discovery algorithm would try to reach each known peer
sequentially requiring each consequent request to succeed until the next
one can be attempted. This would not work in some cases (see test in
previous commit).

So the new algorithm instead makes a single attempt to reach each peer
within a round, and if some failed they're retried in the next round of
requests. This allows overall discovery to succeed in cases when some
of the initial peers never respond.

Closes #54

The `peer_address` parameter is an inbound address used for
communication with the peer. It shouldn't be confused with the listen
address. The persisted `peer_address` may become obsolete due to
circumstances beyond picodata control (e.g. DNS or IP changes). Thus
there's no point in its prior validation, including the uniqueness
check.

There's also no such task as getting peer by peer_address.

To sum up, an index over `peer_address` is useless. It only creates
problems and causes panics.

Close https://git.picodata.io/picodata/picodata/picodata/-/issues/88

Georgy Moshkin authored 2 years ago and Georgy Moshkin committed 2 years ago

since 1.61 rust has added a `whole-archive` modifier for link targets
which is unset by default.

Previously to this version this was probably undefined behavior, because
the linking would work fine when building an executable (like picodata)
but would fail with undefined symbols when building a static library.

But now with the advent of this option, we can probably refactor
picodata into a library which can be useful for testing, i.e. we can
exclude testing stuff from the binary but still have a test binary with
all the necessary runtime symbols.

Closes #81

Pytest has a feature to segregate setup, test, and teardown logs. The
setup phase is considered to be an intialization of the fixtures. In
order to split logs properly `cluster.deploy()` is now called inside a
fixture.

Picodata already assigns `replicaset_id` to an instance when it joins,
but it wasn't used in Tarantool `box.cfg` yet. Now it is.

It's also important to set up listen port in `start_join` immediately.
Without it Tarantool will stuck waiting for connection to self.

Part of https://git.picodata.io/picodata/picodata/picodata/-/issues/52

It's already formatted in conformity to usual `cargo test`.
Also, remove unused command-line arguments from `picodata test` command.

Close https://git.picodata.io/picodata/picodata/picodata/-/issues/61

The intention is to eliminate ambiguities in the `Instance` API. Make
it more like `subprocess` module (as regards `kill` and `terminate`
functions).

Behavior of `killpg` slightly differs in Mac and Linux. For some reason,
`killpg` returns error EPERM when sending a signal to a zomibie process.
And that is the reason of `test_process_management` failure on mac -
there's a small gap between killing child and and subreaper calls
`waitpid`.

Now pytest handles this exception properly.

Close https://git.picodata.io/picodata/picodata/picodata/-/issues/70

See also:

- Stackoverflow: Why would `killpg` return "not permitted" when ownership is correct?
  https://stackoverflow.com/questions/12521705/why-would-killpg-return-not-permitted-when-ownership-is-correct

- Linux `man 2 killpg`:
  https://linux.die.net/man/2/killpg#Notes

  > Notes
  >
  > There are various differences between the permission checking in
  > BSD-type systems and System V-type systems. See the POSIX rationale
  > for kill(). A difference not mentioned by POSIX concerns the return
  > value EPERM: BSD documents that no signal is sent and EPERM returned
  > when the permission check failed for **at least one** target process,
  > while POSIX documents EPERM only when the permission check failed for
  > **all** target processes.

- MacOS `man 2 killpg`:
  https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/killpg.2.html

  > [EPERM] The sending process is not the super-user and
  >         **one or more** of the target processes has an effective
  >         user ID different from that of the sending process.

- Linux `man 2 kill`:
  https://linux.die.net/man/2/kill

  >  EPERM  The process does not have permission to send the signal
  >         *to any* of the target processes.
  >

- Process states in Linux:
  https://kerneltalks.com/linux/process-states-in-linux/

- Reproduce killpg returning EPERM on MacOS:
  https://git.picodata.io/picodata/picodata/picodata/-/snippets/7

Pytest supports running tests in parallel using the `xdist` plugin. In
order to support it in Picodata, one should avoid ports collision. It
assigns each worker a dedicated IP address `127.7.n.1`, where
`n = xdist_worker_number`. Unfortunately, it doesn't work on MacOS,
because Mac doesn't provide any loopback aliases except `127.0.0.1`
by default.

This patch provides another address generation logics. The `subnet`
parameter is superseeded with a `base_port`, that is `3300 + n * 100`.
In this way, every pytest (xdist) worker gets a dedicated port range
`[3301, 3399]`, `[3401, 3499]` and so on.

Close https://git.picodata.io/picodata/picodata/picodata/-/issues/65

When bootstrapping an instance, there're two possible execution paths -
`start_boot` and `start_join`. While `start_join` takes all uuids from
JoinResponse, `start_boot` already deals with a bootstrapped `box.cfg`
(it's done in `start_discover`, refer to [1]). In order to make uuids
consistent across `box.cfg` and topology module, `start_boot` stage is
preceded with rebootstrap.

This case is also covered with a pytest.

- [1] doc/clustering.md

- Add corresponding field to the Peer struct.
- Generate it in the topology module.
- Use it in `box.cfg`.

Close https://git.picodata.io/picodata/picodata/picodata/-/issues/51

Address `replication_factor` when choosing `relicaset_id` for a new
instance. It dosn't consider `failure_domain` yet, but takes into
account the number of instances.

Close https://git.picodata.io/picodata/picodata/picodata/-/issues/68

- Choose it in the topology module if it's not provided in a
  `JoinRequest`.
- Persist in `raft_group` space.
- Respond with an error if `JoinRequest` contains different
  `replicaset_id`.
- In `JoinResponse` it's transferred automatically.

Part of https://git.picodata.io/picodata/picodata/picodata/-/issues/51

Follow-up for https://git.picodata.io/picodata/picodata/picodata/-/issues/50

- Generate it in the topology module
- Persist it in `raft_group` space
- Transfer it in `JoinResponse`
- Use it in `box.cfg`

Close https://git.picodata.io/picodata/picodata/picodata/-/issues/50

Alexander Tolstoy authored 2 years ago and Yaroslav Dynnikov committed 2 years ago

Part of https://git.picodata.io/picodata/picodata/docs/-/issues/1

It's necessary to incapsulate topology management logics away from main.

It encapsulates the logics of a JoinRequest batch processing.

Topology module will be quite important in picodata. This first version
misses a lot of features, but a few commits later it's going to
implement quite a lot of logics.

When a new instance is joined - there's one complex thing: raft leader
has to decide where this new instance is going to be emplaced, i.e. what
replicaset should it join. There're many different parameters have an
influence - `repliction_factor`, `failure-domain`, and of course the
existing topology. So, this new `topology` module must make the decision.

This patch only refactors the current Picodata behavior, and doesn't
bring new features for its users. Instead, it opens the door to a future
development.

Also, this patch provides a unit-testing basis for the future features.

Both JoinRequest and JoinResponse are going to be used in other modules.
Move them one level above from `traft::node::*` to `traft::*`.

One of the most tricky Raft cases is a so-called ABA problem [1].
In that case it's important to protect a batch of join requests with a
term number. Since the whole batch is supplied with atomicity-sensitive
uuids, applying it on a different term may cause an inconsistency, which
is very, veeeery bad.

[1] https://en.wikipedia.org/wiki/ABA_problem

Implementation of `net_box` in `tarantool-module` resolves hostnames
with a `to_socket_addrs` function that is blocking. Pytest uses
fake addresses in one test, and sometimes it results in 5-second
blockage and consequent test failure.

This patch only provides a workaround. It makes a connection to fail
even before the blocking DNS request.

See also:
- https://git.picodata.io/picodata/picodata/tarantool-module/-/issues/81