Specifying the particular directory with tests
significantly speeds up their collection and execution.

Before:

```console
$ time pytest -k nothing
12 deselected in 1.36s
```

After:

```
12 deselected in 0.04s
```

Yaroslav Dynnikov authored 2 years ago and Yaroslav Dynnikov committed 2 years ago

This patch covers one more case when discovery request is handled
by an instance that has the discovery module unitialized.

By default cargo runs tests in parallel in multiple threads.  Both
`test_log_level` and `test_parse` access environment variables which are
shared across threads. Consequently, their concurrent modification
results in the test failure.

This patch unites these two tests making it linear.

Close https://git.picodata.io/picodata/picodata/picodata/-/issues/28

Commit 1a3b5233 missed a bug. Iteration over instances could be aborted
by an exception during teardown. It resulted in garbage process
remaining alive after pytest termination.

1. Lower log level of connection errors in `netork.rs`.
2. Give raft fibers a name.

There were some problems with join requests synchronization. Raft
forbids proposing a configuration change if there's another one
uncommitted (see [1]). In that case, it replaces an `EntryConfChange`
with an `EntryNormal`. It could happen at any time even without bugs in
code due to the network partitioning, and its the repsonsibility of
the picodata product to handle it properly.

Earlier, there was no way to wait when raft leaves the joint state. It
used to slow down cluster assembling and made it race-prone. The waiting
for the cluster readiness is also important in tests. Some operations
(the most important amongst them is leader switching) are impossible
until instance finishes promotion to a voter. For instance, raft rejects
`MsgTimeoutNow` unless the node is promotable (see [2]). It makes some
testing scenarios flaky.

This patch introduces new synchronization primitive - `JointStateLatch`.
The latch is held on the leader and is locked upon
`raw_node.propose_conf_change()`. It's unlocked only when the second
(implicit) conf change that represents leaving joint state is committed.
The latch also tracks the index of the corresponding `EntryConfChange`.
Even if raft ignores it for any reason, the latch is still unlocked as
soon as the committed index exceeds the one of the latch.

[1] https://github.com/tikv/raft-rs/blob/v0.6.0/src/raft.rs#L2014-L2026
[2] https://github.com/tikv/raft-rs/blob/v0.6.0/src/raft.rs#L2314

Close https://git.picodata.io/picodata/picodata/picodata/-/issues/47
Close https://git.picodata.io/picodata/picodata/picodata/-/issues/53

Waiting for a valid `leader_id` on a node isn't enough. It may already
have one, but still be a Learner. Instead, the fixture should wait until
the node is promoted to voter.

The assertion `status == "Leader"` was in the first place, and
`raft_timeout_now` call was unreachable.

1. Print logs to the stderr so that they interleave with tarantool logs.
2. Fix `cluster.__repr__()`.

1. Review `Pipfile`:
  - Remove unused `filelock`;
  - Install `mypy` - static type checker for Python.
2. Add new command `pipenv run lint`.
3. Enable `mypy` in CI. Fix reported errors in `test_basics.py`.
4. Renew readme.

In `conftest.py`:

  - *Add* function `xdist_worker_number`. It converts `str(worker_id)` into `int`.
    It serves as a substitute for `session_data_mutex` for parallel test runs.

  - *Change* `normalize_net_box_result`. Replace a function with a decorator. Also,
    handle typical kinds of picodata responses. Extensively test it in
    `test_basics.py::test_eval/call_normalization`.

  - *Add* dataclass `RaftStatus`. It shouldn't be used outside `conftest.py`. It only makes
    assertions more brief in logs and understndable in code.

  - *Add* all raft stuff into the `Instance` class. This implies `raft_propose_eval`,
    `assert_raft_status` all `promote_or_fail` moved from `util.py`.

  - *Change* fixture `compile`. No need in extra logics since commit 59c31cb8.

  - *Preserve* fixture `binary_path`.

  - *Remove* fixtures `session_data_mutex` and `run_id`. Superseded with `xdist_worker_number`.

  - *Remove* fixtures `run_cluster` and `run_instance`. Superseded with `cluster.deploy(...)`.

  - *Remove* function `wait_tcp_port`. It's never enough to check raw socket. Superseded with
    `instance.wait_ready()`.

  - Give the instances clean names `i1, i2, ...`, and simple addresses `127.7.0.1:3301`.
    For the parallel test run use different IPs `127.7.N.1` etc.

In `test_basics.py`:

  - *Add* `test_xdist_worker_number`.

  - *Add* `test_call_normalization` and `test_eval_normalization`.

  - *Add* `test_process_management`. It's brand new, never implemented in luatest before.

  - *Rename* `test_single_instance_raft_eval` to `test_propose_eval` and extend it with additional
    assertion from `single_test.lua`.

  - *Remove* `test_instance`. A part of its logics is moved to `test_call/eval_normalization`. The
    other part is rewritten and extended in `test_process_management`.

  - *Remove* `test_cluster`. It was completely useless because of inappropriate synchronization and
    no valuable assertions.

  - *Remove* `test_propose_eval`. It wasn't that useful, but failed because of inappropriate
    synchronization.

In `test_couple.py`:

  - *Preserve* `test_follower_proposal` and `test_failover`. Just slightly refactor according to the
    new `conftest.py` API.

In `util.py` (completely removed):

  - *Remove* decorator `retry`. Needless.

  - *Remove* decorator `retry_on_network_errors`. Inappropriate predicate didn't catch Lua errors.

  - *Remove* everything related to raft. Move it into `conftest.py`.

Part of https://git.picodata.io/picodata/picodata/picodata/-/issues/59

We've recently noticed many files in `vendor/` dir have windows-like EOL
`crlf`. When cloning the repo, git produces a huge diff due to the EOL
normalization built-in feature. This patch disbles it for the certain
path.

Close https://git.picodata.io/picodata/picodata/picodata/-/issues/56

We don't want a child process to live without the supervisor.

Usually, supervisor waits for child forever and retransmits termination
signals. But if the parent is killed with a SIGKILL there's no way to
pass anything.

This patch supplies a child process with a `supervisor_fuse` fiber. It
tries to read from a pipe (that supervisor never writes to), and if the
writing end is closed, it means the supervisor has terminated. In this
case, child process terminates too.

Part of https://git.picodata.io/picodata/picodata/picodata/-/issues/56

This patch reduces the variable passing trace length. Instead of
returning it from function to function, call `exit()` in place.

The trace of exit code before the patch:
```
exit() <- main <- main_run <- fork(child) <- tarantool_main
```

And now:
```
exit() <- tarantool_main
```

Type safety is ensured at comilation time by using `!` type for
`fn main()`. See https://doc.rust-lang.org/reference/types/never.html

```console
$ cargo clippy
   Compiling picodata v0.1.0 (/home/rosik/w/picodata)
warning: 'tarantool-sys/patches-applied' exists, so patching step is
skipped
warning: single-character string constant used as pattern
   --> src/args.rs:194:47
    |
194 |     let (host, port) = match text.rsplit_once(":") {
    |                                               ^^^ help: try using
a `char` instead: `':'`
    |
    = note: `#[warn(clippy::single_char_pattern)]` on by default
    = help: for further information visit
https://rust-lang.github.io/rust-clippy/master/index.html#single_char_pattern

warning: called `cloned().next()` on an `Iterator`. It may be more
efficient to call `next().cloned()` instead
   --> src/discovery.rs:154:27
    |
154 |                 let res =
    peers.difference(&self.visited).cloned().next();
    |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: try this:
`peers.difference(&self.visited).next().cloned()`
    |
    = note: `#[warn(clippy::iter_overeager_cloned)]` on by default
    = help: for further information visit
https://rust-lang.github.io/rust-clippy/master/index.html#iter_overeager_cloned

warning: you seem to be trying to use `match` for destructuring a single
pattern. Consider using `if let`
  --> src/ipc.rs:27:9
   |
27 | /         match rmp_serde::encode::write(&mut self.fd, msg) {
28 | |             Err(e) => tlog!(Error, "ipc error: {e}"),
29 | |             Ok(()) => (),
30 | |         }
   | |_________^ help: try this: `if let Err(e) =
rmp_serde::encode::write(&mut self.fd, msg) { tlog!(Error, "ipc error:
{e}") }`
   |
   = note: `#[warn(clippy::single_match)]` on by default
   = help: for further information visit
https://rust-lang.github.io/rust-clippy/master/index.html#single_match

warning: `picodata` (bin "picodata") generated 3 warnings
    Finished dev [unoptimized + debuginfo] target(s) in 1.41s
```

1. Remove it from functions args where it's not necessary.
2. In other places rename it into `supervisor_ipc`.

Since dynamic discovery isn't implemented yet, instance after
`raft_join` has discovery uninitialized. It's not a reason to panic.