Panic on graceful shutdown

Steps to reproduce:

• Deploy a cluster of 4 instances (but I guess 2 is enough).
• Stop them all
• Start one instance. It'll stuck with no quorum.
• Stop it gracefully.

Result:

thread 'main' panicked at 'leader_id deinitialized', src/traft/failover.rs:37:42

Full log:

2022-07-26 00:11:35.684 [34630] main/103/interactive I> >>>>> postjoin()
2022-07-26 00:11:35.684 [34630] main/103/interactive I> leaving orphan mode
2022-07-26 00:11:35.684 [34630] main/103/interactive I> set 'replication_connect_quorum' configuration option to 0
2022-07-26 00:11:35.685 [34630] main/103/interactive I> switched to configuration, config: Configuration { voters: Configuration { incoming: Configuration { voters: {3} }, outgoing: Configuration { voters: {} } }, learners: {4, 1, 2}, learners_next: {}, auto_leave: false }, raft_id: 1
2022-07-26 00:11:35.685 [34630] main/103/interactive I> became follower at term 5, term: 5, raft_id: 1
2022-07-26 00:11:35.685 [34630] main/103/interactive I> newRaft, peers: Configuration { incoming: Configuration { voters: {3} }, outgoing: Configuration { voters: {} } }, last term: 5, last index: 45, ap
plied: 45, commit: 45, term: 5, raft_id: 1
2022-07-26 00:11:35.685 [34630] main/103/interactive I> RawNode created with id 1., id: 1, raft_id: 1 

(it's stuck because it has no quorum)
^C

[supervisor:34629] got signal 2
thread 'main' panicked at 'leader_id deinitialized', src/traft/failover.rs:37:42
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
fatal runtime error: failed to initiate panic, error 5
[supervisor:34629] subprocess finished: Ok(Signaled(Pid(34630), SIGABRT, true))

$ ttc 3301
connected to localhost:3301
localhost:3301> box.space.raft_group:fselect()
---
- - ​+-----------+--------------------------+-------+------------+-------------+-------------------------+------------+---------+---------------+
  - ​|instance_id|      instance_uuid       |raft_id|peer_address|replicaset_id|     replicaset_uuid     |commit_index| health  |failure_domains|
  - ​+-----------+--------------------------+-------+------------+-------------+-------------------------+------------+---------+---------------+
  - ​|   "i1"    |"68d4a766-4144-3248-aeb4-e|   1   | "i1:3301"  |    "r1"     |"e0df68c5-e7f9-395f-86b3-|     42     |"Offline"| {"DC":"MSK"}  |
  - ​|   "i2"    |"24c4ac5f-4981-3441-879c-a|   2   | "i2:3301"  |    "r2"     |"eff4449e-feb2-3d73-87bc-|     44     |"Offline"| {"DC":"MSK"}  |
  - ​|   "i3"    |"5d7a7353-3e82-30fd-af0d-2|   3   | "i3:3301"  |    "r3"     |"1e94f382-b6a4-3ab8-ac07-|     31     |"Online" | {"DC":"MSK"}  |
  - ​|   "i4"    |"826cbe5e-6979-3191-9e22-e|   4   | "i4:3301"  |    "r1"     |"e0df68c5-e7f9-395f-86b3-|     41     |"Offline"| {"DC":"MSK"}  |
  - ​+-----------+--------------------------+-------+------------+-------------+-------------------------+------------+---------+---------------+
...

changed milestone to %Release 0.2 - Sharding MVP

Repro (fcaaec46):

pytest test/rand/ --seed FE9q4cdJ

pytest test/rand --seed FE9q4cdJ --delay 1000 покажет SUCCESS

pytest test/rand --seed FE9q4cdJ --delay 200 покажет FAIL

delay - это задержка между действиями (стоп/старт процессов)

mentioned in merge request !199 (merged)

assigned to @gmoshkin

mentioned in merge request !205 (merged)

closed with merge request !205 (merged)