fix: used to persist applied index before commit index

Summary

• fix: don't compact unapplied entries

• test: implement basic error injection & add some tests for main loop

• fix: used to persist applied index before commit index

This could result in a broken instance if it terminated in between the two storage writes. Now we always persist the commit index before applying the persisted entries and do so in a transaction with unstable entries or snapshot if they are present.

• fix: minor tweaks

• refactor: extract prepare_for_snapshot function

• refactor: use a watch channel instead of Event::EntryApplied

Close #418 (closed)

Ensure that

• New code is covered by tests
• API is documented
• Changelog is up to date
• (if Lua API changed) Lua API version is bumped in luamod.rs
• (if necessary) A follow-up doc issue is created in picodata/docs and linked here

src/traft/error.rs

@@ -12,10 +12,10 @@ use thiserror::Error;
 pub enum Error {
     #[error("uninitialized yet")]
     Uninitialized,
-    #[error("events system is uninitialized yet")]
-    EventsUninitialized,
     #[error("timeout")]
     Timeout,
+    #[error("current instance is expelled from the cluster")]
+    Expelled,
     #[error("{0}")]
     Raft(#[from] raft::Error),
     #[error("downcast error: expected {expected:?}, actual: {actual:?}")]