fix: used to persist applied index before commit index

Summary

• fix: don't compact unapplied entries

• test: implement basic error injection & add some tests for main loop

• fix: used to persist applied index before commit index

This could result in a broken instance if it terminated in between the two storage writes. Now we always persist the commit index before applying the persisted entries and do so in a transaction with unstable entries or snapshot if they are present.

• fix: minor tweaks

• refactor: extract prepare_for_snapshot function

• refactor: use a watch channel instead of Event::EntryApplied

Close #418 (closed)

Ensure that

• New code is covered by tests
• API is documented
• Changelog is up to date
• (if Lua API changed) Lua API version is bumped in luamod.rs
• (if necessary) A follow-up doc issue is created in picodata/docs and linked here

src/traft/node.rs

@@ -1645,6 +1645,8 @@ impl NodeImpl {
                 tlog!(Warning, "dropping raft light ready: {light_rd:#?}");
                 panic!("transaction failed: {e}");
             }
+
+            crate::error_injection!(block "BLOCK_AFTER_RAFT_PERSISTS_COMMIT_INDEX");
         }
 
         // Apply committed entries.