fix: bug in raft snapshot application

Summary

• fix: bug in raft snapshot application

There was a hard-to-reproduce bug in our snapshot application code. We always compact the raft log before applying the snapshot, because the snapshot replaces the entries and some of the logic in raft-rs seems to rely on this. The problem was, that our compact_log function would not remove any unapplied entries, which makes sense for compaction triggered automatically by raft log size, but doesn't make sense for raft snapshot, as the snapshot contains the state corresponding to the newer entries. The fix is simple: don't guard from unapplied entry compaction in case the compaction is for raft snapshot.

We don't add any regression tests for this, because the implementation would be too difficult and would need us to pollute the code with error injection logic, which is not a worthy trade off in this case. But also the logic will still be tested, because this bug was responsible for a large amount of flaky tests, so we should see a significant reduction in flakiness from now on in tests concerning raft snapshots.

• test: reduce test_large_snapshot flakiness

---

• Close #1167 (closed), #779 (closed)
• Cherry-pick to: none
• Docs follow-up: not necessary

CONTRIBUTING.md

@@ -169,6 +169,18 @@ poetry run pytest -k test_sql_acl
 poetry run pytest -n 20
 ```
 
+#### Debugging tests
+
+There are a few tricks that help in debugging python tests.
+
+First one is to use NOLOG env var like that: `NOLOG=1 pytest ...`. `NOLOG` removes instance log output printed on test failure.
+This is useful when you're debugging the test itself and instance logs just pollute the output.
+
+Second tip is to use debugger, namely `ipdb`. You just need to place `import ipdb; ipdb.set_trace()` on the line and
+run pytest with `-s` argument to avoid stdout interception. This will open a REPL when you run the test, which is a convenient
+way to look around, run arbitrary code, etc. It is more productive compared to endlessly rerunning the test with various
+print statements.
+
 #### Running manual/test_scaling.py::test_cas_conflicts
 
 This test is not ran by default, but can be used for benchmarking instance join time and cas conflicts.