There was a hard-to-reproduce bug in our snapshot application code.
We always compact the raft log before applying the snapshot, because
the snapshot replaces the entries and some of the logic in raft-rs
seems to rely on this. The problem was, that our compact_log function
would not remove any unapplied entries, which makes sense for compaction
triggered automatically by raft log size, but doesn't make sense for
raft snapshot, as the snapshot contains the state corresponding to the
newer entries. The fix is simple: don't guard from unapplied entry
compaction in case the compaction is for raft snapshot.

We don't add any regression tests for this, because the implementation
would be too difficult and would need us to pollute the code with error
injection logic, which is not a worthy trade off in this case. But also
the logic will still be tested, because this bug was responsible for a
large amount of flaky tests, so we should see a significant reduction in
flakiness from now on in tests concerning raft snapshots.

Maksim Kaitmazian authored 3 months ago and Maksim Kaitmazian committed 3 months ago

Close #1228

TopologyCache is a collection of deserialized structures with
information about cluster topology. This currently includes data from
_pico_instance, _pico_replicaset, _pico_tier & _pico_service_route_table.
The info is automatically kept up to date with the corresponding system
tables. The TopologyCache also caches the immutable info related to the
current instance, like instance name, replicaset uuid, etc.

From now on we should be reading this data from TopologyCache whenever
possible instead of going directly to system tables as we were doing
previosly.

At the moment only the plugin RPC module has transitioned to using
TopologyCache but other modules should be refactored as well.
Especially governor.

Before this fix if requesting RPC by an invalid bucket_id we would send
an RPC to every replicaset to check if they have such a bucket_id.
This is not needed because we know the allowed range.

this adr is a proposal of implementing TRUNCATE sql command, its context and possible problems

- added the ability to build a base image manually

Виталий Шунков authored 3 months ago and Dmitry Rodionov committed 3 months ago

- combined the base image of picodata and sbroad
- refactor CI rules
- add RUST_VERSION and TARANTOOL_VERSION as variables in CI