Recovery after a crash
Generated page
Model gemma-mtp, commit f508a4b65b3f, 2026-08-15, sources: 6. Edit the code or the hand-written documentation instead.
Diagram
FileOffsetStore and atomic position updates
To prevent corruption of the consumer's state, FileOffsetStore avoids overwriting the current offset file directly. Instead, it writes the new offset to a temporary file and uses an atomic move to replace the original, ensuring that a crash during the write process does not leave a truncated or corrupted offset file (FileOffsetStore.kt:55-57).
LogSegment recovery and index rebuilding
When a broker restarts, it must reconstruct its internal lookup structures. The LogSegment.open function allows the system to reopen existing segments, verifying that the nextOffset and the index are consistent with the data stored on disk (RecoveryTest.kt:37-38).
SegmentMode and data integrity boundaries
The behavior of the recovery depends on the SegmentMode used:
| Mode | Recovery Behavior |
|---|---|
FILE_CHANNEL | The file length acts as a boundary; a record with a header claiming more bytes than the file contains is discarded (RecoveryTest.kt:78-79). |
MAPPED | The file length is pre-sized, so recovery relies on the presence of a zero-length prefix to identify the end of the log (RecoveryTest.kt:100-101). |
PartitionLog segment continuity
A PartitionLog is composed of multiple segments. Upon reopening, the system restores the continuity of the partition by identifying all existing segments and maintaining their order, ensuring the nextOffset reflects the total count across all segments (RecoveryTest.kt:117-118).
The at-least-once guarantee in consumer and relay
The system provides at-least-once delivery by ensuring that the position is only updated after the work is completed. In consumeForever, the checkpointing operator saves the offset only after the collect block (which includes handle) has finished (Main.kt:83-84). Similarly, the relay ensures that Kafka is flushed before the booblik position is updated (Directions.kt:119).
Validation of resumed positions
The system distinguishes between a successful recovery and a failure based on the starting point. A consumer is considered to have failed if it starts from the beginning of the log (README.md:37-38), whereas resuming from a position slightly behind the last known offset is considered a valid replay of a batch (check.py:54-55).
Key files
| File | Lines | What is there |
|---|---|---|
…/common/FileOffsetStore.kt | 55-57 | The atomic move mechanism using a temporary file. |
…/consumer/Main.kt | 83-84 | The checkpointing logic that ensures at-least-once delivery. |
…/relay/Directions.kt | 119 | The requirement to flush Kafka before checkpointing. |
Behaviour that surprises
FileOffsetStore.saveuses a.tmpfile to ensure that a crash during a write doesn't leave a "half-saved" position that could cause silent replays (FileOffsetStore.kt:55-57).- In
booblikToKafka, the relay does not store the Kafka key; while it preserves per-key ordering by using the key to pick the booblik partition, the key itself is lost during the crossing (Directions.kt:115-116). - The
check.pyscript considers a consumer "resumed" even if it starts slightly behind its last known position, because replaying a batch is the expected behavior of an at-least-once system (check.py:54-55).