Layer 1: Partitioned event stream with consumer-side position

Diagram

The Key-to-Partition Mapping

The client is responsible for the routing logic. Every event is associated with a key (such as a user ID), and the TopicHandle.partitionFor function hashes this key to determine a specific partition number (Main.kt:79). Crucially, the key itself is never sent over the wire; only the resulting partition number is transmitted in the protocol (README.md:20). This ensures that all events for a single user land in the same partition, guaranteeing that a single consumer handles them in the correct order (README.md:21).

FileOffsetStore and Atomic Position Persistence

To ensure that a consumer's position survives a crash, the FileOffsetStore implements persistence using a file on a dedicated volume (FileOffsetStore.kt:28). To prevent data corruption during a crash, the save operation does not overwrite the existing file directly. Instead, it writes the new offset to a temporary file and then performs an atomic move using StandardCopyOption.ATOMIC_MOVE (FileOffsetStore.kt:57). This prevents a "half-saved" position, which would be parsed as a truncated number and cause silent replay of multiple records (FileOffsetStore.kt:24).

At-least-once Guarantee and Checkpointing

The system provides an at-least-once delivery guarantee through the specific ordering of the checkpointing operation. The consumer handles the batch of records first and only saves the offset to the OffsetStore after the collector has successfully processed the batch (Main.kt:83). If a crash occurs between the handling of the records and the saving of the offset, the consumer will replay the batch upon restart (README.md:36).

Recovery and the resumedFrom Field

When a consumer restarts, it attempts to resume from its last known position. It uses StartPosition.At(offset) if a saved offset exists, otherwise it defaults to StartPosition.Earliest to read from the beginning of the live log (Main.kt:68). The Stats class tracks this via the resumedFrom field, which is populated with the saved offset if a restart occurs, allowing for visibility into whether the consumer is starting from a historical point or from the beginning (Main.kt:69).

Validation via check.sh

The correctness of the consumer's recovery logic is verified by check.sh. The test suite considers it legal for a consumer to resume slightly behind its last known position, as this is a natural consequence of the at-least-once guarantee (README.md:37). However, the test will fail if the consumer attempts to start from the very beginning of the log when it should have resumed from a specific offset (README.md:38).

Key files

Behaviour that surprises

• Silent Replay: Because of the way FileOffsetStore handles file writes, a truncated file (e.g., "12" becoming "1") is still a valid long, which causes the consumer to silently replay eleven records (FileOffsetStore.kt:24).
• Key Erasure: While the client uses a key to select a partition, the key itself is not part of the booblik wire protocol and is not stored or transmitted (README.md:20).
• Earliest vs. Zero: Using StartPosition.Earliest does not necessarily mean starting from offset zero; it means starting from the beginning of the live log, which depends on the current retention settings (Main.kt:65).