Partitioning by key and consumer-side position

Diagram

The Keyed Partitioner

The client uses a key to determine which partition a record belongs to before the record is sent. As described in README.md:19-21, the TopicHandle hashes the key and sends the broker a partition number; the key itself never reaches the wire. This ensures that all events for a specific user are routed to the same partition, allowing a single service to handle them in order.

FileOffsetStore

To ensure that a consumer's position survives a process restart, the FileOffsetStore implements OffsetStore by writing the position to a file on a volume. As detailed in README.md:30-32, this is achieved through a temporary file and an atomic move to prevent corruption. This mechanism ensures that a "half-saved" position is not written, which would otherwise cause silent replays of truncated offsets.

At-least-once Checkpointing

The system guarantees at-least-once delivery by managing the lifecycle of a batch carefully. According to README.md:35-37, checkpointing saves the offset only after the collector has successfully handled the batch. If a failure occurs between the handling of the batch and the saving of the offset, the batch will be replayed upon restart.

Recovery after a crash

When a consumer restarts, it uses its persisted position to resume reading. The resumedFrom field in the consumer's /stats endpoint provides visibility into this process (README.md:52-53). As noted in check.py:14-16, resuming slightly behind the last known position is considered legal behavior because the at-least-once guarantee allows for the replay of a batch that was handled but not yet checkpointed.

The split assertion

The integrity of the partitioning and consumption is validated by an external check. The check.sh script triggers a validation where check.py asserts that every record sent by the publisher reaches exactly one consumer (README.md:44-47). The split mode in check.py:25-38 verifies that the number of records written to a partition matches the position held by the consumer assigned to that partition.

Key files

Behaviour that surprises

• Silent Replays: Because checkpointing happens after processing, a consumer might start from a position slightly behind its last successful work, causing it to replay the last batch (README.md:36-37).
• Key Erasure: A Kafka key does not survive the crossing into booblik; while it is used to pick the partition on the way in, it is not stored in the booblik wire format and cannot be recovered on the way back (README.md:177-180).
• Non-monotonicity in partitionFor(null): Calling partitionFor(null) advances an internal counter, meaning that checking which partition a record would go to before actually sending it can cause records to skip partitions (README.md:204-206).