dev

What this module is responsible for

The dev module provides a comprehensive demonstration of the booblik ecosystem through four architectural layers, ranging from basic partitioned event streams to complex Kafka-to-booblik relaying. It includes the necessary orchestration via docker compose and a suite of validation scripts to assert the correctness of the implementation.

Diagram

Layer 1: Partitioned event stream with consumer-side position

This layer demonstrates how a publisher can distribute events across partitions using a key to ensure all events for a specific user land in the same partition, as described in README.md:19-22. The consumer is responsible for its own state, using FileOffsetStore to persist its position in a file on a volume to ensure at-least-once delivery guarantees (README.md:30-33). This mechanism ensures that if a process restarts, it can resume from its last known position, though it may replay a small number of records if the offset was not saved atomically (README.md:35-38).

More: Layer 1: Partitioned event stream with consumer-side position

Layer 2: Task queue via claims log arbitration

This layer implements a coordination-free task queue where workers compete for tasks by appending claims to a claims topic (README.md:68-70). Instead of a central coordinator, the order of the log acts as the arbiter; the first claim to land in a partition wins the lease (README.md:69-70). A lease is determined by a timestamp written into the claim itself, ensuring that workers with clock skew still reach the same conclusion about whether a lease has expired (README.md:72-75). This design avoids the need for a central coordinator but results in a growing claims log and requires workers to read every claim (README.md:88-92).

Layer 3: Log-based projection and state reconstruction

The projection layer treats state as a pure function of the log, where the service stores nothing and instead builds its view by replaying the log (README.md:129-131). The lifecycle of a projection is managed through two distinct phases: replay() which catches up to the high watermark, and follow() which maintains the view as new events arrive (README.md:133-136). To prevent data corruption, the service does not persist its position separately from the state; instead, it rebuilds the entire view from the beginning to ensure consistency (README.md:141-144).

Layer 4: Kafka-to-booblik relaying

This layer provides bidirectional translation between Kafka and the booblik protocol, acting as a bridge between ecosystems (README.md:164-166). The relay is a single module that changes its behavior based on the environment, specifically regarding who is responsible for remembering the position:

Build and environment configuration

The project uses Gradle with Kotlin DSL, where subprojects are configured to maintain a consistent style using ktlint (build.gradle.kts:13-18). A critical configuration detail is that repositories { } blocks defined in subprojects will replace, rather than append to, the repositories defined in settings.gradle.kts due to the PREFER_PROJECT mode (build.gradle.kts:21-24). Additionally, the Docker images are explicitly built for the amd64 platform to avoid manifest errors on arm64 hosts (README.md:209-210).

Key files

Behaviour that surprises

• Position Persistence: The Projection service does not persist its position because persisting a position without the state it belongs to can lead to silent corruption; it must rebuild everything from the log to be certain of its state (README.md:141-144).
• Task Redistribution: In the queue implementation, a task is not "released" by a worker shutting down gracefully; instead, the task becomes claimable again simply because the lease in the log expires (check-redistribution.sh:7-9).
• Repository Overriding: Declaring repositories { } in a subproject does not add to the parent repositories but replaces them, which can lead to "Could not find" errors if the parent repositories are lost (build.gradle.kts:21-24).