dev/queue-worker

What this module is responsible for

The dev/queue-worker module implements a distributed task queue protocol built on top of a log-based broker. Instead of relying on a centralized coordinator with complex locking mechanisms, this module uses a single partition of a claims topic as a deterministic arbiter. Workers compete for tasks by writing claims to the log; the order of these records in the log determines the winner.

Diagram

ClaimRecord and the Claims Log

The coordination relies on a sequence of ClaimRecord entries written to a specific topic. These records define the lifecycle of a task through two primary types: CLAIM and DONE (Claims.kt:24-25). Each record contains the worker's identity, the task ID, and a timestamp (at) representing the worker's wall clock at the time of the claim (Claims.kt:18-19).

More: ClaimRecord and the Claims Log

ClaimState and the Deterministic Verdict

The core logic of the queue is encapsulated in ClaimState, which acts as a pure function that folds over the log to produce a consistent view of task ownership (Claims.kt:57). A key design feature is that leases are judged using the timestamp written into the ClaimRecord itself, rather than the reader's local clock (Claims.kt:50-51). This ensures that all workers replaying the same log reach the exact same verdict regarding which worker holds a lease (Claims.kt:76-85).

More: ClaimState and the Deterministic Verdict

Worker Lifecycle and the Work Loop

The work function manages the continuous loop of task acquisition and execution (Main.kt:151-217). The lifecycle follows these steps:

1. Identify: Find claimable tasks by checking the current ClaimState (Main.kt:171).
2. Claim: Write a CLAIM record to the claimsTopic (Main.kt:178).
3. Settle: Wait for the claim to appear in the log by observing consumedUpTo (Main.kt:192).
4. Execute: Perform the work for the duration of workMillis (Main.kt:207).
5. Complete: Write a DONE record to the log (Main.kt:210).

More: Worker Lifecycle and the Work Loop

Worker Metrics and Health Monitoring

Each worker exposes an HTTP server via Ktor to provide observability into its internal state (Main.kt:68). The available endpoints are:

Queue Report and Observability

The Report object is a standalone tool used to audit the entire queue by replaying the claims and tasks topics from the beginning (Report.kt:22). It calculates the "wasted" attempts—claims that lost the race—to measure queue efficiency (Report.kt:82). It also performs a critical safety check: if the number of DONE records does not match the number of distinct tasks completed, it flags an error indicating a task was worked twice (Report.kt:98).

Key files

Behaviour that surprises

• The claimable function uses the local clock (now) to determine if a lease has lapsed, but the actual ownership verdict is determined by the timestamps stored within the ClaimRecord itself (Claims.kt:101).
• The apply function in ClaimState is designed to be a pure function, ensuring that the "verdict" is a deterministic result of the log order, regardless of when a worker reads it (Claims.kt:63).
• In Main.kt, a worker must wait for its own claim to "settle" (appear in the log) before it begins work, meaning the worker's view of the world is always slightly behind its own actions (Main.kt:185).