AckPolicy

What this module is responsible for

The AckPolicy defines the durability contract between a producer and the broker. It determines how long a producer waits for an acknowledgement and what level of guarantee is provided regarding the persistence of the data. This policy is processed by the PartitionWriter, which manages the lifecycle of a write from the moment it enters the mailbox until it is either written to the log or flushed to disk.

Diagram

AckPolicy modes

The durability promises are represented as a single byte on the wire and are defined in AckPolicy.kt:21-25:

The PartitionWriter write loop and group commit

The PartitionWriter uses an actor-like model where a single coroutine owns the write side of a partition. The runLoop function in PartitionWriter.kt:163-196 implements a group commit mechanism. Instead of processing one message at a time, it takes the first command and then drains all other currently queued messages from the mailbox without suspending. This allows the writer to aggregate multiple WriteCommand objects into a single group, ensuring that if any command in the group requires a durability guarantee, the cost of the disk barrier is shared.

The distinction between WRITTEN and FORCED

The distinction lies in the execution of the disk barrier. As seen in PartitionWriter.kt:178-184, the writer iterates through the group and checks if any command's policy is AckPolicy.FORCED. If so, it triggers flushNow(), which calls log.force(). In contrast, AckPolicy.WRITTEN only ensures the bytes are appended to the log segment, and as verified in PartitionWriterTest.kt:96-97, it must not trigger a force operation.

AckPolicy.NONE and the fire-and-forget mechanism

When AckPolicy.NONE is used, the append function in PartitionWriter.kt:135-149 returns null immediately. This is because the offset is not known until the writer coroutine actually processes the batch in the runLoop. Because no CompletableDeferred is created for the acknowledgement, the producer receives no reply, making it a true fire-and-forget operation.

Verification of offset continuity and batching

The system ensures that offsets are assigned in a strict, gap-free sequence. Tests in PartitionWriterTest.kt:45-55 verify that a batch of records results in a single base offset and that subsequent records follow consecutively. Furthermore, tests in PartitionWriterTest.kt:129-155 confirm that even with many concurrent producers, the resulting offsets are unique and form a continuous range from 0 to N.

Testing group commit efficiency

The efficiency of the group commit is validated by simulating high contention. In PartitionWriterTest.kt:105-126, multiple producers send FORCED requests simultaneously. The test asserts that the number of actual force calls (disk barriers) is significantly lower than the number of producers, proving that the PartitionWriter successfully collapses multiple requests into a single barrier.

Key files

Behaviour that surprising

• PartitionWriter.append returns null for AckPolicy.NONE because the offset is not determined until the command is processed by the actor, as noted in PartitionWriter.kt:130-133.
• The highWatermark is updated only once per group of writes, rather than per record, to avoid unnecessary writes on the hot path (PartitionWriter.kt:92-95).
• A FORCED policy in a group will trigger a single log.force() for all members of that group, meaning the latency of the barrier is shared across all producers in that batch (PartitionWriter.kt:37-43).