Producer and Consumer Mechanics

Diagram

The Producer and the Connection Ownership

A Producer maintains exclusive ownership of its Connection to ensure that responses are matched correctly to requests in the order they were sent (README.md:44-45). Because a second writer on the same socket would take someone else's answer, the Producer is the sole writer to that socket (README.md:44-45). It is critical to use DisposeAsync to flush queued records; failing to do so results in silent data loss during shutdown (README.md:46-47).

Batching Strategies: Manual vs Producer

Performance is heavily dependent on batching, as single-record requests are significantly less efficient than batches of a hundred (README.md:22-23).

AckPolicy.None and the Unknown Offset

When using AckPolicy.None, the broker does not return an empty response, but rather nothing at all, because no offset exists until the writer reaches the batch (README.md:51-52). In this mode:

• ProduceAsync returns null (README.md:52).
• A Producer completes with Producer.OffsetUnknown (README.md:52).
• The broker may drop an accepted record silently (README.md:53-54).

Consumer Position and IAsyncEnumerable Back-pressure

Reading is implemented via IAsyncEnumerable, which provides back-pressure by construction: the next fetch does not occur until the current loop body is finished (README.md:81-82). The Position property represents the number to be persisted; it is the responsibility of the caller to persist this after records are dealt with to ensure a restart re-delivers rather than skips data (README.md:89-90).

Record Fragmentation and MaxBytes

A consumer may encounter a stall if a record is larger than MaxBytes. In such cases, the record is never delivered whole, and a RecordExceedsMaxBytesException is thrown (README.md:100). Additionally, a response can stop inside a record if MaxBytes cuts on a byte boundary; the fragment is dropped, and the Position stops before it, requiring the next poll to request that record from its start (README.md:102-103).

The Claims Log and Worker Coordination

In a task queue built on top of the log, the order of a specific partition acts as the arbiter for task acquisition (Main.kt:39-40). Workers write claims into a claims topic, and the first claim on a free task wins (Main.kt:39-40). This process involves a round-trip cost: a worker must write the claim and then read the log until the claim comes back to settle the task (Main.kt:187-188).

The Write Actor and Group Commit

High-throughput durable writes are achieved through group commits, where a single barrier (operation) can service multiple producers (benchmarking.md:263-264). While a single producer is limited by the frequency of barriers (approx. 250/s), 64 producers can achieve over 7,500 durable writes per second by sharing the same disk barrier (benchmarking.md:270-273).

Key files

Behaviour that surprises

• AckPolicy.None does not return an empty response; it returns nothing, making it impossible to know if a record was accepted (README.md:51).
• The key is not sent to the broker; the client uses the key to pick a partition and sends only the partition number (README.md:55-56).
• BrokerException is a result of a declined request (framing was intact), not a connection outage (README.md:59).