Producer Configuration and Batching

Generated page

Model gemma-mtp, commit ef58254ca7be, 2026-08-16, sources: 6. Edit the code or the hand-written documentation instead.

ProducerConfig

The configuration for the producer determines how the accumulator behaves.

Parameter	Type	Description
`maxBatchSize`	`int`	The maximum number of records per request; reaching this triggers an immediate send (`Producer.kt:23`).
`lingerMillis` / `linger`	`long` / `float`	The time a batch waits for more records; zero is not the fast setting, as it sends every record individually (`ProducerConfig.java:15`).
`ack` / `ackPolicy`	`AckPolicy`	Determines the acknowledgment behavior (`producer.py:32`).

The accumulator is the core performance driver of the client. A single record per request is highly inefficient; for example, batches of a hundred can reach 4,335,482 records/s compared to only 80,592 for single records (README.md:37). The accumulation loop uses a windowing logic where the deadline is calculated from the arrival of the first record in a batch, not the last, to prevent a steady trickle of records from postponing a send indefinitely (Producer.kt:139).

AckPolicy.NONE

When AckPolicy.NONE is selected, the client does not wait for a broker response to confirm the record's existence. Because no offset is assigned until the broker's writer reaches the batch, the client returns OFFSET_UNKNOWN (or Offset.ZERO in some implementations) to remain honest about the state of the record (Producer.kt:205).

The `batch` mechanism

The batch function provides a way to bypass the accumulator entirely. This is used to send a group of records as a single request, guaranteeing that the records land contiguously in the partition with consecutive offsets (Publishing.kt:74).

The `close` lifecycle and `_fail_leftovers`

During shutdown, the producer must ensure no records are left hanging. The close method flushes queued data, and if any records remain in the mailbox after the loop has terminated, _fail_leftovers (or drainPending) is called to fail those records with an exception so the caller is not left waiting forever (producer.py:108).

Race conditions in the accumulation loop

A critical edge case occurs when a timeout and a new record arrival happen simultaneously. Using a standard receive with a timeout can cause a "cancelled receive" that swallows a record, leaving the caller's CompletableDeferred hanging (Producer.kt:143). The implementation uses select to ensure that the mailbox either takes the element or takes the timeout, but never both or neither, preventing data loss (Producer.kt:154).

ProducerLostRecordTest

This test verifies the integrity of the accumulator by simulating a scenario where two different topics with different arrival rates are sent through a single producer (ProducerLostRecordTest.kt:52). It ensures that the accumulator does not drop records when the timing of a record arrival coincides with the expiration of the linger window.

Key files

File	Lines	What is there
`…/java/ProducerConfig.java`	`14-16`	Default producer configuration factory.
`…/client/Producer.kt`	`139-146`	Logic regarding the accumulation window and `select` usage.
`…/client/Publishing.kt`	`74-79`	Guarantees regarding contiguous offsets in a `batch`.
`…/booblik/producer.py`	`108-112`	Handling of leftover records during producer closure.

Behaviour that surprises

Producer.send is not immediate; it returns a handle to a result that will only be completed once the accumulator decides to flush the batch (Producer.kt:75).
AckPolicy.NONE results in OFFSET_UNKNOWN because the broker has not yet assigned an offset to a record that hasn't been processed by the writer (producer.py:20).
Producer.batch is not an atomic transaction; a crash during the write can result in a partial batch surviving on disk (Publishing.kt:76).