ProduceAsync and AckPolicy

Diagram

AckPolicy

The AckPolicy enumeration defines how the client waits for confirmation from the broker:

Connection.cs:8-22

ProduceAsync mechanics

The ProduceAsync method constructs a binary payload containing the topic name, partition ID, the chosen AckPolicy, and the list of records. The records are written to the partition such that they land contiguously, meaning one request results in a sequence of offsets from BaseOffset to LogEndOffset with no interleaving from other requests Connection.cs:183-213.

The write actor and batching

Performance is heavily dependent on how records are grouped. While Topic.SendAsync allows sending a single record, doing so one at a time is inefficient compared to using Connection.ProduceAsync with a list of records Connection.cs:382-387. In the Kotlin implementation, the batch function provides a way to collect records into a BatchScope to ensure they are sent as a single request, bypassing the standard accumulator to maintain the guarantee that records land contiguously Publishing.kt:85-99.

Failure modes and BrokerException

The client handles various broker responses. If a batch is empty or contains empty records, the broker may return a CORRUPT_REQUEST Connection.cs:178-180. If the broker refuses a request (e.g., UNKNOWN_TOPIC_OR_PARTITION), a BrokerException is thrown to the caller Connection.cs:131. Under AckPolicy.None, ProduceAsync returns null because no response is expected from the broker Connection.cs:214-216.

Recovery after a crash

Writes are not atomic. If a crash occurs mid-write, the system is designed so that recovery keeps the prefix of the batch that passed its checksums, meaning a partial batch can survive on its own Publishing.kt:76-79.

Connection integrity tests

The test suite ensures high reliability through several checks:

• Byte-for-byte delivery: Verifying that records arrive exactly as sent ConnectionTests.cs:24-30.
• Error propagation: Ensuring that a broker refusal is treated as a BrokerException but does not close the connection, allowing for reuse ConnectionTests.cs:56-64.
• Protocol integrity: Confirming that a truncated response (e.g., due to a broker restart) results in a ProtocolException ConnectionTests.cs:100-103.

Key files

Behaviour that not obvious

• Connection.ProduceAsync returns null when AckPolicy.None is used, which can be interpreted as "no answer is coming" Connection.cs:214-216.
• A BrokerException does not necessarily close the Connection; if the framing remains intact, the connection remains usable for subsequent requests ConnectionTests.cs:60-64.
• Topic.SendAsync is a convenience method that wraps a single record into a list, which is significantly slower than batching multiple records in one ProduceAsync call Connection.cs:382-387.