The Fetch mechanism

Diagram

FetchRequest

The FetchRequest struct defines the parameters that govern how a consumer requests data from a partition consumer.go:94-112. The request includes the target Topic, Partition, and the starting Offset.

A critical distinction exists between MaxBytes and the size of individual records: MaxBytes limits the total response size, which can cause a response to end in the middle of a record consumer.go:100-101.

Fetched

A successful response is encapsulated in the Fetched struct consumer.go:115-129. It contains the HighWatermark, which represents the first offset that does not yet exist in the log. The Records field is a slice of byte slices that point directly into the response frame without copying the data, meaning the caller must copy any data they wish to retain beyond the lifetime of the frame consumer.go:118-120.

Truncation and MaxBytes

Because MaxBytes limits the response size in bytes, a response may end inside a record. This is indicated by the Truncated flag in the Fetched struct consumer.go:123-125. When this occurs, the TruncatedRecordBytes field provides the size of the record that was cut off, allowing the client to understand why the response was incomplete consumer.go:127-128.

CorruptRecordError

The client is the only party capable of detecting data corruption on the read path because the broker uses zero-copy to stream segment bytes directly to the socket without inspecting them consumer.go:41-43. The client verifies each record using CRC-32C (Castagnoli) via the Checksum function consumer.go:36. If the computed checksum does not match the stored checksum, a CorruptRecordError is returned, which includes the Offset, the Stored checksum, and the Computed checksum consumer.go:57-69.

RecordExceedsMaxBytesError

A permanent stall occurs if a single record is larger than the consumer's MaxBytes limit. In this scenario, the broker returns a truncated response, and the client identifies this as a RecordExceedsMaxBytesError consumer.go:54-55. This error carries the Offset, the RecordBytes required, and the current MaxBytes limit to inform the caller that they must increase their buffer to proceed consumer.go:74-85.

Consumer Position and Lag

The Consumer manages the position (the offset of the next record to be read) and the highWatermark consumer.go:243-250. The position is only advanced past whole records during a Poll operation consumer.go:347. Lag is calculated as the difference between the highWatermark and the current position, representing how many records the consumer is behind the head of the log consumer.go:306-310.

Records Iterator

The Records method provides a range-over-func iterator that yields records one at a time consumer.go:376-377. This iterator runs on the caller's goroutine at the caller's pace, avoiding the issues of buffered channels such as leaking goroutines or silently advancing the position past unhandled records consumer.go:366-371.

Key files

Behaviour that surprises

• Poll only advances the position past whole records; if a response is truncated, the position remains at the start of the partial record to allow for a retry with a larger MaxBytes consumer.go:347.
• The Records iterator does not use a channel; it executes on the caller's goroutine to ensure that the position is not advanced until the caller has actually processed the records consumer.go:371.