Protocol Constants and Versioning

Diagram

Protocol Constants

The wire format relies on fixed-size headers and strict limits to ensure efficient parsing and resource protection. The following table summarizes the core constants defined in Protocol.kt:15-58:

ApiKey and Versioning Support

The protocol uses an ApiKey to identify the type of request, and versioning is handled per request rather than globally. As specified in Protocol.kt:50-57, the supports function determines compatibility:

• PRODUCE: Supports only VERSION (1).
• FETCH: Supports VERSION (1) and FETCH_VERSION (2).
• METADATA: Supports only VERSION (1).

ErrorCode and Request Failure Modes

Errors are communicated via the ErrorCode enum, which maps to specific protocol violations. The behavior of the correlationId during a failure is critical for pipelined clients, as detailed in ErrorCodeTest.kt:15-20:

RequestDecoder and Frame Validation

The RequestDecoder is responsible for transforming raw bytes into structured Request objects. It performs rigorous bounds-checking to ensure that a remote party cannot cause memory issues, as seen in Requests.kt:79-132. Key validation mechanics include:

• Header Integrity: The correlationId is read first so it can be echoed even if the body is corrupt (Requests.kt:93-96).
• Unsatisfiable Constraints: In decodeFetch, if minBytes > maxBytes, a CorruptRequestException is thrown because the request can never be satisfied (Requests.kt:209-210).
• Positive Counts: recordCount in PRODUCE and topicCount in METADATA must be non-negative and must not exceed the remaining bytes in the buffer (Requests.kt:149, Requests.kt:176).

Partial Frame Assembly and Socket Resilience

The protocol is designed to handle fragmented network delivery. PartialFrameTest.kt:29-40 demonstrates that a frame split across multiple packets (e.g., sending the length prefix, waiting, then sending the tail) is correctly assembled by the session. Furthermore, the protocol ensures connection resilience: if a frame is well-formed but contains an invalid ApiKey or apiVersion, the broker responds with an error but keeps the connection open (ErrorCodeTest.kt:32-41).

Frame Length Limits and Resource Protection

To prevent Denial of Service (DoS) attacks where an attacker sends a massive int32 length prefix to force large memory allocations, the broker enforces a strict limit. As tested in PartialFrameTest.kt:112-129, if a frame length exceeds the allowed bounds, the broker drops the connection immediately rather than attempting to allocate the requested memory.

Key files

Behaviour that surprises

• CorrelationId Zeroing: If a frame is so short that the correlationId cannot be parsed, the broker returns 0 as the ID in the error response (ErrorCodeTest.kt:78-81).
• Zero-Copy and CRC: The broker does not calculate the CRC for FETCH responses because it uses transferTo to move data directly from the disk to the socket, bypassing the heap (protocol-wire.md:154-156).
• Unsatisfiable Fetch: A FETCH request where minBytes is greater than maxBytes is rejected during decoding as a CORRUPT_REQUEST because it is logically impossible to satisfy (Requests.kt:209-210).