clients/node

Diagram

BrokerError and ProtocolError

The module distinguishes between errors caused by the broker's logic and errors caused by the transport layer. A BrokerError is raised when the broker understands the request but declines it (e.g., UNKNOWN_TOPIC_OR_PARTITION), as defined in [errors.js:25-33](https://github.com/youndie/booblik/blob/ef58254ca7be0c2e8c83b5ee75d4ce32647cf800/clients/node/src/errors.js#L25-L33). In contrast, a ProtocolError is raised when the bytes on the connection do not make sense, such as a bad length or a lost socket, as seen in [errors.js:35-40`](https://github.com/youndie/booblik/blob/ef58254ca7be0c2e8c83b5ee75d4ce32647cf800/clients/node/src/errors.js#L35-L40).

More: BrokerError and ProtocolError

FNV-1a Partitioning

To ensure deterministic record placement, the client uses the FNV-1a hash algorithm to map keys to specific partitions. The fnv1a32 function in partition.js:28-35 implements this using Math.imul to ensure 32-bit integer wraparound, which is critical because standard JavaScript numbers are doubles. The partitionFor function then folds this hash into the available partition count partition.js:48-52.

More: FNV-1a Partitioning

CRC-32C Checksum Verification

Data integrity is maintained via the CRC-32C (Castagnoli) polynomial 0x1EDC6F41 crc32c.js:25. The implementation in crc32c.js:25-38 uses a pre-computed table for performance. Because JavaScript bitwise operators produce signed 32-bit results, the implementation uses >>> 0 to ensure the final result is an unsigned 32-bit integer, preventing mismatches with the broker crc32c.js:57.

More: CRC-32C Checksum Verification

Connection Framing and Pipelining

The Connection class manages the lifecycle of a socket and supports pipelining by using a FIFO queue of pending requests connection.js:50. Each request is assigned a correlationId to ensure that responses are matched to the correct caller, even if they arrive out of order relative to the request stream connection.js:141-153. The framing logic uses a 4-byte length prefix to delineate messages connection.js:97-105.

Topic and Partition Management

The client provides high-level abstractions for interacting with topics. The topic method in connection.js:280-284 retrieves metadata from the broker to determine available partitions. Once a Topic is obtained, the partitionFor method handles routing: it uses the FNV-1a hash for keyed records or a round-robin approach for null keys connection.js:311-316.

Consumer Polling and Async Iteration

The Consumer class manages the reading of a partition, tracking the position (the offset of the next record to be read) consumer.js:111. It provides an async iterator via the records() method, which uses poll() to fetch batches of data consumer.js:189-192. If a fetch returns a record that is larger than the maxBytes limit, a RecordExceedsMaxBytesError is thrown to prevent infinite retry loops consumer.js:156-161.

FakeBroker for Conformance Testing

The FakeBroker class in clients/node/test/fake-broker.js is used to verify that the client correctly encodes requests and decodes responses. Unlike a real broker, it is designed to decode the client's payload to ensure the client's encoding is valid, rather than just pattern-matching bytes fake-broker.js:5-7. It can also simulate data corruption to test the client's checksum verification fake-broker.js:178.

Key files

Behaviour that surprise

• The partitionFor function in Topic uses a round-robin counter that advances on every call when the key is null, meaning that calling partitionFor multiple times without sending a record will cause the client to skip partitions connection.js:305-307.
• The Consumer.poll method advances the position by the number of records returned, but if a fetch is truncated, the partial record is dropped and the next poll starts from the beginning of that same record consumer.js:139-142.
• In Connection.#onData, if a response's correlationId does not match the correlationId of the oldest pending request, the client rejects the request with a ProtocolError rather than attempting to find the correct match connection.js:120-122.