FNV-1a Partitioning

Documentation for the FNV-1a partitioning mechanism used by clients to select partitions.

Diagram

What this module is responsible for

The partitioning mechanism is responsible for mapping a record's key to a specific partition index. Because the broker's record format has no room for a key, the client must perform this calculation locally and send only the resulting partition number partition.js:4-5.

fnv1a32

The fnv1a32 function implements the 32-bit FNV-1a hash algorithm partition.js:28. Because JavaScript numbers are doubles, a standard multiplication would lose precision and fail to wrap around correctly after three bytes; therefore, Math.imul is used to ensure 32-bit integer multiplication with the required wraparound partition.js:16-20. The final result is converted to an unsigned 32-bit integer using the >>> 0 bitwise operator partition.js:34.

partitionFor

The partitionFor function takes a Uint8Array key and the number of available partitions to determine the target partition partition.js:48. It uses an unsigned remainder (modulo) to fold the 32-bit hash into the range [0, partitions) partition.js:52. The function throws a RangeError if the partition count is not a positive integer, as there must be at least one valid partition to select partition.js:49-50.

The signed byte trap

A critical edge case in cross-language implementation is the interpretation of bytes. While the Node.js client iterates over a Uint8Array which yields unsigned bytes, languages like Java or Kotlin treat bytes as signed partition.js:31. Specifically, the byte 0x80 must be treated as 128 (unsigned) rather than -128 (signed) to ensure the hash remains consistent across different client implementations README.md:74-75. Failure to mask these values results in a client that passes ASCII tests but fails on non-ASCII keys README.md:75-76.

The partitioner conformance vectors

To ensure that any client implementation (regardless of language) produces the same partition for a given key, the project uses "golden vectors" generate.py:101-105. These vectors are generated by a second, independent implementation in Python to ensure that the Kotlin client is not simply being tested against itself generate.py:5-6. The harness checks both the 32-bit hash and the resulting partition index across various partition counts generate.py:30.

Key files

Behaviour that surprise

• partitionFor will throw a RangeError if the partitions argument is not a positive integer, preventing the client from sending a record to a non-existent partition partition.js:49-50.
• The fnv1a32 function relies on Math.imul not just for speed, but because it is the only way to achieve the specific 32-bit wraparound required by the FNV-1a specification in JavaScript partition.js:16-19.
• The broker never actually sees the key used for partitioning; it only receives the calculated partition number from the client partition.js:4-5.