CRC-32C Checksum Implementation

Diagram

The CRC-32C Polynomial and Reflected Tables

The implementation uses the Castagnoli polynomial 0x1EDC6F41. Because many implementations, such as the one in crc32c.js:32, utilize right-shifting, the table must be built using the reflected form of the polynomial, which is 0x82F63B78 (crc32c.js:25). Using the standard polynomial with a right-shift results in a "stable, plausible, everywhere-wrong sum" (crc32c.js:16).

The Zero-Copy Read Path and Client-Side Verification

A critical design decision in the booblik protocol is that the broker does not touch the payload bytes on the read path to enable zero-copy performance; consequently, the responsibility for verifying the checksum shifts entirely to the client (SegmentWriter.kt:78-80). While the broker computes the checksum on write, the client is the only party on the read path capable of performing the verification (SegmentWriter.kt:79).

The Recovery Process and Torn Records

The checksum serves as a vital sentinel during the broker's startup recovery process. Because a length prefix alone cannot distinguish between a valid record and a partially written (torn) record after a crash, the checksum allows recovery to stop at the first record whose bytes do not match their own header (SegmentWriter.kt:73-74).

Language-Specific Arithmetic Traps

Different programming languages require specific handling to manage 32-bit integer arithmetic and sign bits:

• JavaScript: Requires the >>> 0 operator to ensure bitwise operations produce unsigned 32-bit results rather than signed ones (crc32c.js:57).
• Java: Requires a cast from the long returned by CRC32C.getValue() to an int (README.md:52).
• Python: Must handle the fact that its integers never overflow, requiring specific logic to match the expected signed/unsigned behavior (README.md:38).
• C#: Does not require special handling because a cast between int and uint preserves the bits (README.md:55).

Conformance and Golden Vectors

To ensure correctness across all implementations, the module is validated against "golden vectors" (README.md:37). These vectors, stored in conformance/vectors/crc32c.tsv, are used to catch errors where an implementation might be self-consistent but mathematically incorrect, such as using a mis-derived reflected polynomial (README.md:83-84).

Key files

Behaviour that surprises

• The crc32c function in JavaScript requires a >>> 0 at the end because bitwise operators produce signed 32-bit results, which would otherwise cause the sum to be negative half the time (crc32c.js:17-19).
• In the .NET client, the Crc32C implementation is provided via forty lines of code in Crc32C.cs rather than a NuGet package to avoid making verification an optional dependency (README.md:94-96).
• A Producer in .NET returns null for ProduceAsync when using AckPolicy.None, as no offset exists until the batch reaches the broker (README.md:51-52).