PartitionLog

What this module is responsible for

PartitionLog is the core storage component of the booblik system, responsible for managing an ordered sequence of LogSegment objects. It provides an append-only log abstraction where data is partitioned into segments to facilitate efficient retention (deletion of old data) and high-performance reads.

Diagram

PartitionLog mechanics

The PartitionLog architecture is built on a single-writer, multi-reader concurrency model. To avoid the overhead of locks on the hot path, the segment list is maintained as an immutable List behind a @Volatile field, which is replaced rather than mutated whenever the log structure changes (e.g., during a roll or retention) PartitionLog.kt:41-42. This ensures that readers always see a consistent snapshot of the log, either before or after a change, but never a partially mutated list PartitionLog.kt:19-22.

LogSegment lifecycle and retention

Segments follow a strict lifecycle to ensure data is not deleted while being actively read. Readers must use acquire() to claim a segment, which prevents the retention process from closing the segment's descriptors until the reader calls release() PartitionLog.kt:111-116.

Retention is performed at the segment level via retainAtMost or retainNewerThan, which unlinks files from the filesystem to reclaim space PartitionLog.kt:185-195. A key safety mechanism is that a segment is removed from the volatile segments list before retire() is called, ensuring new readers cannot find it, while existing readers can continue streaming data from the unlinked file until they release their handle PartitionLog.kt:27-31.

SegmentMode and zero-copy transfer

The log supports different modes for data persistence and access:

For high-performance data movement, transferTo utilizes FileChannel.transferTo to move bytes directly from the file cache to a WritableByteChannel without copying data through the JVM heap PartitionLog.kt:173-175.

Data integrity and recovery properties

The log ensures consistency across restarts by scanning the partition directory and rebuilding the segment list from existing .log files PartitionLog.kt:238-245. The LogPropertiesTest verifies that a reopened log recovers exactly the same records and offsets as the original session LogPropertiesTest.kt:97-103. Additionally, the truncateTo operation on a LogSegment allows for precise data removal, ensuring that the log remains consistent by reusing the freed offset for subsequent appends LogPropertiesTest.kt:147-151.

PartitionLog boundary and error invariants

The implementation enforces several strict invariants to prevent corruption and resource leaks:

• Record Size Limits: A record that is larger than an empty segment is rejected immediately to prevent the creation of "stray" empty files PartitionLog.kt:64-65.
• Segment Rolling: The log automatically rolls to a new segment when the current activeSegment cannot accommodate a new record PartitionLog.kt:75.
• Boundary Integrity: Tests confirm that records are readable even when they span across segment boundaries and that transferTo never attempts to cross a segment boundary in a single call PartitionLogTest.kt:84-96.

Key files

Behaviour that surprising

• PartitionLog.append will throw an IllegalArgumentException if a record is too large for a segment, rather than attempting to roll a new segment first PartitionLog.kt:72-74.
• When a segment is unlinked via retire, a reader holding a FetchSlice can continue to read the data until the slice is closed, thanks to POSIX file descriptor behavior PartitionLog.kt:28-30.
• PartitionLog.open performs a directory scan to recover state, which can be a significant startup cost for logs with many segments StartupProbe.kt:42-48.