Benchmark execution and reporting

Diagram

The mainCiBenchmark mode

The CI environment uses a specific configuration designed to filter out noise inherent to shared, hosted runners. Instead of running the full suite, it executes mainCiBenchmark as defined in build.gradle.kts:97-110, which focuses on tasks that are not heavily influenced by the non-deterministic performance of a hosted runner's disk.

The benchmark-floor.sh mechanism

To avoid the pitfalls of comparing runs from different machines—where a 20-40% swing is common—the system does not look for percentage-based regressions. Instead, benchmark-floor.sh:24-26 implements a "floor" mechanism that only triggers if performance collapses by an order of magnitude. It compares the current throughput against a hardcoded threshold (e.g., 20,000 for FILE_CHANNEL and 1,000,000 for MAPPED) to ensure that only catastrophic failures, rather than minor runner variance, cause a build failure.

The PartitionWriterBenchmark lifecycle

The benchmark lifecycle is designed to measure the real-world cost of a broker's operations. In PartitionWriterBenchmark.kt:98-102, the log.retainAtMost call is explicitly included within the measured @Benchmark loop. This ensures that the overhead of log retention (cleaning up old segments) is accounted for in the throughput numbers, reflecting the actual performance a user would experience in a production environment.

The StartupProbe and recovery scan

The StartupProbe measures the efficiency of the system's recovery mechanism. As detailed in StartupProbe.kt:41-49, the probe measures the time taken to scan segments and rebuild the index. It performs three separate attempts to capture the difference between a "cold" restart (paying for the initial page cache miss) and "warm" restarts, providing a realistic range for how long a broker takes to recover after a crash.

The mainCiBenchmark exclusion rules

To maintain a stable CI signal, certain benchmarks are explicitly excluded from the ci configuration in build.gradle.kts:111-117. Specifically, GroupCommitBenchmark is excluded because its performance is heavily dependent on disk barriers and fsync latency, which vary wildly on hosted runners. Additionally, the flushEveryAppend parameter is set to false to avoid measuring paths that are primarily bounded by disk I/O latency rather than code efficiency.

Key files

Behaviour that surprises

• Non-comparable numbers: Because of runner variance, the system explicitly refuses to compare the current CI run to the previous one; it only checks if the current run is "not a total collapse" via benchmark-floor.sh:70-71.
• Intentional failure in build.yml: The build.yml:42-43 step compiles benchmarks but never runs them, ensuring that broken benchmarks are caught during the build phase without producing unreliable data.
• The allOpen requirement: In build.gradle.kts:46-51, the allOpen plugin is used to make @State classes non-final, which is necessary because JMH generates subclasses of these classes during execution.