Improving a Competitor's Numbers (On Purpose)

If you’re claiming to compare peak throughput, every runtime needs to have finished compiling before the stopwatch starts. Otherwise you’re comparing one runtime’s peak against another’s still-warming-up numbers, and calling it a day.

Our old warmup was a single 2-minute continuous blast followed by a 30-second cooldown and a 30-second measurement. Quarkus was done compiling in 60 seconds. Spring wasn’t done when measurement began.

That’s a framework problem: Quarkus doesn’t have it despite higher throughput, because its leaner request processing leaves CPU gaps for the compiler. Or as Mytkowicz et al. put it: we were "producing wrong data without doing anything obviously wrong". In production, nobody pauses traffic so the JIT can catch up — continuous load is the realistic scenario. But it’s still our call to make sure the benchmark gives every runtime the best chance to show its peak.

We can’t just make the warmup longer. The whole point of the previous analysis was that C2 never gets CPU during continuous load. More time under load just means more time starved.

The trick: break the warmup into short bursts with cooldown gaps. During each 15-second cooldown, the 2 C2 compiler threads get full access to the 4 available cores and can drain the compilation queue. 10 cycles of load+cooldown, 5 minutes total, then measure.

Quarkus peaks by cycle 2. Spring ramps through 5 cycles and stabilizes. By the time measurement starts, both are fully compiled. In cloud deployments with fewer cores, the starvation would be worse — reaching peak quickly vs. slowly translates directly to provisioning cost.

The C2 compiler thread activity confirms it. Compare this to the previous article’s charts where C2 was starved for the entire warmup:

Both frameworks show near-zero C2 activity during the load test (rightmost region). The compiler is done.

Spring went from ~7,850 to ~8,500 TPS — an ~8% improvement just from letting C2 finish its job. Quarkus held steady at ~14,800. The ratio narrowed from ~1.9x to ~1.7x.

The performance gap is real, and Quarkus’s lead remains definitive. What we removed is the artificial penalty from measuring before Spring finished compiling.

The multi-cycle warmup fixed the peak measurement. But here’s something the average still can’t tell you. These are the per-second TPS traces during the load test with the new warmup, across multiple iterations:

Quarkus is stable within the measurement window. spring4-virtual is stable across all three iterations, with occasional single-second dips. spring4-jvm wobbles continuously across all iterations, despite no C2 compiler activity during the load test (as we saw in the previous section). The instability is not a compilation problem.

A violin plot makes the distributions visible at a glance:

In a Quarkus Insights session with Holly Cummins, we discussed the value of having a noiseless benchmark: if the measurement is stable enough, you don’t need statistics at all. A single number tells the story. When it doesn’t, violin plots show what’s really happening, but they add cognitive load. The goal is to make statistics unnecessary, not to add more of them.

This isn’t the first time we’ve faced this dilemma. In issue #521, we found that both frameworks were given 512MB heap even though Quarkus could run with far less. Rather than give Quarkus a memory advantage, we kept the same heap for both: a common baseline, even if it doesn’t showcase Quarkus’s density. Same principle: measure what you claim to measure.