Quarkus Insights # 263: Everything You Didn't Want to Know About Quarkus Build Reproducibility
This summary was generated using AI, reviewed by humans - watch the video for the full story.
Quarkus Insights # 263: Everything You Didn’t Want to Know About Quarkus Build Reproducibility
Achieving reproducible builds in modern software is often described as straightforward: pin your dependencies, use a deterministic compiler, and let your build tool do the rest. In a framework like Quarkus, however, where the build process performs heavy ahead-of-time bytecode generation and executes build steps across multiple parallel threads, determinism is anything but simple. In episode 263, community contributor Teymur (Tim) Babayev and Quarkus core maintainers Guillaume Smet and Ladislav Thon join host Holly Cummins to unpack the technical challenges of making Quarkus builds bit-for-bit reproducible, the insidious sources of non-determinism they uncovered, and the infrastructure now in place to ensure our binaries continue to stay reproducible.
Quarkus News
Holly opened the episode with an unusual announcement: for the first time in months, there was no new minor or micro release during the week. With the release cadence briefly paused, Guillaume took a well-earned break from preparing backports. The core team is currently focused on iterating on the Quarkus 4 betas leading up to the final release, with micro releases on earlier streams reserved strictly for critical CVEs. Community members are encouraged to try out the Quarkus 4 beta, test their applications, and report any issues or feedback.
What Is Build Reproducibility and Why Does It Matter?
Guillaume kicked off the technical discussion by defining build reproducibility according to the principles of Reproducible Builds: given the exact same source code, same Java Development Kit (JDK), same operating system, and identical build inputs, the build must produce the exact same binary output bit-for-bit.
Two major requirements drive the need for reproducible builds:
-
Supply chain security and forensics: High-profile security breaches across ecosystems like npm have highlighted the risk of compromised Continuous Integration (CI) and build infrastructure. If a build is fully reproducible, anyone can independently rebuild an artifact from source using the matching build environment and verify that the output matches what was published to Maven Central. If the hashes match, developers have cryptographic assurance that the binary contains no backdoors injected during the build.
-
Build caching: In modern build tools like Gradle and Apache Maven (with build caching enabled), non-deterministic task outputs invalidate downstream caches. If a build step generates non-identical bytecode across runs, the build tool assumes the output has changed and is forced to re-execute downstream steps, eliminating the performance benefits of caching.
Why Reproducibility Is Uniquely Challenging in Quarkus
For traditional Java frameworks, reproducible builds are relatively simple because they rely on standard sequential compilation where the JDK and build plugins handle determinism. Quarkus is fundamentally different:
-
Ahead-of-time augmentation: Quarkus analyzes application classes and generates extensive bytecode at build time rather than at runtime.
-
Massive concurrency: To keep build times fast, the Quarkus build engine executes build steps concurrently across a thread pool. Concurrency inherently introduces non-deterministic execution timing.
-
Bytecode recorders: Quarkus extensions execute configuration and initialization logic at build time and record invocations into generated bytecode classes.
-
Parallel packaging: Packaging steps also push class files into JAR archives in parallel.
Because reproducibility requires 100% bit-for-bit equality, a single non-deterministic timestamp, random collection iteration, or static counter anywhere in any extension invalidates the entire resulting artifact.
The Roadblocks and Architectural Fixes
The effort to make Quarkus reproducible began in 2020 when Peter Palaga, working on Apache Camel and Apache CXF, submitted early pull requests to eliminate non-determinism in some outputs. At the time, the fixes were considered too invasive. The effort was revived over the past year when Tim Babayev started investigating Gradle build caching and teamed up with Guillaume and Ladislav to tackle the core root causes.
1. Non-Deterministic Multi-Build Item Ordering
In Quarkus, multiple parallel build steps can produce build items of the same type. Downstream consumers receive these as a list. Because the producer steps run concurrently, the order of items in that list was originally timing-dependent.
The team resolved this by resurrecting Peter Palaga’s original approach: enforcing a deterministic sorting order on multi-build items by having them implement Comparable with clear tie-breakers.
However, fixing item ordering introduced an unexpected side effect: recorded bytecode execution order shifted, delaying logging initialization. Because the application had to buffer early log messages in memory until the logging layer was initialized, application memory footprint grew. Guillaume resolved this by making logging initialization an explicit upstream dependency for all related steps, ensuring early startup without sacrificing reproducibility.
2. Static Counters in Bytecode Recorders
When generating proxy classes, bytecode recorder implementations previously appended an incrementing static counter (proxyKey + counter++) shared across parallel worker threads. Because thread scheduling varied across builds, proxy class names differed between runs.
The fix eliminated the static shared counter, scoping unique identifiers strictly per class generation instance rather than globally across threads.
3. Non-Deterministic Collections
A major source of unstable bytecode came from standard Java collections used in build steps and recorders. Methods such as Set.copyOf() and Map.copyOf(), as well as iterating over standard HashSet and HashMap instances containing objects with identity-based hash codes (such as java.lang.Class), produced different iteration orders across builds.
Whenever recorder methods were invoked in loops over unstable collections, or unstable collections were passed as parameters, the generated bytecode diverged. The solution involved switching to deterministic collections, sorting inputs, and defining comprehensive tie-breaker comparisons.
4. Jandex Composite Index Ordering and Arc
Quarkus relies extensively on Jandex for class indexing and offline annotation scanning. While individual Jandex indices are deterministic, Quarkus builds a composite index across multiple JARs.
Ladislav explained that the composite index previously returned classes in arbitrary order because Quarkus passed indexed JARs to the composite index in non-deterministic order. Fixing the composite index construction in Quarkus and refining Jandex and Arc internals eliminated dozens of individual extension-level workarounds in one stroke.
Diagnosis, Tooling, and the Role of AI
Answering an audience question from Douglas Allen about whether AI assisted in tracking down issues, Tim and Guillaume highlighted the complementary roles of tooling and LLMs:
-
diffoscope: The team used diffoscope paired with the Procyon Java decompiler to inspect differing JARs, archive entries, and class bytecode.
-
AI for triage and categorization: Running the initial reproducibility test harness against Quarkus extensions produced roughly 2,000 test failures. The team fed large batches of diffoscope outputs to LLMs to categorize failure modes (e.g., timestamps, collection iteration, counters) and pinpoint likely culprits.
-
Human architectural decisions: While AI accelerated mechanical diagnosis, human insight was essential for avoiding simplistic fixes (such as blindly replacing maps with
TreeSeteverywhere, which would degrade build performance) in favor of upstream framework improvements.
Testing Infrastructure: Three Tiers of Verification
To prevent regressions, Tim developed a multi-tiered test harness:
-
In-JVM augmentation check: Integrated into
QuarkusExtensionTestand activated via-Dquarkus.internal.test.reproducibility-check=true. It runs build augmentation three times in memory within the same JVM, capturing generated bytecode and asserting bit-for-bit equality. If a difference is detected, diffs and decompiled.javafiles are written totarget/debug/reproducibility-mismatch/. -
Cross-JVM check: Re-runs augmentation across separate JVM processes.
-
Integration test file-system check: A shell script (
check-reproducibility.sh) that executes complete Maven builds from scratch and compares final output JARs on disk with diffoscope reports.
The team also introduced ReproducibilityCheckBuildItem to allow test-only constructs (such as temporary directories used by WebJars in Dev Services) to be isolated from production artifact comparisons, and fixed timestamp stability in CycloneDX Software Bill of Materials (SBOM) generation using the PackageConfig build item.
Can Reproducibility Ever Be Guaranteed?
Addressing a question from attendee Douglas Allen on whether these changes represent a patchwork or a deliberate deterministic framework, Ladislav and Guillaume noted that concurrency and underlying system interactions (filesystem system calls, OS metadata, thread scheduling) mean absolute theoretical guarantees are virtually impossible.
Instead, confidence is built through continuous verification and falsification:
-
Nightly CI workflows now run reproducibility test suites across core extension deployment tests, the Jakarta REST TCK, integration tests, and quickstarts.
-
Current reproducibility status can be monitored live on status.quarkus.io, where checks are consistently green.
-
A comprehensive guide documenting reproducibility rules and common pitfalls has been merged into the root of the Quarkus repository for extension authors.
What’s Next?
Looking ahead, the team outlined several next steps:
-
New service architecture: Ongoing work by David Lloyd on a new service start/stop model will gradually replace legacy bytecode recorders, further simplifying the framework and reinforcing determinism at its core.
-
Faster CI feedback: Optimizing reproducibility test execution so checks can eventually run on pull requests rather than solely during nightly builds.
-
Ecosystem outreach: Collaborating with Quarkiverse extension maintainers (including Camel Quarkus) to expand reproducibility verification across third-party extensions.
Key Takeaways
-
Bit-for-bit reproducibility matters: Identical builds provide cryptographic assurance for supply chain security and prevent cache invalidation in Gradle and Maven.
-
Concurrency makes determinism hard: Running build steps and packaging in parallel produces non-deterministic ordering unless inputs and collections are strictly controlled.
-
Multi-build item ordering: Sorting build items deterministically resolved list ordering issues, coupled with explicit logging dependencies for fast startup.
-
Scoping recorder identifiers: Replacing shared static counters with instance-scoped identifiers eliminated race conditions in generated proxy class names.
-
Collection stability: Avoiding iteration over unstable hash codes and using deterministic tie-breakers prevents subtle bytecode diffs.
-
Jandex and Arc framework fixes: Upstream improvements to Jandex composite indexing fixed dozens of extension-level inconsistencies simultaneously.
-
diffoscope+ AI triage: Combining diffoscope bytecode decompilation with LLM analysis enabled rapid classification of thousands of initial test failures. -
Multi-tiered test harness: Developers can test extensions in memory using
-Dquarkus.internal.test.reproducibility-check=trueor across full builds withcheck-reproducibility.sh. -
Status is green: Nightly CI runs for core deployment tests, integration tests, and quickstarts are tracked on status.quarkus.io.
-
A collaborative milestone: The effort united core maintainers and community contributors.
Conclusion
Achieving reproducible builds in a high-performance, ahead-of-time framework like Quarkus is a remarkable engineering accomplishment. By tackling non-determinism directly at the framework level (spanning Jandex, Arc, build items, and recorder infrastructure) rather than applying brittle workarounds, the Quarkus team and community have established a robust foundation for supply chain security and build caching. Next week, Quarkus Insights returns with a deep dive into JobRunr.
