Project Leyden

Another name to describe Java’s lazy loading is 'Just in Time' (JIT). JIT is a well known term used to describe the operation of Java’s runtime compilers. What is less well known is that it has a much wider use. JIT is not limited to compilation: many other operations performed by the JVM are done lazily at runtime or 'Just In Time'.

An alternative to doing things 'Just in Time' (JIT) is to do them 'Ahead Of Time' (AOT). For example, GraalVM’s Native Image runtime loads and analyses the bytecode of every single class needed by an application, including JDK runtime classes, 'Ahead Of Time' i.e. at image build time. It uses the type and method information encoded in that bytecode to 'Ahead Of Time' compile a complete program that includes code for every method that might possibly be executed by the application.

The approach of GraalVM’s native images lies at one extreme: everything is done AOT, while the traditional Java runtime model lies at the other extreme, as much as possible is done JIT. However, it is actually possible to mix and match AOT and JIT models of execution in one runtime: re-balancing that AOT vs JIT mix is the goal of the first EA release of project Leyden.

Interestingly, this time-shifting concept is also applied by Quarkus; we called it "augmentation" and essentially consists in booting popular frameworks during the build time of the application, to not incur such performance penalties at runtime.

A native image build might also take advantage of Profile Guided Optimisations (PGO), which allows it to leverage some data about what’s presumably happening at runtime back into the compilation process, guiding its optimisations. It’s essentially peeking into the future - another form of time-shifting. However, it’s only peeking into a simulation of runtime metrics, and ultimately, the compiler still needs to make all optimisation tradeoffs Ahead Of Time; this has pros and cons. The primary disadvantage is that any suboptimal decision is cast in stone; luckily there is a fallback mechanism to recover from outright bad decisions, but this mechanism cannot produce new optimal code. The advantage is more decisive for short-lived applications as the tradeoff of carrying all support for JIT optimisations in the runtime is less justifiable when there is barely an opportunity to take advantage of it.

Shifting work so it is done AOT is not a wholly new idea as far as the OpenJDK runtime is concerned. OpenJDK has supported a hybrid AOT/JIT class loading model for years with CDS. The observation that led to Class Data Sharing (CDS) being proposed was that most applications load the same classes every time they run, both JDK classes during JDK bootstrap and application classes during application startup and warmup.

Loading requires locating a class bytecode file, possibly calling out to a Java ClassLoader, parsing the bytecode then building a JVM-internal model of the class. This internal model unpacks the information packed into the bytecode into a format that enables fast interpreted or compiled execution. If this loading and unpacking work could be done once and the resulting class model efficiently reused on subsequent runs, then that would save time during startup and warm up.

Initially CDS optimized loading for a large set of core JDK classes. It worked by booting the JVM and dumping the class model for all classes loaded during startup into an archive file laid out in memory format. The resulting JDK module, class, field, and method graph can then be quickly remapped into memory next time the JVM runs. Loading a class that is present in the archive involves a simple lookup in the AOT class model. Loading a class not present in the archive requires the normal JIT steps of bytecode lookup, parsing and unpacking i.e. CDS implements a hybrid JIT/AOT execution model.

A default CDS archive for JDK runtime classes has been shipped with every JVM release since JDK17, halving JDK startup time. Improvements were made to CDS to allow application classes to be included in a CDS archive after executing a short application training run. The resulting mixed AOT/JIT operation can provide significant improvements to application startup and warmup times, depending on how well the training run exercises application code. So, selective JIT/AOT operation is not some new thing.

Quarkus makes it really easy to generate CDS archives specific to your application code; this feature has been around since some years already: see the AppCDS guide in Quarkus. As Leyden is coming, we aim to evolve this further and fully automate it for Leyden as well, so to get you even more benefits at no additional hassle.

The goal of Project Leyden is extending the AOT vs JIT trade-off from class loading (as done by CDS) to other JIT operations in the JVM; there’s a number of operations which could be "moved in time" to AOT, such as creation of heap objects to represent constants, gathering execution profile information, and many more. Most importantly, it’s moving AOT the lazy linking that normally happens during interpreted execution and the lazy compilation and recompilation that happens when methods have been executed enough times to justify the cost of compilation.

Linking of classes is another operation that the JVM does lazily. When class bytecode is processed the class is directly linked to its owning module and its owned methods and fields. JIT linkage connects elements of each independent, linked class sub-graph into a fully connected graph where elements from different (class or module) files cross-reference each other.

Loading and linking needs to proceed recursively. As one example, every class (except Object) needs to be linked to its super class. Super linkage cannot complete without ensuring the super class is loaded. Indeed, if the super’s bytecode cannot be found or is not valid (say it identifies an interface not a class) then a linkage error may occur. Likewise, a new operation or a field get/put operation occurring in some method’s bytecode can only be linked after loading the class (and field) named in the new bytecode.

Linking is sometimes, but not always, done lazily. Indeed, it is necessary to do some linkage lazily in order to allow loading also to be lazy, otherwise the whole class graph would end up being linked and loaded as soon as the main routine was entered. Super linkage is always done eagerly at the point where the subclass has just been loaded. That is because it is not possible to use a subclass to create instances or execute methods without knowing how the superclass is defined. By contrast, field and method linkage is done lazily. In these cases linkage happens as a side-effect of execution. When a method executes a field get/put or method invoke bytecode for the first time the target field or method is looked up via its owner class, loading it if necessary. The field type or method signature is checked for consistency and details of where to find the field or how to call the method are cached, allowing the next execution of the bytecode to bypass the linkage step.

As with lazy loading, this lazy approach results in almost the exact same linkage being established on every run. The time spent stopping and restarting execution to lazily connect the class graph comprises a noticeable percentage of JDK startup, application startup and application warm up (time to peak running). We could speed up startup and, more crucially, warm up time if we could pre-compute this linkage and avoid the need to establish it at runtime.

Loading and linking of classes is an important step in the warm up of the application because it involves searching through the whole classpath for all classes and objects referenced by the bytecode the JVM is going to run. By default, this is done as a lazy operation because loading and linking all existing classes in the classpath would not only require a bigger memory footprint, but also a bigger warm up time. This is why the JVM only compiles and links the bytecode that is going to be used.

This is a process that Quarkus already speeds up by, among other strategies, aggressively reducing the set of classes included in the classpath, so the search for matches is faster. The search for classes is also accelerated by indexes which Quarkus can generate when it fully analyzes the application at build time. But it is still a heavy operation that is difficult to execute ahead of time, before we know what is going to be run and how. Quarkus might be able to provide some additional hints to the linker in the future.

The first improvement Leyden is offering to improve startup time is to upgrade the AOT model originally developed as part of the CDS project to encompass not just pre-loading of classes but also pre-linking, as described in JEP Ahead-of-Time Class Linking.

An AOT Cache can be generated during a training run that bootstraps the JVM and, optionally, executes application-specific code. As with a CDS archive, the AOT Cache stores a class graph for all classes loaded during the training run in a format that allows it to be quickly remapped on a subsequent run. The stored graph also includes any linkage information established by code executed during the training run. Pre-cached links avoid the need to stop and start execution to perform linkage on subsequent runs.

Remember that the training run enables some of the loading and linking to be done AOT but that anything not trained for will still be performed via the regular JIT process: the AOT approach is not required to be applied comprehensively, so that the JVM can fallback to the regular loading system for the use cases which can not benefit from AOT processing. This ability to fallback to "regular JIT processing" is a luxury that GraalVM native images can’t use.

Another well-known lazy operation the JVM performs is JIT (runtime) compilation. Method bytecode is normally interpreted, but the JVM will lazily translate bytecode to equivalent machine code. Since generating optimal machine code is an expensive operation, it performs this compilation task selectively, only bothering to compile methods that have been invoked quite a few times.

JIT compilation is also 'adaptive' i.e. the JVM will lazily recompile some methods, using different 'tiers' or levels of compilation.

Tier 1 - 3 compilations omit many possible optimizations in order to deliver compiled code quickly. A tier 4 compilation can take much longer to complete so it is only attempted for a small subset of very frequently executed methods.

Sometimes, the code is compiled with substantial optimisations based on 'speculative' assumptions extrapolated from the profiling data. In such cases, the compiler will make an optimistic assumption about a condition to be consistently true in the future yet include an efficient check to verify the assumption during execution so that the semantics of the program are not affected in case this educated guess eventually turns out to be false; when this is detected, the code is de-optimised, returning at a previous tier of compilation and the profiling data is adjusted, so that it will eventually be recompiled with better information. Essentially, some parts of code might get recompiled multiple times and occasionally revert to a lower tier: it’s an highly dynamic process.

Peak optimization is reached when most of the running code is compiled at the highest tier, and background compilation activities become very rare or, ideally, none at all.

Compiling code for peak performance also requires quite some resources, so performing this work ahead of time can also save precious CPU cycles during the application bootstrap, and can manifest in substantial memory savings as well: Java developers aren’t used to measure the memory costs of the JIT compiler, but the fact that it’s hidden doesn’t imply it’s non-existent; and while this might be a detail for large enterprise servers, it’s quite important to be aware of such resource costs when developing microservices or simply aiming for smaller, more power efficient targets.

But there are some limitations on what we can optimise before runtime just by examining the bytecode. For example, extensive use of reflection prevents the compiler from predicting which symbols will be loaded, linked, and most used at runtime.

The Leyden project has already sucessfully prototyped shifting the work of method compilation from JIT to AOT. Execution and compilation of methods is tracked during the training run. At the end of the run any associated profiling information and compiled code for the method are saved to the AOT Cache, allowing them to be quickly mapped back into memory and reused when the application is next run.

As with AOT loading and linking, the training run enables some of the work of profiling and compiling to be done AOT but allows anything not trained still to be compiled via the regular JIT compilation process. Note that method code does not need to have been compiled at the highest tier in order to be saved. Also, when code compiled at a lower tier is restored it can still be recompiled at a higher level.

Compiled code can also be deoptimized and re-optimized to adapt to different runtime conditions, just as with code compiled in the current runtime. So, the use of AOT compilation is fully integrated into OpenJDK’s adaptive, dynamic compilation and recompilation model: even if some assumptions made during AOT compilation turn out to be suboptimal, the just-in-time compiler can intervene at runtime and improve the code with the new information.

To benefit from the natively compiled code in AOT archives, the garbage collector used at runtime needs to match the same garbage collector used when you recorded the AOT archives.

Remember that the JVM’s default choice of garbage collector is based on ergonomics; normally this is nice but it can cause some confusion in this case; for example if you build on a large server it will pick G1GC by default, but then when you run the application on a server with constrained memory it would, by default, pick SerialGC.

To avoid this mismatch it’s best to pick a garbage collector explicitly; and since several AOT related optimisations today only apply to G1, let’s enforce the use of G1GC.

Force using G1GC:

N.B. you need to use this consistently on both the process generating the AOT archives and the runtime.

As identified and reported by the Quarkus team to our colleagues working on Project Leyden, beyond enforcing a specific garbage collector, one should also ensure that the code stored in AOT archives is being generated with the same G1 region sizes as what’s going to be used at runtime, or one risks segmentation faults caused by it wrongly identifying regions. See https://bugs.openjdk.org/browse/JDK-8335440 for details, or simply set:

Configure G1HeapRegionSize explicitly:

N.B. you need to use this consistently on both the process generating the AOT archives and the runtime.

This issue has already been resolved, but in case you’re using an older version of project Leyden and it fails to exit on regular container termination, you might be affected by JDK-8333794.

Workaround for JDK-8333794:

This is an experimental project being developed by multiple teams having different approaches and focuses. Limitations explained here are being worked on at the time of writing this blog post.

One of the main issues is that functionality is currently only available for x86_64 and AArch64 architectures at the moment.

Also, current developments rely on a flat classpath. If the application is using custom classloaders, then it may not benefit as much as it could as it may miss caching many classes.

The same happens if the application is intensively using reflection. Quarkus avoids reflection whenever possible, preferring to resolve reflective calls at build time as well - so there’s a nice synergy at play.

However Quarkus in “fast-jar” mode, which is the default packaging mode, will use a custom classloader which currently would get in the way of some Leyden optimisations. One could use a different packaging mode in Quarkus to get more prominent benefits from Leyden, but doing so would disable other Quarkus optimisations, so the comparison wouldn’t be entirely fair today. We hope to work on improvements in this area to have all possible benefits, combined.

The focus on these first early releases has been on bootstrap times. There are measurable, significant startup time improvements, due to AOT loading and linking. In some cases, these improvements on startup time have worsened the memory footprint of some applications. That’s an already known issue that is being worked on, and the expected outcome is to improve memory footprint as well, so we would suggest not worrying too much about total memory consumption at this stage.

Since the AOT archives include machine specific optimisations such as the native code generated by the C2 compiler, the training run and the production run must be done on the same type of hardware and JDK versions; it also requires using the same JAR-based classpaths and the same command line options.

Although the training run can use a different Main class to the one used for running the application, for example a test class that simulates usage.

There’s still work to be done regarding classes that can’t be loaded and linked in AOT with the current implementation. For example, classes loaded using a user-defined class loader. There’s also room to improve the way the training runs are made, maybe allowing the user to tweak the results to influence decisions.

Currently, the Z Garbage Collector does not support AOT object archiving. There is an active effort to make sure all Garbage Collectors are compatible with these enhancements.

There are also other things planned in the roadmap for Leyden, like adding condensers. Condensers will be composable transformers of the source code in AOT that modify the source code optimising it. Each developer will be able to define a pipeline of condensers that improves their source code before compiling it into bytecode; this is very interesting to the Quarkus team but condensers aren’t available yet.

The OpenJDK team is actively extending the range of compiled code that can be saved to and restored from the AOT cache. Our colleagues from Red Hat’s OpenJDK team are directly involved in this effort, looking into save and restore of auxiliary code that is normally generated at runtime and used to provide optimized code for 'intrinsic' methods or to link compiled Java method code to the compiled C code that implements the JVM, the interpreter and other compiled C libraries.