How Project Leyden brought a new perspective

The Netty case is particularly interesting because it reads annotations to determine the capabilities of interface implementations.

For this kind of use case, we recommend using marker interfaces or methods instead of annotations.

Georgios once again resorted to bytecode rewriting to eliminate the annotation lookup. Again, this isn’t something we want to maintain in the long term.

Hibernate ORM is also extremely interesting in this context.

The first important point is that, with Hibernate ORM, in Quarkus, we still build the metadata at runtime. That means we end up reading a lot of annotations at runtime. A long-term effort has already started to improve this situation, but it’s a significant undertaking and will take time before we can move metadata building to build time instead.

Let’s set that aside for now.

What’s also interesting is that Hibernate ORM collects metadata about its own annotations at runtime, both Hibernate-specific annotations and JPA annotations. And that’s a lot of annotations, and a lot of metadata to process.

For example, for each JPA or Hibernate annotation, it determines the annotation target (class, method, field), or whether it is inherited.

In the end, this results in a substantial amount of annotation processing for something that changes very rarely.

I identified this issue some time ago, and our colleague Luca Molteni from the Hibernate team will be looking into it soon. We’re not yet sure how easy it will be to fix, but you get the idea. Whenever possible, this kind of metadata should be resolved once and for all. And you should be able to enforce its correctness with tests to ensure it remains accurate and up to date.

Hopefully, we’ll be able to improve this soon. And the nice part is that any improvement here will benefit all applications using Hibernate ORM, not just Quarkus applications.

Why would you try to load a non-existing class? That’s a good question.

Remember package-info.java files? Hibernate ORM, for instance, tries to load them to inspect package-level annotations. In a lot of cases, these files don’t exist, and that’s perfectly normal.

Caching class loading is within the scope of Leyden, but Leyden does not cache negative lookups. Why? Because Leyden, while AOT, is still true Java. Java is a dynamic language: even if a class wasn’t present when you recorded the cache, it might be added later. Now, in practice, that’s often not the case, especially in Quarkus, where we assume a closed world, but Leyden cannot rely on that assumption.

In Quarkus, when using Leyden, we decided to generate empty package-info classes for all packages containing entities that don’t already have one. This way, Hibernate ORM doesn’t have to attempt to load non-existing classes.

Another example is JBoss Logging internationalization: it attempts to load a class for the current locale, and if that class doesn’t exist, it falls back to the default class.

You see similar patterns with resources. An application might try to load a configuration file or a service descriptor that doesn’t exist. That’s perfectly normal, the only way to know it’s missing is to try.

In Quarkus, we have a few class loader tricks to mitigate this. But in the general case, you have to deal with it.

And here’s the catch: for all these cases, the runtime will walk the entire classpath (remember, it won’t find anything), trying to locate the class or resource. In doing so, it may open a large number of JAR files and read from them, just to conclude that the class or resource doesn’t exist.

Granted, it doesn’t read the entire JAR, each archive has an index, but still.

Let’s look at the ServiceLoader case in more detail, it’s particularly interesting.

Once class loading is out of the picture, it becomes clear that a significant portion of startup time is spent loading service descriptors from JAR files. We managed to improve this for some services using a class loader trick, but that only works for services loaded through the thread context class loader. We don’t yet have a good solution for services loaded by the JDK class loaders. At least not for now.

We’ll go into more detail in the next blog post, where we’ll talk more about Project Leyden itself and how we integrated it into Quarkus. Stay tuned.

This one is easy: whenever you generate a UUID, the JDK will initialize a SecureRandom instance. And initializing a SecureRandom instance doesn’t come for free, oh no.

Sure, if your application ends up needing a SecureRandom anyway, you don’t care. But if it doesn’t, having your favorite framework generate UUIDs for internal use is not ideal.

In the same vein, the BigDecimal class has a static initializer that actually performs a fair amount of work. Initializing BigDecimal can take a noticeable amount of time.

We stumbled upon this because we were using BigDecimal to perform some calculations when printing the Quarkus startup time.

D’oh.

We replaced that code, only to discover another issue: BigDecimal was also being eagerly initialized in Hibernate ORM for a very narrow use case in the DurationJavaType class. That has since been fixed as well.

We all know time zones are hard. But now we also know that loading them is slow.

The time zone database is quite large, and loading it, for example when calling TimeZone.getDefault(), can take a noticeable amount of time.

Who would need a time zone in a server application, right? For instance, your logging layer, which wants to print timestamps in the local time zone.

And what’s also interesting is that there’s an additional cost for transforming the time zone to a ZoneId, as you also have to load the zone rules.

These two issues still exist and I have no idea if we can even solve them, but we were able to mitigate their cost somewhat in Quarkus when using the specific packaging we developed for AOT.