Micrometer Metrics

This guide demonstrates how your Quarkus application can utilize the Micrometer metrics library for runtime and application metrics.

Apart from application-specific metrics, which are described in this guide, you may also utilize built-in metrics exposed by various Quarkus extensions. These are described in the guide for each particular extension that supports built-in metrics.

Micrometer is the recommended approach to metrics for Quarkus.

Prerequisites

To complete this guide, you need:

  • less than 15 minutes

  • an IDE

  • JDK 11+ installed with JAVA_HOME configured appropriately

  • Apache Maven 3.8.1+

Architecture

Micrometer defines a core library providing a registration mechanism for Metrics, and core metric types (Counters, Gauges, Timers, Distribution Summaries, etc.). These core types provide an abstraction layer that can be adapted to different backend monitoring systems. In essence, your application (or a library) can register a Counter, Gauge, Timer, or DistributionSummary with a MeterRegistry. Micrometer will then delegate that registration to one or more implementations, where each implementation handles the unique considerations for the associated monitoring stack.

Micrometer uses naming conventions to translate between registered Meters and the conventions used by various backend registries. Meter names, for example, should be created and named using dots to separate segments, a.name.like.this. Micrometer then translates that name into the format that the selected registry prefers. Prometheus uses underscores, which means the previous name will appear as a_name_like_this in Prometheus-formatted metrics output.

Solution

We recommend that you follow the instructions in the next sections and create the application step by step. You can skip right to the solution if you prefer. Either:

The solution is located in the micrometer-quickstart directory.

Creating the Maven Project

Quarkus Micrometer extensions are structured similarly to Micrometer itself: quarkus-micrometer provides core micrometer support and runtime integration and other Quarkus and Quarkiverse extensions bring in additional dependencies and requirements to support specific monitoring systems.

For this example, we’ll use the Prometheus registry.

First, we need a new project. Create a new project with the following command:

mvn io.quarkus.platform:quarkus-maven-plugin:2.3.1.Final:create \
    -DprojectGroupId=org.acme \
    -DprojectArtifactId=micrometer-quickstart \
    -DclassName="org.acme.micrometer.ExampleResource" \
    -Dpath="/" \
    -Dextensions="resteasy,micrometer-registry-prometheus"
cd micrometer-quickstart

This command generates a Maven project, that imports the micrometer-registry-prometheus extension as a dependency. This extension will load the core micrometer extension as well as additional library dependencies required to support prometheus.

If you already have your Quarkus project configured, you can add the micrometer-registry-prometheus extension to your project by running the following command in your project base directory:

./mvnw quarkus:add-extension -Dextensions="micrometer-registry-prometheus"

This will add the following to your pom.xml:

<dependency>
    <groupId>io.quarkus</groupId>
    <artifactId>quarkus-micrometer-registry-prometheus</artifactId>
</dependency>

Writing the application

Micrometer provides an API that allows you to construct your own custom metrics. The most common types of meters supported by monitoring systems are gauges, counters, and summaries. The following sections build an example endpoint, and observes endpoint behavior using these basic meter types.

To register meters, you need a reference to a MeterRegistry, which is configured and maintained by the Micrometer extension. The MeterRegistry can be injected into your application as follows:

package org.acme.micrometer;

import io.micrometer.core.instrument.MeterRegistry;

import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.Produces;

@Path("/example")
@Produces("text/plain")
public class ExampleResource {

    private final MeterRegistry registry;

    ExampleResource(MeterRegistry registry) {
        this.registry = registry;
    }
}

Micrometer maintains an internal mapping between unique metric identifier and tag combinations and specific meter instances. Using register, counter, or other methods to increment counters or record values does not create a new instance of a meter unless that combination of identifier and tag/label values hasn’t been seen before.

Gauges

Gauges measure a value that can increase or decrease over time, like the speedometer on a car. Gauges can be useful when monitoring the statistics for a cache or collection. Consider the following simple example that observes the size of a list:

    LinkedList<Long> list = new LinkedList<>();

    // Update the constructor to create the gauge
    ExampleResource(MeterRegistry registry) {
        this.registry = registry;
        registry.gaugeCollectionSize("example.list.size", Tags.empty(), list);
    }

    @GET
    @Path("gauge/{number}")
    public Long checkListSize(@PathParam("number") long number) {
        if (number == 2 || number % 2 == 0) {
            // add even numbers to the list
            list.add(number);
        } else {
            // remove items from the list for odd numbers
            try {
                number = list.removeFirst();
            } catch (NoSuchElementException nse) {
                number = 0;
            }
        }
        return number;
    }

Note that even numbers are added to the list, and odd numbers remove an element from the list. Try the following sequence and look for example_list_size in the plain text output:

# Compile and run the app in dev mode:
./mvnw compile quarkus:dev

curl http://localhost:8080/example/gauge/1
curl http://localhost:8080/example/gauge/2
curl http://localhost:8080/example/gauge/4
curl http://localhost:8080/q/metrics
curl http://localhost:8080/example/gauge/6
curl http://localhost:8080/example/gauge/5
curl http://localhost:8080/example/gauge/7
curl http://localhost:8080/q/metrics

It is important to note that gauges are sampled rather than set; there is no record of how the value associated with a gauge might have changed between measurements. In this example, the size of the list is observed when the Prometheus endpoint is visited.

Micrometer provides a few additional mechanisms for creating gauges. Note that Micrometer does not create strong references to the objects it observes by default. Depending on the registry, Micrometer either omits gauges that observe objects that have been garbage-collected entirely or uses NaN (not a number) as the observed value.

When should you use a Gauge? Only if you can’t use something else. Never gauge something you can count. Gauges can be less straight-forward to use than counters. If what you are measuring can be counted (because the value always increments), use a counter instead.

Counters

Counters are used to measure values that only increase. In the example below, you will count the number of times you test a number to see if it is prime:

    @GET
    @Path("prime/{number}")
    public String checkIfPrime(@PathParam("number") long number) {
        if (number < 1) {
            return "Only natural numbers can be prime numbers.";
        }
        if (number == 1 || number == 2 || number % 2 == 0) {
            return number + " is not prime.";
        }

        if ( testPrimeNumber(number) ) {
            return number + " is prime.";
        } else {
            return number + " is not prime.";
        }
    }

    protected boolean testPrimeNumber(long number) {
        // Count the number of times we test for a prime number
        registry.counter("example.prime.number").increment();
        for (int i = 3; i < Math.floor(Math.sqrt(number)) + 1; i = i + 2) {
            if (number % i == 0) {
                return false;
            }
        }
        return true;
    }

It might be tempting to add a label or tag to the counter indicating what value was checked, but remember that each unique combination of metric name (example.prime.number) and label value produces a unique time series. Using an unbounded set of data as label values can lead to a "cardinality explosion", an exponential increase in the creation of new time series.

Label and tag can be used interchangably. You may also see "attribute" used in this context in some documentation. The gist is each that each label or tag or attribute defines an additional bit of information associated with the single numerical measurement that helps you classify, group, or aggregate the measured value later. The Micrometer API uses Tag as the mechanism for specifying this additional data.

It is possible to add a tag that would convey a little more information, however. Let’s adjust our code, and move the counter to add some tags to convey additional information.

    @GET
    @Path("prime/{number}")
    public String checkIfPrime(@PathParam("number") long number) {
        if (number < 1) {
            registry.counter("example.prime.number", "type", "not-natural").increment();
            return "Only natural numbers can be prime numbers.";
        }
        if (number == 1 ) {
            registry.counter("example.prime.number", "type", "one").increment();
            return number + " is not prime.";
        }
        if (number == 2 || number % 2 == 0) {
            registry.counter("example.prime.number", "type", "even").increment();
            return number + " is not prime.";
        }

        if ( testPrimeNumber(number) ) {
            registry.counter("example.prime.number", "type", "prime").increment();
            return number + " is prime.";
        } else {
            registry.counter("example.prime.number", "type", "not-prime").increment();
            return number + " is not prime.";
        }
    }

    protected boolean testPrimeNumber(long number) {
        for (int i = 3; i < Math.floor(Math.sqrt(number)) + 1; i = i + 2) {
            if (number % i == 0) {
                return false;
            }
        }
        return true;
    }

Looking at the data produced by this counter, you can tell how often a negative number was checked, or the number one, or an even number, and so on. Try the following sequence and look for example_prime_number_total in the plain text output. Note that the _total suffix is added when Micrometer applies Prometheus naming conventions to example.prime.number, the originally specified counter name.

# If you did not leave quarkus running in dev mode, start it again:
./mvnw compile quarkus:dev

curl http://localhost:8080/example/prime/-1
curl http://localhost:8080/example/prime/0
curl http://localhost:8080/example/prime/1
curl http://localhost:8080/example/prime/2
curl http://localhost:8080/example/prime/3
curl http://localhost:8080/example/prime/15
curl http://localhost:8080/q/metrics

When should you use a counter? Only if you are doing something that can not be either timed (or summarized). Counters only record a count, which may be all that is needed. However, if you want to understand more about how a value is changing, a timer (when the base unit of measurement is time) or a distribution summary might be more appropriate.

Summaries and Timers

Timers and distribution summaries in Micrometer are very similar. Both allow you to record an observed value, which will be aggregated with other recorded values and stored as a sum. Micrometer also increments a counter to indicate the number of measurements that have been recorded and tracks the maximum observed value (within a decaying interval).

Distribution summaries are populated by calling the record method to record observed values, while timers provide additional capabilities specific to working with time and measuring durations. For example, we can use a timer to measure how long it takes to calculate prime numbers using one of the record methods that wraps the invocation of a Supplier function:

    protected boolean testPrimeNumber(long number) {
        Timer timer = registry.timer("example.prime.number.test");
        return timer.record(() -> {
            for (int i = 3; i < Math.floor(Math.sqrt(number)) + 1; i = i + 2) {
                if (number % i == 0) {
                    return false;
                }
            }
            return true;
        });
    }

Micrometer will apply Prometheus conventions when emitting metrics for this timer. Prometheus measures time in seconds. Micrometer converts measured durations into seconds and includes the unit in the metric name, per convention. After visiting the prime endpoint a few more times, look in the plain text output for the following three entries: example_prime_number_test_seconds_count, example_prime_number_test_seconds_sum, and example_prime_number_test_seconds_max.

# If you did not leave quarkus running in dev mode, start it again:
./mvnw compile quarkus:dev

curl http://localhost:8080/example/prime/256
curl http://localhost:8080/q/metrics
curl http://localhost:8080/example/prime/7919
curl http://localhost:8080/q/metrics

Both timers and distribution summaries can be configured to emit additional statistics, like histogram data, precomputed percentiles, or service level objective (SLO) boundaries. Note that the count, sum, and histogram data can be re-aggregated across dimensions (or across a series of instances), while precomputed percentile values cannot.

Review automatically generated metrics

To view metrics, execute curl localhost:8080/q/metrics/

The Micrometer extension automatically times HTTP server requests. Following Prometheus naming conventions for timers, look for http_server_requests_seconds_count, http_server_requests_seconds_sum, and http_server_requests_seconds_max. Dimensional labels have been added for the requested uri, the HTTP method (GET, POST, etc.), the status code (200, 302, 404, etc.), and a more general outcome field.

# HELP http_server_requests_seconds
# TYPE http_server_requests_seconds summary
http_server_requests_seconds_count{method="GET",outcome="SUCCESS",status="200",uri="/example/prime/{number}",} 1.0
http_server_requests_seconds_sum{method="GET",outcome="SUCCESS",status="200",uri="/example/prime/{number}",} 0.017385896
# HELP http_server_requests_seconds_max
# TYPE http_server_requests_seconds_max gauge
http_server_requests_seconds_max{method="GET",outcome="SUCCESS",status="200",uri="/example/prime/{number}",} 0.017385896
#

Note that metrics appear lazily, you often won’t see any data for your endpoint until something tries to access it, etc.

Ignoring endpoints

You can disable measurement of HTTP endpoints using the quarkus.micrometer.binder.http-server.ignore-patterns property. This property accepts a comma-separated list of simple regex match patterns identifying URI paths that should be ignored. For example, setting quarkus.micrometer.binder.http-server.ignore-patterns=/example/prime/[0-9]+ will ignore a request to http://localhost:8080/example/prime/7919. A request to http://localhost:8080/example/gauge/7919 would still be measured.

URI templates

The micrometer extension will make a best effort at representing URIs containing path parameters in templated form. Using examples from above, a request to http://localhost:8080/example/prime/7919 should appear as an attribute of http_server_requests_seconds_* metrics with a value of uri=/example/prime/{number}.

Use the quarkus.micrometer.binder.http-server.match-patterns property if the correct URL can not be determined. This property accepts a comma-separated list defining an association between a simple regex match pattern and a replacement string. For example, setting quarkus.micrometer.binder.http-server.match-patterns=/example/prime/[0-9]+=/example/{jellybeans} would use the value /example/{jellybeans} for the uri attribute any time the requested uri matches /example/prime/[0-9]+.

Using MeterFilter to configure metrics

Micrometer uses MeterFilter instances to customize the metrics emitted by MeterRegistry instances. The Micrometer extension will detect MeterFilter CDI beans and use them when initializing MeterRegistry instances.

@Singleton
public class CustomConfiguration {

    @ConfigProperty(name = "deployment.env")
    String deploymentEnv;

    /** Define common tags that apply only to a Prometheus Registry */
    @Produces
    @Singleton
    @MeterFilterConstraint(applyTo = PrometheusMeterRegistry.class)
    public MeterFilter configurePrometheusRegistries() {
        return MeterFilter.commonTags(Arrays.asList(
                Tag.of("registry", "prometheus")));
    }

    /** Define common tags that apply globally */
    @Produces
    @Singleton
    public MeterFilter configureAllRegistries() {
        return MeterFilter.commonTags(Arrays.asList(
                Tag.of("env", deploymentEnv)));
    }

    /** Enable histogram buckets for a specific timer */
    @Produces
    @Singleton
    public MeterFilter enableHistogram() {
        return new MeterFilter() {
            @Override
            public DistributionStatisticConfig configure(Meter.Id id, DistributionStatisticConfig config) {
                if(id.getName().startsWith("myservice")) {
                    return DistributionStatisticConfig.builder()
                        .percentiles(0.5, 0.95)     // median and 95th percentile, not aggregable
                        .percentilesHistogram(true) // histogram buckets (e.g. prometheus histogram_quantile)
                        .build()
                        .merge(config);
                }
                return config;
            }
        };
    }
}

In this example, a singleton CDI bean will produce two different MeterFilter beans. One will be applied only to Prometheus MeterRegistry instances (using the @MeterFilterConstraint qualifier), and another will be applied to all MeterRegistry instances. An application configuration property is also injected and used as a tag value. Additional examples of MeterFilters can be found in the official documentation.

Does Micrometer support annotations?

Micrometer does define two annotations, @Counted and @Timed, that can be added to methods. The @Timed annotation will wrap the execution of a method and will emit the following tags in addition to any tags defined on the annotation itself: class, method, and exception (either "none" or the simple class name of a detected exception).

Using annotations is limited, as you can’t dynamically assign meaningful tag values. Also note that many methods, e.g. REST endpoint methods or Vert.x Routes, are counted and timed by the micrometer extension out of the box.

Using other Registry implementations

If you aren’t using Prometheus, you have a few options. Some Micrometer registry implementations have been wrapped in Quarkiverse extensions. To use the Micrometer StackDriver MeterRegistry, for example, you would use the quarkus-micrometer-registry-stackdriver extension:

<dependency>
    <groupId>io.quarkiverse.micrometer.registry</groupId>
    <artifactId>quarkus-micrometer-registry-stackdriver</artifactId>
</dependency>

If the Micrometer registry you would like to use does not yet have an associated extension, use the quarkus-micrometer extension and bring in the packaged MeterRegistry dependency directly:

<dependency>
    <groupId>io.quarkus</groupId>
    <artifactId>quarkus-micrometer</artifactId>
</dependency>
<dependency>
    <groupId>com.acme</groupId>
    <artifactId>custom-micrometer-registry</artifactId>
</dependency>

You will then need to specify your own provider to configure and initialize the MeterRegistry, as discussed in the next section.

Creating a customized MeterRegistry

Use a custom @Produces method to create and configure a customized MeterRegistry if you need to.

The following example customizes the line format used for StatsD:

@Produces
@Singleton
public StatsdMeterRegistry createStatsdMeterRegistry(StatsdConfig statsdConfig, Clock clock) {
    // define what to do with lines
    Consumer<String> lineLogger = line -> logger.info(line);

    // inject a configuration object, and then customize the line builder
    return StatsdMeterRegistry.builder(statsdConfig)
          .clock(clock)
          .lineSink(lineLogger)
          .build();
}

This example corresponds to the following instructions in the Micrometer documentation: https://micrometer.io/docs/registry/statsD#_customizing_the_metrics_sink

Note that the method returns the specific type of MeterRegistry as a @Singleton. Use MicroProfile Config to inject any configuration attributes you need to configure the registry. Most Micrometer registry extensions, like quarkus-micrometer-registry-statsd, define a producer for registry-specific configuration objects that are integrated with the Quarkus configuration model.

Support for the MicroProfile Metrics API

If you use the MicroProfile Metrics API in your application, the Micrometer extension will create an adaptive layer to map those metrics into the Micrometer registry. Note that naming conventions between the two systems is different, so the metrics that are emitted when using MP Metrics with Micrometer will change. You can use a MeterFilter to remap names or tags according to your conventions.

@Produces
@Singleton
public MeterFilter renameApplicationMeters() {
    final String targetMetric = MPResourceClass.class.getName() + ".mpAnnotatedMethodName";

    return MeterFilter() {
        @Override
        public Meter.Id map(Meter.Id id) {
            if (id.getName().equals(targetMetric)) {
                // Drop the scope tag (MP Registry type: application, vendor, base)
                List<Tag> tags = id.getTags().stream().filter(x -> !"scope".equals(x.getKey()))
                        .collect(Collectors.toList());
                // rename the metric
                return id.withName("my.metric.name").replaceTags(tags);
            }
            return id;
        }
    };
}

Ensure the following dependency is present in your pom.xml file if you require the Microprofile Metrics API:

<dependency>
  <groupId>org.eclipse.microprofile.metrics</groupId>
  <artifactId>microprofile-metrics-api</artifactId>
</dependency>
The MP Metrics API compatibility layer will be moved to a different extension in the future.

Configuration Reference

Configuration property fixed at build time - All other configuration properties are overridable at runtime

Configuration property

Type

Default

Micrometer metrics support. Micrometer metrics support is enabled by default.

boolean

true

Micrometer MeterRegistry discovery. Micrometer MeterRegistry implementations discovered on the classpath will be enabled automatically by default.

boolean

true

Micrometer MeterBinder discovery. Micrometer MeterBinder implementations discovered on the classpath will be enabled automatically by default.

boolean

true

Outbound HTTP request metrics support. Support for HTTP client metrics will be enabled if Micrometer support is enabled, the REST client feature is enabled, and either this value is true, or this value is unset and quarkus.micrometer.binder-enabled-default is true.

boolean

Inbound HTTP metrics support. Support for HTTP server metrics will be enabled if Micrometer support is enabled, an extension serving HTTP traffic is enabled, and either this value is true, or this value is unset and quarkus.micrometer.binder-enabled-default is true.

boolean

Micrometer JVM metrics support. Support for JVM metrics will be enabled if Micrometer support is enabled, and either this value is true, or this value is unset and quarkus.micrometer.binder-enabled-default is true.

boolean

Kafka metrics support. Support for Kafka metrics will be enabled if Micrometer support is enabled, the Kafka Consumer or Producer interface is on the classpath and either this value is true, or this value is unset and quarkus.micrometer.binder-enabled-default is true.

boolean

Eclipse MicroProfile Metrics support.

Support for MicroProfile Metrics will be enabled if Micrometer support is enabled and the MicroProfile Metrics dependency is present:

<dependency>
  <groupId>org.eclipse.microprofile.metrics</groupId>
  <artifactId>microprofile-metrics-api</artifactId>
</dependency>

The Micrometer extension currently provides a compatibility layer that supports the MP Metrics API, but metric names and recorded values will be different. Note that the MP Metrics compatibility layer will move to a different extension in the future.

boolean

Micrometer System metrics support. Support for System metrics will be enabled if Micrometer support is enabled, and either this value is true, or this value is unset and quarkus.micrometer.binder-enabled-default is true.

boolean

Vert.x metrics support. Support for Vert.x metrics will be enabled if Micrometer support is enabled, Vert.x MetricsOptions is on the classpath and either this value is true, or this value is unset and quarkus.micrometer.binder-enabled-default is true.

boolean

Support for export to JSON format. Off by default.

boolean

false

The path for the JSON metrics endpoint. The default value is metrics.

string

metrics

Statistics like max, percentiles, and histogram counts decay over time to give greater weight to recent samples. Samples are accumulated to such statistics in ring buffers which rotate after the expiry, with this buffer length.

int

3

Statistics like max, percentiles, and histogram counts decay over time to give greater weight to recent samples. Samples are accumulated to such statistics in ring buffers which rotate after this expiry, with a particular buffer length.

Duration

P3D

Support for export to Prometheus. Support for Prometheus will be enabled if Micrometer support is enabled, the PrometheusMeterRegistry is on the classpath and either this value is true, or this value is unset and quarkus.micrometer.registry-enabled-default is true.

boolean

The path for the prometheus metrics endpoint (produces text/plain). The default value is metrics.

string

metrics

By default, this extension will create a Prometheus MeterRegistry instance. Use this attribute to veto the creation of the default Prometheus MeterRegistry.

boolean

true

Comma-separated list of regular expressions used to specify uri labels in http metrics.

Outbount HTTP client instrumentation will attempt to transform parameterized resource paths, /item/123, into a generic form, /item/{id}, to reduce the cardinality of uri label values.

Patterns specified here will take precedence over those computed values.

For example, if /item/\\\\d+=/item/custom or /item/[0-9]+=/item/custom is specified in this list, a request to a matching path (/item/123) will use the specified replacement value (/item/custom) as the value for the uri label. Note that backslashes must be double escaped as \\\\.

list of string

Comma-separated list of regular expressions defining uri paths that should be ignored (not measured).

list of string

Maximum number of unique URI tag values allowed. After the max number of tag values is reached, metrics with additional tag values are denied by filter.

int

100

Comma-separated list of regular expressions used to specify uri labels in http metrics.

Vertx instrumentation will attempt to transform parameterized resource paths, /item/123, into a generic form, /item/{id}, to reduce the cardinality of uri label values.

Patterns specified here will take precedence over those computed values.

For example, if /item/\\\\d+=/item/custom or /item/[0-9]+=/item/custom is specified in this list, a request to a matching path (/item/123) will use the specified replacement value (/item/custom) as the value for the uri label. Note that backslashes must be double escaped as \\\\.

list of string

Comma-separated list of regular expressions defining uri paths that should be ignored (not measured).

list of string

Maximum number of unique URI tag values allowed. After the max number of tag values is reached, metrics with additional tag values are denied by filter.

int

100

Prometheus registry configuration properties.

A property source for configuration of the Prometheus MeterRegistry, see https://micrometer.io/docs/registry/prometheus.

Map<String,String>

About the Duration format

The format for durations uses the standard java.time.Duration format. You can learn more about it in the Duration#parse() javadoc.

You can also provide duration values starting with a number. In this case, if the value consists only of a number, the converter treats the value as seconds. Otherwise, PT is implicitly prepended to the value to obtain a standard java.time.Duration format.