By Version

Using OpenTelemetry

Prerequisites
Architecture
Solution
Creating the Maven project
- Examine the JAX-RS resource
- Create the configuration
Run the application
- JDBC
Additional configuration
Additional instrumentation
Exporters
OpenTelemetry Configuration Reference

This guide explains how your Quarkus application can utilize OpenTelemetry to provide distributed tracing for interactive web applications.

Prerequisites

To complete this guide, you need:

Roughly 15 minutes
An IDE
JDK 11+ installed with JAVA_HOME configured appropriately
Apache Maven 3.8.6
Docker and Docker Compose or Podman, and Docker Compose
Optionally the Quarkus CLI if you want to use it
Optionally Mandrel or GraalVM installed and configured appropriately if you want to build a native executable (or Docker if you use a native container build)

Architecture

In this guide, we create a straightforward REST application to demonstrate distributed tracing.

Solution

We recommend that you follow the instructions in the next sections and create the application step by step. However, you can skip right to the completed example.

Clone the Git repository: git clone https://github.com/quarkusio/quarkus-quickstarts.git, or download an archive.

The solution is located in the opentelemetry-quickstart directory.

Creating the Maven project

First, we need a new project. Create a new project with the following command:

CLI

quarkus create app org.acme:opentelemetry-quickstart \
    --extension='resteasy-reactive,quarkus-opentelemetry' \
    --no-code
cd opentelemetry-quickstart

To create a Gradle project, add the --gradle or --gradle-kotlin-dsl option.

For more information about how to install the Quarkus CLI and use it, please refer to the Quarkus CLI guide.

Maven

mvn io.quarkus.platform:quarkus-maven-plugin:2.16.5.Final:create \
    -DprojectGroupId=org.acme \
    -DprojectArtifactId=opentelemetry-quickstart \
    -Dextensions='resteasy-reactive,quarkus-opentelemetry' \
    -DnoCode
cd opentelemetry-quickstart

To create a Gradle project, add the -DbuildTool=gradle or -DbuildTool=gradle-kotlin-dsl option.

This command generates the Maven project and imports the quarkus-opentelemetry extension, which includes the default OpenTelemetry support, and a gRPC span exporter for OTLP.

If you already have your Quarkus project configured, you can add the quarkus-opentelemetry extension to your project by running the following command in your project base directory:

CLI

quarkus extension add 'opentelemetry'

Maven

./mvnw quarkus:add-extension -Dextensions='opentelemetry'

Gradle

./gradlew addExtension --extensions='opentelemetry'

This will add the following to your build file:

pom.xml

<dependency>
    <groupId>io.quarkus</groupId>
    <artifactId>quarkus-opentelemetry</artifactId>
</dependency>

build.gradle

implementation("io.quarkus:quarkus-opentelemetry")

Examine the JAX-RS resource

Create a src/main/java/org/acme/opentelemetry/TracedResource.java file with the following content:

package org.acme.opentelemetry;

import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;
import org.jboss.logging.Logger;

@Path("/hello")
public class TracedResource {

    private static final Logger LOG = Logger.getLogger(TracedResource.class);

    @GET
    @Produces(MediaType.TEXT_PLAIN)
    public String hello() {
        LOG.info("hello");
        return "hello";
    }
}

Notice that there is no tracing specific code included in the application. By default, requests sent to this endpoint will be traced without any required code changes.

Create the configuration

There are two ways to configure the default OTLP gRPC Exporter within the application.

The first approach is by providing the properties within the src/main/resources/application.properties file:

quarkus.application.name=myservice (1)
quarkus.opentelemetry.enabled=true (2)
quarkus.opentelemetry.tracer.exporter.otlp.endpoint=http://localhost:4317 (3)

quarkus.opentelemetry.tracer.exporter.otlp.headers=Authorization=Bearer my_secret (4)

quarkus.log.console.format=%d{HH:mm:ss} %-5p traceId=%X{traceId}, parentId=%X{parentId}, spanId=%X{spanId}, sampled=%X{sampled} [%c{2.}] (%t) %s%e%n  (5)

# Alternative to the console log
quarkus.http.access-log.pattern="...traceId=%{X,traceId} spanId=%{X,spanId}" (6)

1	All spans created from the application will include an OpenTelemetry `Resource` indicating the span was created by the `myservice` application. If not set, it will default to the artifact id.
2	Whether OpenTelemetry is enabled or not. The default is `true`, but shown here to indicate how it can be disabled
3	gRPC endpoint for sending spans
4	Optional gRPC headers commonly used for authentication
5	Add tracing information into log message.
6	You can also only put the trace info into the access log. In this case you must omit the info in the console log format.

Run the application

The first step is to configure and start the OpenTelemetry Collector to receive, process and export telemetry data to Jaeger that will display the captured traces.

Jaeger supports the OTel protocols out of the box since version 1.35. In this case you do not need to install the collector but can directly send the trace data to Jaeger (after enabling OTLP receivers there, see e.g. this blog entry).

Configure the OpenTelemetry Collector by creating an otel-collector-config.yaml file:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: otel-collector:4317

exporters:
  jaeger:
    endpoint: jaeger-all-in-one:14250
    tls:
      insecure: true

processors:
  batch:

extensions:
  health_check:

service:
  extensions: [health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger]

Start the OpenTelemetry Collector and Jaeger system via the following docker-compose.yml file that you can launch via docker-compose up -d:

version: "2"
services:

  # Jaeger
  jaeger-all-in-one:
    image: jaegertracing/all-in-one:latest
    ports:
      - "16686:16686"
      - "14268:14268"
      - "14250:14250"
  # Collector
  otel-collector:
    image: otel/opentelemetry-collector:latest
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml:Z
    ports:
      - "13133:13133" # Health_check extension
      - "4317:4317"   # OTLP gRPC receiver
    depends_on:
      - jaeger-all-in-one

Now we are ready to run our application. If using application.properties to configure the tracer:

CLI

quarkus dev

Maven

./mvnw quarkus:dev

Gradle

./gradlew --console=plain quarkusDev

or if configuring the OTLP gRPC endpoint via JVM arguments:

CLI

quarkus dev -Djvm.args="-Dquarkus.opentelemetry.tracer.exporter.otlp.endpoint=http://localhost:4317"

Maven

./mvnw quarkus:dev -Djvm.args="-Dquarkus.opentelemetry.tracer.exporter.otlp.endpoint=http://localhost:4317"

Gradle

./gradlew --console=plain quarkusDev -Djvm.args="-Dquarkus.opentelemetry.tracer.exporter.otlp.endpoint=http://localhost:4317"

With the OpenTelemetry Collector, the Jaeger system and the application running, you can make a request to the provided endpoint:

$ curl http://localhost:8080/hello
hello

When the first request has been submitted, you will be able to see the tracing information in the logs:

10:49:02 INFO  traceId=, parentId=, spanId=, sampled= [io.quarkus] (main) Installed features: [cdi, opentelemetry, rest-client, resteasy, smallrye-context-propagation, vertx]
10:49:03 INFO  traceId=17ceb8429b9f25b0b879fa1503259456, parentId=3125c8bee75b7ad6, spanId=58ce77c86dd23457, sampled=true [or.ac.op.TracedResource] (executor-thread-1) hello
10:49:03 INFO  traceId=ad23acd6d9a4ed3d1de07866a52fa2df, parentId=, spanId=df13f5b45cf4d1e2, sampled=true [or.ac.op.TracedResource] (executor-thread-0) hello

Then visit the Jaeger UI to see the tracing information.

Hit CTRL+C to stop the application.

JDBC

The JDBC instrumentation will add a span for each JDBC queries done by your application, to enable it, add the following dependency to your build file:

pom.xml

<dependency>
    <groupId>io.opentelemetry.instrumentation</groupId>
    <artifactId>opentelemetry-jdbc</artifactId>
</dependency>

build.gradle

implementation("io.opentelemetry.instrumentation:opentelemetry-jdbc")

As it uses a dedicated JDBC driver, you must configure your datasource and Hibernate ORM to use it.

quarkus.datasource.db-kind=postgresql
# add ':otel' to your database URL
quarkus.datasource.jdbc.url=jdbc:otel:postgresql://localhost:5432/mydatabase
# use the 'OpenTelemetryDriver' instead of the one for your database
quarkus.datasource.jdbc.driver=io.opentelemetry.instrumentation.jdbc.OpenTelemetryDriver

Additional configuration

Some use cases will require custom configuration of OpenTelemetry. These sections will outline what is necessary to properly configure it.

ID Generator

The OpenTelemetry extension will use by default a random ID Generator when creating the trace and span identifier.

Some vendor-specific protocols need a custom ID Generator, you can override the default one by creating a producer. The OpenTelemetry extension will detect the IdGenerator CDI bean and will use it when configuring the tracer producer.

@Singleton
public class CustomConfiguration {

    /** Creates a custom IdGenerator for OpenTelemetry */
    @Produces
    @Singleton
    public IdGenerator idGenerator() {
        return AwsXrayIdGenerator.getInstance();
    }
}

Propagators

OpenTelemetry propagates cross-cutting concerns through propagators that will share an underlying Context for storing state and accessing data across the lifespan of a distributed transaction.

By default, the OpenTelemetry extension enables the W3C Trace Context and the W3C Baggage propagators, you can however choose any of the supported OpenTelemetry propagators by setting the propagators config that is described in the OpenTelemetry Configuration Reference.

The b3, b3multi, jaeger and ottrace propagators will need the trace-propagators extension to be added as a dependency to your project.

pom.xml

<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-extension-trace-propagators</artifactId>
</dependency>

build.gradle

implementation("io.opentelemetry:opentelemetry-extension-trace-propagators")

The xray propagator will need the aws extension to be added as a dependency to your project.

pom.xml

<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-extension-aws</artifactId>
</dependency>

build.gradle

implementation("io.opentelemetry:opentelemetry-extension-aws")

Resource

A resource is a representation of the entity that is producing telemetry, it adds attributes to the exported trace to characterize who is producing the trace.

You can add attributes by setting the resource-attributes tracer config that is described in the OpenTelemetry Configuration Reference. Since this property can be overridden at runtime, the OpenTelemetry extension will pick up its value following the order of precedence that is described in the Quarkus Configuration Reference.

If by any means you need to use a custom resource or one that is provided by one of the OpenTelemetry SDK Extensions you can create multiple resource producers. The OpenTelemetry extension will detect the Resource CDI beans and will merge them when configuring the tracer producer.

@ApplicationScoped
public class CustomConfiguration {

    @Produces
    @ApplicationScoped
    public Resource osResource() {
        return OsResource.get();
    }

    @Produces
    @ApplicationScoped
    public Resource ecsResource() {
        return EcsResource.get();
    }
}

Sampler

A sampler decides whether a trace should be sampled and exported, controlling noise and overhead by reducing the number of sample of traces collected and sent to the collector.

You can set a built-in sampler simply by setting the desired sampler config described in the OpenTelemetry Configuration Reference.

If you need to use a custom sampler or to use one that is provided by one of the OpenTelemetry SDK Extensions you can create a sampler producer. The OpenTelemetry extension will detect the Sampler CDI bean and will use it when configuring the tracer producer.

@Singleton
public class CustomConfiguration {

    /** Creates a custom sampler for OpenTelemetry */
    @Produces
    @Singleton
    public Sampler sampler() {
        return JaegerRemoteSampler.builder()
        .setServiceName("my-service")
        .build();
    }
}

Additional instrumentation

Some Quarkus extensions will require additional code to ensure traces are propagated to subsequent execution. These sections will outline what is necessary to propagate traces across process boundaries.

The instrumentation documented in this section has been tested with Quarkus and works in both standard and native mode.

CDI

Annotating a method in any CDI aware bean with the io.opentelemetry.extension.annotations.WithSpan annotation will create a new Span and establish any required relationships with the current Trace context.

Method parameters can be annotated with the io.opentelemetry.extension.annotations.SpanAttribute annotation to indicate which method parameters should be part of the Trace.

Example:

@ApplicationScoped
class SpanBean {
    @WithSpan
    void span() {

    }

    @WithSpan("name")
    void spanName() {

    }

    @WithSpan(kind = SERVER)
    void spanKind() {

    }

    @WithSpan
    void spanArgs(@SpanAttribute(value = "arg") String arg) {

    }
}

Available OpenTelemetry CDI injections

As per MicroProfile Telemetry Tracing specification, Quarkus supports the CDI injections of the following classes:

io.opentelemetry.api.OpenTelemetry
io.opentelemetry.api.trace.Tracer
io.opentelemetry.api.trace.Span
io.opentelemetry.api.baggage.Baggage

You can inject these classes in any CDI enabled bean. For instance, the Tracer is particularly useful to start custom spans:

@Inject
Tracer tracer;

...

public void tracedWork() {
    Span span = tracer.spanBuilder("My custom span")
        .setAttribute("attr", "attr.value")
        .setParent(Context.current().with(Span.current()))
        .setSpanKind(SpanKind.INTERNAL)
        .startSpan();

    // traced work

    span.end();
}

SmallRye Reactive Messaging - Kafka

When using the SmallRye Reactive Messaging extension for Kafka, we are able to propagate the span into the Kafka Record with:

TracingMetadata tm = TracingMetadata.withPrevious(Context.current());
Message out = Message.of(...).withMetadata(tm);

The above creates a TracingMetadata object we can add to the Message being produced, which retrieves the OpenTelemetry Context to extract the current span for propagation.

Exporters

Quarkus OpenTelemetry defaults to the standard OTLP exporter defined in OpenTelemetry.

Additional exporters will be available in the Quarkiverse quarkus-opentelemetry-exporter project.

OpenTelemetry Configuration Reference

Configuration property fixed at build time - All other configuration properties are overridable at runtime

Configuration property	Type	Default
`quarkus.opentelemetry.enabled` OpenTelemetry support. OpenTelemetry support is enabled by default. Environment variable: `QUARKUS_OPENTELEMETRY_ENABLED`	boolean	`true`
`quarkus.opentelemetry.propagators` Comma separated list of OpenTelemetry propagators which must be supported. Valid values are `b3, b3multi, baggage, jaeger, ottrace, tracecontext, xray`. Default value is `traceContext,baggage`. Environment variable: `QUARKUS_OPENTELEMETRY_PROPAGATORS`	list of string	`tracecontext,baggage`
`quarkus.opentelemetry.tracer.enabled` Support for tracing with OpenTelemetry. Support for tracing will be enabled if OpenTelemetry support is enabled and either this value is true, or this value is unset. Environment variable: `QUARKUS_OPENTELEMETRY_TRACER_ENABLED`	boolean
`quarkus.opentelemetry.tracer.resource-attributes` A comma separated list of name=value resource attributes that represents the entity producing telemetry (eg. `service.name=authservice`). Environment variable: `QUARKUS_OPENTELEMETRY_TRACER_RESOURCE_ATTRIBUTES`	list of string
`quarkus.opentelemetry.tracer.sampler` The sampler to use for tracing. Valid values are `off, on, ratio`. Defaults to `on`. Environment variable: `QUARKUS_OPENTELEMETRY_TRACER_SAMPLER`	string	`on`
`quarkus.opentelemetry.tracer.sampler.ratio` Environment variable: `QUARKUS_OPENTELEMETRY_TRACER_SAMPLER_RATIO`	double
`quarkus.opentelemetry.tracer.sampler.parent-based` If the sampler to use for tracing is parent based. Valid values are `true, false`. Defaults to `true`. Environment variable: `QUARKUS_OPENTELEMETRY_TRACER_SAMPLER_PARENT_BASED`	boolean	`true`
`quarkus.opentelemetry.tracer.suppress-non-application-uris` Suppress non-application uris from trace collection. This will suppress tracing of `/q` endpoints. Suppressing non-application uris is enabled by default. Environment variable: `QUARKUS_OPENTELEMETRY_TRACER_SUPPRESS_NON_APPLICATION_URIS`	boolean	`true`
`quarkus.opentelemetry.tracer.include-static-resources` Include static resources from trace collection. Include static resources is disabled by default. Environment variable: `QUARKUS_OPENTELEMETRY_TRACER_INCLUDE_STATIC_RESOURCES`	boolean	`false`
`quarkus.opentelemetry.tracer.exporter.otlp.enabled` OTLP SpanExporter support. OTLP SpanExporter support is enabled by default. Environment variable: `QUARKUS_OPENTELEMETRY_TRACER_EXPORTER_OTLP_ENABLED`	boolean	`true`
`quarkus.opentelemetry.tracer.exporter.otlp.endpoint` The OTLP endpoint to connect to. The endpoint must start with either http:// or https://. Environment variable: `QUARKUS_OPENTELEMETRY_TRACER_EXPORTER_OTLP_ENDPOINT`	string
`quarkus.opentelemetry.tracer.exporter.otlp.headers` Key-value pairs to be used as headers associated with gRPC requests. The format is similar to the `OTEL_EXPORTER_OTLP_HEADERS` environment variable, a list of key-value pairs separated by the "=" character. See Specifying headers for more details. Environment variable: `QUARKUS_OPENTELEMETRY_TRACER_EXPORTER_OTLP_HEADERS`	list of string
`quarkus.opentelemetry.tracer.exporter.otlp.export-timeout` The maximum amount of time to wait for the collector to process exported spans before an exception is thrown. A value of `0` will disable the timeout: the exporter will continue waiting until either exported spans are processed, or the connection fails, or is closed for some other reason. Environment variable: `QUARKUS_OPENTELEMETRY_TRACER_EXPORTER_OTLP_EXPORT_TIMEOUT`	Duration	`10S`
`quarkus.opentelemetry.tracer.exporter.otlp.compression` Compression method to be used by exporter to compress the payload. See Configuration Options for the supported compression methods. Environment variable: `QUARKUS_OPENTELEMETRY_TRACER_EXPORTER_OTLP_COMPRESSION`	string

Configuration property

Type

Default

quarkus.opentelemetry.enabled

OpenTelemetry support. OpenTelemetry support is enabled by default.

Environment variable: QUARKUS_OPENTELEMETRY_ENABLED

boolean

true

quarkus.opentelemetry.propagators

Comma separated list of OpenTelemetry propagators which must be supported. Valid values are b3, b3multi, baggage, jaeger, ottrace, tracecontext, xray. Default value is traceContext,baggage.

Environment variable: QUARKUS_OPENTELEMETRY_PROPAGATORS

list of string

tracecontext,baggage

quarkus.opentelemetry.tracer.enabled

Support for tracing with OpenTelemetry. Support for tracing will be enabled if OpenTelemetry support is enabled and either this value is true, or this value is unset.

Environment variable: QUARKUS_OPENTELEMETRY_TRACER_ENABLED

boolean

quarkus.opentelemetry.tracer.resource-attributes

A comma separated list of name=value resource attributes that represents the entity producing telemetry (eg. service.name=authservice).

Environment variable: QUARKUS_OPENTELEMETRY_TRACER_RESOURCE_ATTRIBUTES

list of string

quarkus.opentelemetry.tracer.sampler

The sampler to use for tracing. Valid values are off, on, ratio. Defaults to on.

Environment variable: QUARKUS_OPENTELEMETRY_TRACER_SAMPLER

string

on

quarkus.opentelemetry.tracer.sampler.ratio

Environment variable: QUARKUS_OPENTELEMETRY_TRACER_SAMPLER_RATIO

double

quarkus.opentelemetry.tracer.sampler.parent-based

If the sampler to use for tracing is parent based. Valid values are true, false. Defaults to true.

Environment variable: QUARKUS_OPENTELEMETRY_TRACER_SAMPLER_PARENT_BASED

boolean

true

quarkus.opentelemetry.tracer.suppress-non-application-uris

Suppress non-application uris from trace collection. This will suppress tracing of /q endpoints. Suppressing non-application uris is enabled by default.

Environment variable: QUARKUS_OPENTELEMETRY_TRACER_SUPPRESS_NON_APPLICATION_URIS

boolean

true

quarkus.opentelemetry.tracer.include-static-resources

Include static resources from trace collection. Include static resources is disabled by default.

Environment variable: QUARKUS_OPENTELEMETRY_TRACER_INCLUDE_STATIC_RESOURCES

boolean

false

quarkus.opentelemetry.tracer.exporter.otlp.enabled

OTLP SpanExporter support. OTLP SpanExporter support is enabled by default.

Environment variable: QUARKUS_OPENTELEMETRY_TRACER_EXPORTER_OTLP_ENABLED

boolean

true

quarkus.opentelemetry.tracer.exporter.otlp.endpoint

The OTLP endpoint to connect to. The endpoint must start with either http:// or https://.

Environment variable: QUARKUS_OPENTELEMETRY_TRACER_EXPORTER_OTLP_ENDPOINT

string

quarkus.opentelemetry.tracer.exporter.otlp.headers

Key-value pairs to be used as headers associated with gRPC requests. The format is similar to the OTEL_EXPORTER_OTLP_HEADERS environment variable, a list of key-value pairs separated by the "=" character. See Specifying headers for more details.

Environment variable: QUARKUS_OPENTELEMETRY_TRACER_EXPORTER_OTLP_HEADERS

list of string

quarkus.opentelemetry.tracer.exporter.otlp.export-timeout

The maximum amount of time to wait for the collector to process exported spans before an exception is thrown. A value of 0 will disable the timeout: the exporter will continue waiting until either exported spans are processed, or the connection fails, or is closed for some other reason.

Environment variable: QUARKUS_OPENTELEMETRY_TRACER_EXPORTER_OTLP_EXPORT_TIMEOUT

Duration

10S

quarkus.opentelemetry.tracer.exporter.otlp.compression

Compression method to be used by exporter to compress the payload. See Configuration Options for the supported compression methods.

Environment variable: QUARKUS_OPENTELEMETRY_TRACER_EXPORTER_OTLP_COMPRESSION

string

About the Duration format

The format for durations uses the standard java.time.Duration format. You can learn more about it in the Duration#parse() javadoc.

You can also provide duration values starting with a number. In this case, if the value consists only of a number, the converter treats the value as seconds. Otherwise, PT is implicitly prepended to the value to obtain a standard java.time.Duration format.