Using OpenTelemetry
This guide explains how your Quarkus application can utilize OpenTelemetry (OTel) to provide Observability for interactive web applications.
On these page we show the signal independent features of the extension.
|
Introduction
OpenTelemetry is an Observability framework and toolkit designed to create and manage telemetry data such as traces, metrics, and logs. Crucially, OpenTelemetry is vendor- and tool-agnostic.
Quarkus provides manual and automatic instrumentation for tracing and manual instrumentation capabilities for metrics.
This will allow Quarkus based applications to be observable by tools and services supporting OpenTelemetry.
Automatic metrics instrumentation in Quarkus is done by the Quarkus Micrometer extension. We plan to provide, in the future, a bridge for those metrics to be available in OpenTelemetry as well. |
Quarkus supports the OpenTelemetry Autoconfiguration. The configurations match what you can see at
OpenTelemetry SDK Autoconfigure
with the quarkus.*
prefix.
This guide provides a crosscutting explanation of the OpenTelemetry extension and how to use it. If you need details about any particular signal (tracing or metrics), please refer to the signal specific guide.
With the introduction of OpenTelemetry Metrics, the original, single page guide had to be split according to signal types, as follows:
OpenTelemetry Tracing Guide
The tracing functionality is supported and on by default.
OpenTelemetry Metrics Guide
Enable Metrics
The metrics functionality is experimental and off by default. You will need to activate it by setting:
quarkus.otel.metrics.enabled=true
At build time on your application.properties
file.
Manual instrumentation only
For now only manual instrumentation is supported. You can use the OpenTelemetry API to create and record metrics but Quarkus is not yet providing automatic instrumentation for metrics.
In the future, we plan to bridge current Micrometer metrics to OpenTelemetry and maintain compatibility when possible.
Using the extension
If you already have your Quarkus project, you can add the quarkus-opentelemetry
extension
to it by running the following command in your project base directory:
quarkus extension add opentelemetry
./mvnw quarkus:add-extension -Dextensions='opentelemetry'
./gradlew addExtension --extensions='opentelemetry'
This will add the following to your build file:
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-opentelemetry</artifactId>
</dependency>
implementation("io.quarkus:quarkus-opentelemetry")
Create the configuration
By default, the exporters will send out data in batches, using the gRPC protocol and endpoint http://localhost:4317
.
If you need to change any of the default property values, here is an example on how to configure the default OTLP gRPC Exporter within the application, using the src/main/resources/application.properties
file:
quarkus.application.name=myservice (1)
quarkus.otel.exporter.otlp.endpoint=http://localhost:4317 (2)
quarkus.otel.exporter.otlp.headers=authorization=Bearer my_secret (3)
quarkus.log.console.format=%d{HH:mm:ss} %-5p traceId=%X{traceId}, parentId=%X{parentId}, spanId=%X{spanId}, sampled=%X{sampled} [%c{2.}] (%t) %s%e%n (4)
# Alternative to the console log
quarkus.http.access-log.pattern="...traceId=%{X,traceId} spanId=%{X,spanId}" (5)
1 | All telemetry created from the application will include an OpenTelemetry Resource attribute indicating the telemetry was created by the myservice application. If not set, it will default to the artifact id. |
2 | gRPC endpoint to send the telemetry. If not set, it will default to http://localhost:4317 . |
3 | Optional gRPC headers commonly used for authentication |
4 | Add tracing information into log messages. |
5 | You can also only put the trace info into the access log. In this case you must omit the info in the console log format. |
We provide signal agnostic configurations for the connection related properties, meaning that you can use the same properties for both tracing and metrics when you set:
quarkus.otel.exporter.otlp.endpoint=http://localhost:4317
If you need different configurations for each signal, you can use the specific properties:
quarkus.otel.exporter.otlp.traces.endpoint=http://trace-uri:4317 (1)
quarkus.otel.exporter.otlp.metrics.endpoint=http://metrics-uri:4317 (2)
quarkus.otel.exporter.otlp.logs.endpoint=http://logs-uri:4317 (3)
1 | The endpoint for the traces exporter. |
2 | The endpoint for the metrics exporter. |
3 | The endpoint for the logs exporter. |
Disable all or parts of the OpenTelemetry extension
Once you add the dependency, the extension will generate tracing data by default. To enable metrics or disable the OpenTelemetry extension globally or partially these are the properties to use (they are extracted from the config reference bellow):
Affected Signal | Property name | Default value | Description |
---|---|---|---|
All |
|
true |
If false, disable the OpenTelemetry usage at build time. |
All |
|
false |
Comes from the OpenTelemetry autoconfiguration. If true, will disable the OpenTelemetry SDK usage at runtime. |
All output |
|
true |
Deprecated for removal. If false will disable the default OTLP exporter at build time. |
Traces |
|
true |
If false, disable the OpenTelemetry tracing usage at build time. |
Traces output |
|
cdi |
List of exporters to be used for tracing, separated by commas. Has one of the values from ExporterType: |
Metrics |
|
false |
Metrics are disabled by default at build time because they are experimental. |
Metrics output |
|
cdi |
List of exporters to be used for metrics, separated by commas. Has one of the values from ExporterType: |
If you need to enable or disable the exporter at runtime, you can use the sampler because it has the ability to filter out all the spans if needed.
Particular instrumentation components can be disabled in tracing, like ignore client requests but keep server ones. For more details, please check the OpenTelemetry Tracing Guide.
Resource
A resource is a representation of the entity that is producing telemetry, it adds attributes to the exported trace or metric to characterize who is producing the telemetry. Quarkus follows the resources auto-configuration specified by the Java OpenTelemetry SDK.
Default
The following attributes are added by default to resources.
Attribute name | Content example | Origin |
---|---|---|
service.name |
"opentelemetry-quickstart" |
Value comes from the artifactId, from the |
host.name |
"myHost" |
Resolved at startup |
service.version |
"1.0-SNAPSHOT" |
Resolved at build time from the artifact version |
telemetry.sdk.language |
"java" |
Static value |
telemetry.sdk.name |
"opentelemetry" |
Resolved at build time |
telemetry.sdk.version |
"1.32.0" |
Resolved at build time |
webengine.name |
"Quarkus" |
Static value |
webengine.version |
"999-SNAPSHOT" |
Quarkus version resolved at build time |
Using configuration
You can add additional attributes by setting the quarkus.otel.resource.attributes
config property that is described in the OpenTelemetry Configuration Reference.
Since this property can be overridden at runtime, the OpenTelemetry extension will pick up its value following the order of precedence that
is described in the Quarkus Configuration Reference.
quarkus.otel.resource.attributes=deployment.environment=dev,service.name=cart,service.namespace=shopping
This will add the attributes for deployment.environment
, service.name
and service.namespace
to the resource and be included in traces and metrics.
Using CDI beans
If by any means you need to use a custom resource or one that is provided by one of the OpenTelemetry SDK Extensions
you can create multiple resource producers. The OpenTelemetry extension will detect the Resource
CDI beans and will merge them when configuring the OTel SDK.
@ApplicationScoped
public class CustomConfiguration {
@Produces
@ApplicationScoped
public Resource osResource() {
return OsResource.get();
}
@Produces
@ApplicationScoped
public Resource ecsResource() {
return EcsResource.get();
}
}
Semantic conventions
OpenTelemetry provides a set of semantic conventions to standardize the data collected by the instrumentation.
When creating manual instrumentation, while naming metrics or attributes you should follow those conventions and not create new names to represent existing conventions. This will make data correlation easier to perform across services.
Exporters
Default
The Quarkus OpenTelemetry extension uses its own signal exporters built on top of Vert.x for optimal performance and maintainability.
The exporter is automatically wired with CDI, that’s why the quarkus.otel.traces.exporter
and quarkus.otel.metrics.exporter
properties default to cdi
.
The quarkus.otel.exporter.otlp.protocol
defaults to grpc
but http/protobuf
can also be used.
If you change the protocol, you also need to change the port in the endpoint. The default port for grpc is 4317 and for http/protobuf is 4318 .
|
On Quarkiverse
Additional exporters will be available in the Quarkiverse quarkus-opentelemetry-exporter project.
Currently, are available the following exporters (may be outdated) for:
-
Legacy Jaeger
-
Microsoft Azure
-
Google Cloud
Also on Quarkiverse, the Quarkus AWS SDK has integration with OpenTelemetry.
Logging exporter (for debugging)
You can output all metrics to the console, for debugging/development purposes.
Don’t use this in production. |
You will need to add the following dependency to your project:
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-exporter-logging</artifactId>
</dependency>
implementation("io.opentelemetry:opentelemetry-exporter-logging")
Then, setting the exporter to logging
in the application.properties
file:
quarkus.otel.metrics.exporter=logging (1)
quarkus.otel.metric.export.interval=10000ms (2)
quarkus.otel.traces.exporter=logging (3)
1 | Set the metrics exporter to logging . Normally you don’t need to set this. The default is cdi . |
2 | Set the interval to export the metrics. The default is 1m , which is too long for debugging. |
3 | Set the traces exporter to logging . Normally you don’t need to set this. The default is cdi . |
Visualizing the data
We recommend the Getting Started with Grafana-OTel-LGTM.
This provides a Quarkus Dev service using an "all-in-one" Grafana OTel LGTM.
Grafana is used to visualize data, Loki to store logs, Tempo to store traces and Prometheus to store metrics. Also provides and OTel collector to receive the data.
This provides an easy way to visualize all OpenTelemetry data generated by the application.
You can also use the logging exporter to output all traces and metrics to the console.
OpenTelemetry Configuration Reference
Quarkus supports the OpenTelemetry Autoconfiguration for Traces.
The configurations match what you can see at
OpenTelemetry SDK Autoconfigure
adding the usual quarkus.*
prefix.
Quarkus OpenTelemetry configuration properties now have the quarkus.otel.*
prefix.
Configuration property fixed at build time - All other configuration properties are overridable at runtime
Configuration property |
Type |
Default |
---|---|---|
If false, disable the OpenTelemetry usage at build time. All other Otel properties will be ignored at runtime. Will pick up value from legacy property quarkus.opentelemetry.enabled Defaults to Environment variable: Show more |
boolean |
|
List of exporters supported by Quarkus. List of exporters to be used for tracing, separated by commas. Has one of the values on Default on Quarkus is Environment variable: Show more |
list of string |
|
The sampler to use for tracing. Has one of the values on Fallbacks to the legacy property Environment variable: Show more |
string |
|
If OpenTelemetry End User attributes should be added as Span attributes on a best-efforts basis. Environment variable: Show more |
boolean |
|
Enable metrics with OpenTelemetry. This property is not available in the Open Telemetry SDK. It’s Quarkus specific. Support for metrics will be enabled if OpenTelemetry support is enabled and either this value is true, or this value is unset. Environment variable: Show more |
boolean |
|
The Metrics exporter to use. Environment variable: Show more |
list of string |
|
Enable logs with OpenTelemetry. This property is not available in the Open Telemetry SDK. It’s Quarkus specific. Support for logs will be enabled if OpenTelemetry support is enabled and either this value is true, or this value is unset. Environment variable: Show more |
boolean |
|
The Logs exporter to use. Environment variable: Show more |
list of string |
|
The propagators to be used. Use a comma-separated list for multiple propagators. Has values from Default is Environment variable: Show more |
list of string |
|
Enables instrumentation for gRPC. Environment variable: Show more |
boolean |
|
Enables instrumentation for Messaging. Environment variable: Show more |
boolean |
|
Enables instrumentation for REST Client backed by RESTEasy Classic. Environment variable: Show more |
boolean |
|
Enables instrumentation for Quarkus REST. Environment variable: Show more |
boolean |
|
Enables instrumentation for RESTEasy Classic. Environment variable: Show more |
boolean |
|
Whether exporting of the security events is enabled. Environment variable: Show more |
boolean |
|
Selects security event types. Environment variable: Show more |
list of |
|
If true, disable the OpenTelemetry SDK. Runtime configuration. Defaults to Environment variable: Show more |
boolean |
|
Suppress non-application uris from trace collection. This will suppress tracing of Providing a custom This is a Quarkus specific property. Suppressing non-application uris is enabled by default. Fallbacks to the legacy property Environment variable: Show more |
boolean |
|
Include static resources from trace collection. This is a Quarkus specific property. Include static resources is disabled by default. Providing a custom Fallbacks to the legacy property Environment variable: Show more |
boolean |
|
Sampler argument. Depends on the When setting the stock sampler to Defaults to Environment variable: Show more |
string |
|
The interval, between the start of two metric export attempts. Default is 1min. Environment variable: Show more |
|
|
Determine whether to enable the OpenTelemetry logging handler. This is a Quarkus specific property. The OpenTelemetry logging handler is enabled by default. Environment variable: Show more |
boolean |
|
The maximum length of attribute values. Applies to spans and logs. By default, there is no limit. Environment variable: Show more |
string |
|
The maximum number of attributes. Applies to spans, span events, span links, and logs. Default is Environment variable: Show more |
int |
|
The maximum length of span attribute values. Takes precedence over By default, there is no limit. Environment variable: Show more |
int |
|
The maximum number of attributes per span. Takes precedence over Default is Environment variable: Show more |
int |
|
The maximum number of events per span. Default is Environment variable: Show more |
int |
|
The maximum number of links per span. Default is Environment variable: Show more |
int |
|
The interval, in milliseconds, between two consecutive exports. Default is Environment variable: Show more |
|
|
The maximum queue size. Default is Environment variable: Show more |
int |
|
The maximum batch size. Default is Environment variable: Show more |
int |
|
The maximum allowed time, in milliseconds, to export data. Default is Environment variable: Show more |
|
|
The interval, in milliseconds, between two consecutive exports. Default is Environment variable: Show more |
|
|
The maximum queue size. Default is Environment variable: Show more |
int |
|
The maximum batch size. Default is Environment variable: Show more |
int |
|
The maximum allowed time, in milliseconds, to export data. Default is Environment variable: Show more |
|
|
Specify resource attributes in the following format: Environment variable: Show more |
list of string |
|
Specify logical service name. Takes precedence over service.name defined with otel.resource.attributes and from quarkus.application.name. Defaults to Environment variable: Show more |
string |
|
Specify resource attribute keys that are filtered. Environment variable: Show more |
list of string |
|
The maximum amount of time Quarkus will wait for the OpenTelemetry SDK to flush unsent spans and shutdown. Environment variable: Show more |
|
|
Enables instrumentation for Vert.x HTTP. Environment variable: Show more |
boolean |
|
Enables instrumentation for Vert.x Event Bus. Environment variable: Show more |
boolean |
|
Enables instrumentation for Vert.x SQL Client. Environment variable: Show more |
boolean |
|
Enables instrumentation for Vert.x Redis Client. Environment variable: Show more |
boolean |
|
Prioritize OpenTelemetry configuration By default, Quarkus configuration has priority over OpenTelemetry configuration. Environment variable: Show more |
boolean |
|
Sets the OTLP endpoint to send telemetry data. If unset, defaults to There is a generic property, that will apply to all signals and a signal specific one, following the pattern: If protocol is Environment variable: Show more |
string |
|
Key-value pairs to be used as headers associated with exporter requests. The format is similar to the There is a generic property, that will apply to all signals and a signal specific one, following the pattern: Environment variable: Show more |
list of string |
|
Sets the method used to compress payloads. If unset, compression is disabled. Currently supported compression methods include There is a generic property, that will apply to all signals and a signal specific one, following the pattern: Environment variable: Show more |
|
|
Sets the maximum time to wait for the collector to process an exported batch of telemetry data. If unset, defaults to `OtlpExporterRuntimeConfig#DEFAULT_TIMEOUT_SECS`s. There is a generic property, that will apply to all signals and a signal specific one, following the pattern: Environment variable: Show more |
|
|
OTLP defines the encoding of telemetry data and the protocol used to exchange data between the client and the server. Depending on the exporter, the available protocols will be different. Currently, only Please mind that changing the protocol requires changing the port in the endpoint as well. There is a generic property, that will apply to all signals and a signal specific one, following the pattern: Environment variable: Show more |
string |
|
Comma-separated list of the path to the key files (Pem format). Environment variable: Show more |
list of string |
|
Comma-separated list of the path to the certificate files (Pem format). Environment variable: Show more |
list of string |
|
Comma-separated list of the trust certificate files (Pem format). Environment variable: Show more |
list of string |
|
The name of the TLS configuration to use. If not set and the default TLS configuration is configured ( There is a generic property, that will apply to all signals and a signal specific one, following the pattern: Environment variable: Show more |
string |
|
If proxy connection must be used. There is a generic property, that will apply to all signals and a signal specific one, following the pattern: Environment variable: Show more |
boolean |
|
Set proxy username. There is a generic property, that will apply to all signals and a signal specific one, following the pattern: Environment variable: Show more |
string |
|
Set proxy password. There is a generic property, that will apply to all signals and a signal specific one, following the pattern: Environment variable: Show more |
string |
|
Set proxy port. There is a generic property, that will apply to all signals and a signal specific one, following the pattern: Environment variable: Show more |
int |
|
Set proxy host. There is a generic property, that will apply to all signals and a signal specific one, following the pattern: Environment variable: Show more |
string |
|
Sets the OTLP endpoint to send telemetry data. If unset, defaults to There is a generic property, that will apply to all signals and a signal specific one, following the pattern: If protocol is Environment variable: Show more |
string |
|
Key-value pairs to be used as headers associated with exporter requests. The format is similar to the There is a generic property, that will apply to all signals and a signal specific one, following the pattern: Environment variable: Show more |
list of string |
|
Sets the method used to compress payloads. If unset, compression is disabled. Currently supported compression methods include There is a generic property, that will apply to all signals and a signal specific one, following the pattern: Environment variable: Show more |
|
|
Sets the maximum time to wait for the collector to process an exported batch of telemetry data. If unset, defaults to `OtlpExporterRuntimeConfig#DEFAULT_TIMEOUT_SECS`s. There is a generic property, that will apply to all signals and a signal specific one, following the pattern: Environment variable: Show more |
|
|
OTLP defines the encoding of telemetry data and the protocol used to exchange data between the client and the server. Depending on the exporter, the available protocols will be different. Currently, only Please mind that changing the protocol requires changing the port in the endpoint as well. There is a generic property, that will apply to all signals and a signal specific one, following the pattern: Environment variable: Show more |
string |
|
Comma-separated list of the path to the key files (Pem format). Environment variable: Show more |
list of string |
|
Comma-separated list of the path to the certificate files (Pem format). Environment variable: Show more |
list of string |
|
Comma-separated list of the trust certificate files (Pem format). Environment variable: Show more |
list of string |
|
The name of the TLS configuration to use. If not set and the default TLS configuration is configured ( There is a generic property, that will apply to all signals and a signal specific one, following the pattern: Environment variable: Show more |
string |
|
If proxy connection must be used. There is a generic property, that will apply to all signals and a signal specific one, following the pattern: Environment variable: Show more |
boolean |
|
Set proxy username. There is a generic property, that will apply to all signals and a signal specific one, following the pattern: Environment variable: Show more |
string |
|
Set proxy password. There is a generic property, that will apply to all signals and a signal specific one, following the pattern: Environment variable: Show more |
string |
|
Set proxy port. There is a generic property, that will apply to all signals and a signal specific one, following the pattern: Environment variable: Show more |
int |
|
Set proxy host. There is a generic property, that will apply to all signals and a signal specific one, following the pattern: Environment variable: Show more |
string |
|
Sets the OTLP endpoint to send telemetry data. If unset, defaults to There is a generic property, that will apply to all signals and a signal specific one, following the pattern: If protocol is Environment variable: Show more |
string |
|
Key-value pairs to be used as headers associated with exporter requests. The format is similar to the There is a generic property, that will apply to all signals and a signal specific one, following the pattern: Environment variable: Show more |
list of string |
|
Sets the method used to compress payloads. If unset, compression is disabled. Currently supported compression methods include There is a generic property, that will apply to all signals and a signal specific one, following the pattern: Environment variable: Show more |
|
|
Sets the maximum time to wait for the collector to process an exported batch of telemetry data. If unset, defaults to `OtlpExporterRuntimeConfig#DEFAULT_TIMEOUT_SECS`s. There is a generic property, that will apply to all signals and a signal specific one, following the pattern: Environment variable: Show more |
|
|
OTLP defines the encoding of telemetry data and the protocol used to exchange data between the client and the server. Depending on the exporter, the available protocols will be different. Currently, only Please mind that changing the protocol requires changing the port in the endpoint as well. There is a generic property, that will apply to all signals and a signal specific one, following the pattern: Environment variable: Show more |
string |
|
Comma-separated list of the path to the key files (Pem format). Environment variable: Show more |
list of string |
|
Comma-separated list of the path to the certificate files (Pem format). Environment variable: Show more |
list of string |
|
Comma-separated list of the trust certificate files (Pem format). Environment variable: Show more |
list of string |
|
The name of the TLS configuration to use. If not set and the default TLS configuration is configured ( There is a generic property, that will apply to all signals and a signal specific one, following the pattern: Environment variable: Show more |
string |
|
If proxy connection must be used. There is a generic property, that will apply to all signals and a signal specific one, following the pattern: Environment variable: Show more |
boolean |
|
Set proxy username. There is a generic property, that will apply to all signals and a signal specific one, following the pattern: Environment variable: Show more |
string |
|
Set proxy password. There is a generic property, that will apply to all signals and a signal specific one, following the pattern: Environment variable: Show more |
string |
|
Set proxy port. There is a generic property, that will apply to all signals and a signal specific one, following the pattern: Environment variable: Show more |
int |
|
Set proxy host. There is a generic property, that will apply to all signals and a signal specific one, following the pattern: Environment variable: Show more |
string |
|
The preferred output aggregation temporality. Options include DELTA, LOWMEMORY, and CUMULATIVE. If CUMULATIVE, all instruments will have cumulative temporality. If DELTA, counter (sync and async) and histograms will be delta, up down counters (sync and async) will be cumulative. If LOWMEMORY, sync counter and histograms will be delta, async counter and up down counters (sync and async) will be cumulative. Default is CUMULATIVE. Environment variable: Show more |
string |
|
The preferred default histogram aggregation. Options include BASE2_EXPONENTIAL_BUCKET_HISTOGRAM and EXPLICIT_BUCKET_HISTOGRAM. Default is EXPLICIT_BUCKET_HISTOGRAM. Environment variable: Show more |
string |
|
Sets the OTLP endpoint to send telemetry data. If unset, defaults to There is a generic property, that will apply to all signals and a signal specific one, following the pattern: If protocol is Environment variable: Show more |
string |
|
Key-value pairs to be used as headers associated with exporter requests. The format is similar to the There is a generic property, that will apply to all signals and a signal specific one, following the pattern: Environment variable: Show more |
list of string |
|
Sets the method used to compress payloads. If unset, compression is disabled. Currently supported compression methods include There is a generic property, that will apply to all signals and a signal specific one, following the pattern: Environment variable: Show more |
|
|
Sets the maximum time to wait for the collector to process an exported batch of telemetry data. If unset, defaults to `OtlpExporterRuntimeConfig#DEFAULT_TIMEOUT_SECS`s. There is a generic property, that will apply to all signals and a signal specific one, following the pattern: Environment variable: Show more |
|
|
OTLP defines the encoding of telemetry data and the protocol used to exchange data between the client and the server. Depending on the exporter, the available protocols will be different. Currently, only Please mind that changing the protocol requires changing the port in the endpoint as well. There is a generic property, that will apply to all signals and a signal specific one, following the pattern: Environment variable: Show more |
string |
|
Comma-separated list of the path to the key files (Pem format). Environment variable: Show more |
list of string |
|
Comma-separated list of the path to the certificate files (Pem format). Environment variable: Show more |
list of string |
|
Comma-separated list of the trust certificate files (Pem format). Environment variable: Show more |
list of string |
|
The name of the TLS configuration to use. If not set and the default TLS configuration is configured ( There is a generic property, that will apply to all signals and a signal specific one, following the pattern: Environment variable: Show more |
string |
|
If proxy connection must be used. There is a generic property, that will apply to all signals and a signal specific one, following the pattern: Environment variable: Show more |
boolean |
|
Set proxy username. There is a generic property, that will apply to all signals and a signal specific one, following the pattern: Environment variable: Show more |
string |
|
Set proxy password. There is a generic property, that will apply to all signals and a signal specific one, following the pattern: Environment variable: Show more |
string |
|
Set proxy port. There is a generic property, that will apply to all signals and a signal specific one, following the pattern: Environment variable: Show more |
int |
|
Set proxy host. There is a generic property, that will apply to all signals and a signal specific one, following the pattern: Environment variable: Show more |
string |
About the Duration format
To write duration values, use the standard You can also use a simplified format, starting with a number:
In other cases, the simplified format is translated to the
|