By Version

Edit this Page

Observability Dev Services with Grafana OTel LGTM

OTel-LGTM is all-in-one Docker image containing OpenTelemetry’s OTLP as the protocol to transport metrics, tracing and logging data to an OpenTelemetry Collector which then stores signals data into Prometheus (metrics), Tempo (traces) and Loki (logs), only to have it visualized by Grafana. It’s used by Quarkus Observability to provide the Grafana OTel LGTM Dev Resource.

Configuring your project

Add the Quarkus Grafana OTel LGTM sink (where data goes) extension to your build file:

pom.xml

<dependency>
    <groupId>io.quarkus</groupId>
    <artifactId>quarkus-observability-devservices-lgtm</artifactId>
    <scope>provided</scope>
</dependency>

build.gradle

implementation("io.quarkus:quarkus-observability-devservices-lgtm")

Metrics

If you need metrics, add the Micrometer OTLP registry to your build file:

pom.xml

<dependency>
    <groupId>io.quarkiverse.micrometer.registry</groupId>
    <artifactId>quarkus-micrometer-registry-otlp</artifactId>
</dependency>

build.gradle

implementation("io.quarkiverse.micrometer.registry:quarkus-micrometer-registry-otlp")

When using the MicroMeter’s Quarkiverse OTLP registry to push metrics to Grafana OTel LGTM, the quarkus.micrometer.export.otlp.url property is automatically set to OTel collector endpoint as seen from the outside of the docker container.

Tracing

For Tracing add the quarkus-opentelemetry extension to your build file:

pom.xml

<dependency>
    <groupId>io.quarkus</groupId>
    <artifactId>quarkus-opentelemetry</artifactId>
</dependency>

build.gradle

implementation("io.quarkus:quarkus-opentelemetry")

The quarkus.otel.exporter.otlp.endpoint property is automatically set to OTel collector endpoint as seen from the outside of the docker container.

The quarkus.otel.exporter.otlp.protocol is set to http/protobuf.

Access Grafana

Once you start your app in dev mode:

CLI

quarkus dev

Maven

./mvnw quarkus:dev

Gradle

./gradlew --console=plain quarkusDev

You will see a log entry like this:

[io.qu.ob.de.ObservabilityDevServiceProcessor] (build-35) Dev Service Lgtm started, config: {grafana.endpoint=http://localhost:42797, quarkus.otel.exporter.otlp.endpoint=http://localhost:34711, otel-collector.url=localhost:34711, quarkus.micrometer.export.otlp.url=http://localhost:34711/v1/metrics, quarkus.otel.exporter.otlp.protocol=http/protobuf}

Remember that Grafana is accessible in an ephemeral port, so you need to check the logs to see which port is being used. In this example, the grafana endpoint is grafana.endpoint=http://localhost:42797.

If you miss the message you can always check the port with this Docker command:

docker ps | grep grafana

Another option is to use the Dev UI as the Grafana URL link will be available and if selected will open a new browser tab directly to the running Grafana instance:

Additional configuration

This extension will configure your quarkus-opentelemetry and quarkus-micrometer-registry-otlp extensions to send data to the OTel Collector bundled with the Grafana OTel LGTM image.

If you don’t want all the hassle with Dev Services (e.g. lookup and re-use of existing running containers, etc) you can simply disable Dev Services and enable just Dev Resource usage:

quarkus.observability.enabled=false
quarkus.observability.dev-resources=true

Tests

And for the least 'auto-magical' usage in the tests, simply disable both (Dev Resources are already disabled by default):

quarkus.observability.enabled=false

And then explicitly list LGTM Dev Resource in the test as a @QuarkusTestResource resource:

@QuarkusTest
@QuarkusTestResource(value = LgtmResource.class, restrictToAnnotatedClass = true)
@TestProfile(QuarkusTestResourceTestProfile.class)
public class LgtmLifecycleTest extends LgtmTestBase {
}

Testing full Grafana OTel LGTM stack - example

Use existing Quarkus MicroMeter OTLP registry

pom.xml

<dependency>
    <groupId>io.quarkiverse.micrometer.registry</groupId>
    <artifactId>quarkus-micrometer-registry-otlp</artifactId>
</dependency>

build.gradle

implementation("io.quarkiverse.micrometer.registry:quarkus-micrometer-registry-otlp")

Simply inject the Meter registry into your code — it will periodically push metrics to Grafana LGTM’s OTLP HTTP endpoint.

@Path("/api")
public class SimpleEndpoint {
    private static final Logger log = Logger.getLogger(SimpleEndpoint.class);

    @Inject
    MeterRegistry registry;

    @PostConstruct
    public void start() {
        Gauge.builder("xvalue", arr, a -> arr[0])
                .baseUnit("X")
                .description("Some random x")
                .tag("my_key", "x")
                .register(registry);
    }

    // ...
}

Where you can then check Grafana’s datasource API for existing metrics data.

public class LgtmTestBase {

    @ConfigProperty(name = "grafana.endpoint")
    String endpoint; // NOTE -- injected Grafana endpoint!

    @Test
    public void testTracing() {
        String response = RestAssured.get("/api/poke?f=100").body().asString();
        System.out.println(response);
        GrafanaClient client = new GrafanaClient(endpoint, "admin", "admin");
        Awaitility.await().atMost(61, TimeUnit.SECONDS).until(
                client::user,
                u -> "admin".equals(u.login));
        Awaitility.await().atMost(61, TimeUnit.SECONDS).until(
                () -> client.query("xvalue_X"),
                result -> !result.data.result.isEmpty());
    }

}

// simple Grafana HTTP client

public class GrafanaClient {
    private static final ObjectMapper MAPPER = new ObjectMapper();

    private final String url;
    private final String username;
    private final String password;

    public GrafanaClient(String url, String username, String password) {
        this.url = url;
        this.username = username;
        this.password = password;
    }

    private <T> void handle(
            String path,
            Function<HttpRequest.Builder, HttpRequest.Builder> method,
            HttpResponse.BodyHandler<T> bodyHandler,
            BiConsumer<HttpResponse<T>, T> consumer) {
        try {
            String credentials = username + ":" + password;
            String encodedCredentials = Base64.getEncoder().encodeToString(credentials.getBytes());

            HttpClient httpClient = HttpClient.newHttpClient();
            HttpRequest.Builder builder = HttpRequest.newBuilder()
                    .uri(URI.create(url + path))
                    .header("Authorization", "Basic " + encodedCredentials);
            HttpRequest request = method.apply(builder).build();

            HttpResponse<T> response = httpClient.send(request, bodyHandler);
            int code = response.statusCode();
            if (code < 200 || code > 299) {
                throw new IllegalStateException("Bad response: " + code + " >> " + response.body());
            }
            consumer.accept(response, response.body());
        } catch (IOException e) {
            throw new UncheckedIOException(e);
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            throw new RuntimeException(e);
        }
    }

    public User user() {
        AtomicReference<User> ref = new AtomicReference<>();
        handle(
                "/api/user",
                HttpRequest.Builder::GET,
                HttpResponse.BodyHandlers.ofString(),
                (r, b) -> {
                    try {
                        User user = MAPPER.readValue(b, User.class);
                        ref.set(user);
                    } catch (JsonProcessingException e) {
                        throw new UncheckedIOException(e);
                    }
                });
        return ref.get();
    }

    public QueryResult query(String query) {
        AtomicReference<QueryResult> ref = new AtomicReference<>();
        handle(
                "/api/datasources/proxy/1/api/v1/query?query=" + query,
                HttpRequest.Builder::GET,
                HttpResponse.BodyHandlers.ofString(),
                (r, b) -> {
                    try {
                        QueryResult result = MAPPER.readValue(b, QueryResult.class);
                        ref.set(result);
                    } catch (JsonProcessingException e) {
                        throw new UncheckedIOException(e);
                    }
                });
        return ref.get();
    }
}

Configuring your project
Testing full Grafana OTel LGTM stack - example