Observability Dev Services with Grafana OTel LGTM
OTel-LGTM is all-in-one Docker image containing OpenTelemetry’s OTLP as the protocol to transport metrics, tracing and logging data to an OpenTelemetry Collector which then stores signals data into Prometheus (metrics), Tempo (traces) and Loki (logs), only to have it visualized by Grafana. It’s used by Quarkus Observability to provide the Grafana OTel LGTM Dev Resource.
Configuring your project
Add the Quarkus Grafana OTel LGTM sink (where data goes) extension to your build file:
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-observability-devservices-lgtm</artifactId>
<scope>provided</scope>
</dependency>implementation("quarkus-observability-devservices-lgtm")Metrics
If you’re using MicroMeter’s Quarkiverse OTLP registry to push metrics to Grafana OTel LGTM, this is how you would define the export endpoint url; where quarkus.otel-collector.url is provided by the Observability Dev Services extension.
# Micrometer OTLP registry
%test.quarkus.micrometer.export.otlp.url=http://${quarkus.otel-collector.url}/v1/metrics
%dev.quarkus.micrometer.export.otlp.url=http://${quarkus.otel-collector.url}/v1/metrics
%prod.quarkus.micrometer.export.otlp.url=http://localhost:4318/v1/metricsPlease note that the ${quarkus.otel-collector.url} value is generated by quarkus when it starts the Grafana OTel LGTM Dev Resource.
Along OTel collector enpoint url, LGTM Dev Resource also provides a Grafana endpoint url - under quarkus.grafana.url property.
In this case LGTM Dev Resource would be automatically started and used by Observability Dev Services.
If you don’t want all the hassle with Dev Services (e.g. lookup and re-use of existing running containers, etc) you can simply disable Dev Services and enable just Dev Resource usage:
quarkus.observability.enabled=false
quarkus.observability.dev-resources=trueTracing
Just add the quarkus-opentelemetry extension to your build file:
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-opentelemetry</artifactId>
</dependency>implementation("io.quarkus:quarkus-opentelemetry")On the application.properties file, you can define:
# OpenTelemetry
quarkus.otel.exporter.otlp.traces.protocol=http/protobuf
%test.quarkus.otel.exporter.otlp.traces.endpoint=http://${quarkus.otel-collector.url}
%dev.quarkus.otel.exporter.otlp.traces.endpoint=http://${quarkus.otel-collector.url}
%prod.quarkus.otel.exporter.otlp.traces.endpoint=http://localhost:4318Access Grafana
Once you start your app in dev mode:
quarkus dev./mvnw quarkus:dev./gradlew --console=plain quarkusDevYou will see a message like this:
Lgtm Dev Services Starting: 2024-02-20 11:15:24,540 INFO [org.tes.con.wai.str.HttpWaitStrategy] (build-32) /loving_chatelet: Waiting for 60 seconds for URL: http://localhost:61907/ (where port 61907 maps to container port 3000)Remember that Grafana is accessible in an ephemeral port, so you need to check the logs to see which port is being used. In this example, it’s port 61907.
If you miss the message you can always check the port with this Docker command:
docker ps | grep grafanaTests
And for the least 'auto-magical' usage in the tests, simply disable both (Dev Resources are already disabled by default):
quarkus.observability.enabled=falseAnd then explicitly list LGTM Dev Resource in the test as a @QuarkusTestResource resource:
@QuarkusTest
@QuarkusTestResource(value = LgtmResource.class, restrictToAnnotatedClass = true)
@TestProfile(QuarkusTestResourceTestProfile.class)
public class LgtmLifecycleTest extends LgtmTestBase {
}Testing full Grafana OTel LGTM stack - example
Use existing Quarkus MicroMeter OTLP registry
<dependency>
<groupId>io.quarkiverse.micrometer.registry</groupId>
<artifactId>quarkus-micrometer-registry-otlp</artifactId>
</dependency>implementation("io.quarkiverse.micrometer.registry:quarkus-micrometer-registry-otlp")On the test application.properties file, you need to define:
# Micrometer OTLP registry
quarkus.micrometer.export.otlp.url=http://${quarkus.otel-collector.url}/v1/metrics
# OpenTelemetry
quarkus.otel.exporter.otlp.traces.protocol=http/protobuf
quarkus.otel.exporter.otlp.traces.endpoint=http://${quarkus.otel-collector.url}Simply inject the Meter registry into your code — it will periodically push metrics to Grafana LGTM’s OTLP HTTP endpoint.
@Path("/api")
public class SimpleEndpoint {
private static final Logger log = Logger.getLogger(SimpleEndpoint.class);
@Inject
MeterRegistry registry;
@PostConstruct
public void start() {
Gauge.builder("xvalue", arr, a -> arr[0])
.baseUnit("X")
.description("Some random x")
.tag("my_key", "x")
.register(registry);
}
// ...
}Where you can then check Grafana’s datasource API for existing metrics data.
public class LgtmTestBase {
@ConfigProperty(name = "quarkus.grafana.url")
String url; // NOTE -- injected Grafana endpoint url!
@Test
public void testTracing() {
String response = RestAssured.get("/api/poke?f=100").body().asString();
System.out.println(response);
GrafanaClient client = new GrafanaClient("http://" + url, "admin", "admin");
Awaitility.await().atMost(61, TimeUnit.SECONDS).until(
client::user,
u -> "admin".equals(u.login));
Awaitility.await().atMost(61, TimeUnit.SECONDS).until(
() -> client.query("xvalue_X"),
result -> !result.data.result.isEmpty());
}
}
// simple Grafana HTTP client
public class GrafanaClient {
private static final ObjectMapper MAPPER = new ObjectMapper();
private final String url;
private final String username;
private final String password;
public GrafanaClient(String url, String username, String password) {
this.url = url;
this.username = username;
this.password = password;
}
private <T> void handle(
String path,
Function<HttpRequest.Builder, HttpRequest.Builder> method,
HttpResponse.BodyHandler<T> bodyHandler,
BiConsumer<HttpResponse<T>, T> consumer) {
try {
String credentials = username + ":" + password;
String encodedCredentials = Base64.getEncoder().encodeToString(credentials.getBytes());
HttpClient httpClient = HttpClient.newHttpClient();
HttpRequest.Builder builder = HttpRequest.newBuilder()
.uri(URI.create(url + path))
.header("Authorization", "Basic " + encodedCredentials);
HttpRequest request = method.apply(builder).build();
HttpResponse<T> response = httpClient.send(request, bodyHandler);
int code = response.statusCode();
if (code < 200 || code > 299) {
throw new IllegalStateException("Bad response: " + code + " >> " + response.body());
}
consumer.accept(response, response.body());
} catch (IOException e) {
throw new UncheckedIOException(e);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new RuntimeException(e);
}
}
public User user() {
AtomicReference<User> ref = new AtomicReference<>();
handle(
"/api/user",
HttpRequest.Builder::GET,
HttpResponse.BodyHandlers.ofString(),
(r, b) -> {
try {
User user = MAPPER.readValue(b, User.class);
ref.set(user);
} catch (JsonProcessingException e) {
throw new UncheckedIOException(e);
}
});
return ref.get();
}
public QueryResult query(String query) {
AtomicReference<QueryResult> ref = new AtomicReference<>();
handle(
"/api/datasources/proxy/1/api/v1/query?query=" + query,
HttpRequest.Builder::GET,
HttpResponse.BodyHandlers.ofString(),
(r, b) -> {
try {
QueryResult result = MAPPER.readValue(b, QueryResult.class);
ref.set(result);
} catch (JsonProcessingException e) {
throw new UncheckedIOException(e);
}
});
return ref.get();
}
}
