Quarkus Insights #265: Observability and Evaluation with Langfuse
This summary was generated using AI, reviewed by humans - watch the video for the full story.
Quarkus Insights #265: Observability and Evaluation with Langfuse
Traditional software testing relies on deterministic inputs and outputs: given the same arguments, a function should return the exact same result, and HTTP 200 responses signal that all systems are healthy. Large language models and generative AI break these assumptions by introducing non-determinism, where code can compile cleanly, tests pass, and operational dashboards glow green while the system produces completely incorrect or hallucinated answers. In episode 265, regular host Eric Deandrea swaps seats to become the featured guest, joining hosts Holly Cummins and Dimitris Andreadis to demonstrate how to bring observability, continuous evaluation, and regression testing to AI applications using Langfuse and LangChain4j.
Quarkus News and Community Updates
Holly opened the episode with notable updates from across the community:
-
Quarkiverse release surge: A newly deployed release announcement automation recorded 27 new Quarkiverse extension releases in a single week.
-
Quarkus 4 compatibility: Quarkiverse extension maintainers continue rolling out updates to ensure day-one compatibility with Quarkus 4.
-
Quarkus:Dev conference announced: Following a community vote, the upcoming virtual conference has been named Quarkus:Dev and is scheduled for February 2nd to 3rd. Community members are encouraged to prepare story proposals and participate in the upcoming call for presentations.
-
Time zone transitions: European daylight saving changes resulted in an hour shift between European and American broadcast schedules, giving Eric an extra hour to fine-tune his live demo.
The Non-Deterministic Challenge: When Green Dashboards Hide Bad Answers
In traditional enterprise architectures, components such as databases, message queues, and downstream microservices have well-understood failure modes. Engineers use unit tests, integration tests, and resilience patterns (circuit breakers, retries, bulkheads) to verify that the system works as designed.
Generative AI models introduce a fundamentally different integration point: they are non-deterministic. A system interacting with an LLM can execute flawlessly from an infrastructure perspective, yet return an answer that is useless, inaccurate, or commercially damaging.
Eric highlighted high-profile real-world examples:
-
A car dealership chatbot that agreed to sell a vehicle for one dollar, which was subsequently upheld in court as a binding agreement.
-
An airline customer service chatbot that fabricated an incorrect bereavement refund policy, resulting in legal liability for the carrier.
In both instances, the applications did not crash. Health checks stayed green, HTTP 200 OK statuses were returned, and operational metrics showed normal latency. The system worked, but it did not work well.
To demonstrate how subtle AI bugs can be, Eric presented a common scenario in LangChain4j: modifying a prompt template string block by removing the line containing the user’s question. The Java code compiles without issue, unit tests expecting non-empty strings pass, and the LLM returns a fluent response. However, the output is completely detached from what the user asked. Simple keyword searches or regular expressions cannot reliably catch these regressions without creating brittle tests that fail randomly across model version bumps.
Redefining the Testing Pyramid for AI
Testing AI applications requires shifting from asking "Did the code execute?" to "How good was the result?" Building on concepts articulated by Angie Jones, Eric illustrated how the traditional testing pyramid evolves for AI systems:
-
Predictable baseline (Unit testing and resilience): At the foundation, developers still write standard unit tests to verify application logic and error handling. Mocking model providers with tools such as WireMock or Toxiproxy allows teams to verify how the application handles HTTP 500 errors, rate limits, or network timeouts.
-
Reproducibility and interaction flows: The middle tier focuses on recording production interactions and replaying them. In agentic architectures, this layer verifies whether agents interact properly, hand off tasks correctly, and select appropriate tools.
-
Performance and degradation tracking: Model behavior drifts over time. Tracking token usage, latency, and response quality across rolling averages ensures that infrastructure upgrades or prompt changes do not quietly degrade performance.
-
Qualitative judgment (The top tier): At the apex sits qualitative evaluation, including LLM-as-a-judge scoring, rubric evaluations, semantic similarity checks, and human-in-the-loop annotations.
Higher levels of the pyramid provide deeper confidence in system quality, but they also incur higher latency and compute costs. Sustainable testing requires balancing fast, deterministic assertions at the base with targeted evaluations higher up.
The AI Observability Landscape
Two years ago, tracking LLM performance in production required building custom logging harnesses. Today, dedicated AI observability and evaluation platforms have matured rapidly, accompanied by significant industry consolidation:
-
Arize Phoenix was acquired by Dynatrace.
-
Traceloop was acquired by ServiceNow.
-
Langfuse was acquired by ClickHouse.
-
LangSmith remains deeply integrated into the LangChain ecosystem.
These platforms specialize in capturing AI interactions, tracking token economics, organizing evaluation datasets, and measuring model performance across time.
Quarkus Dev Services for Langfuse
Deploying Langfuse for local development or production involves multiple components: a web frontend, a background worker, a PostgreSQL database, a Redis cache, and a ClickHouse analytics database. In production environments, this stack is typically deployed using Helm on platforms such as Red Hat OpenShift or via Docker.
The Quarkus Langfuse extension simplifies this operational overhead through Quarkus Dev Services:
-
When running in dev mode (
quarkus dev), Quarkus automatically provisions and starts all required containers in the background. -
The extension automatically configures project keys and connection endpoints without requiring manual setup in
application.properties. -
The Quarkus Dev UI includes a dedicated Langfuse card that links directly to the local dashboard, allowing developers to inspect traces, review token costs, and experiment with evaluators immediately.
Smart Trace Filtering with OpenTelemetry
A key architectural insight in Eric’s demo is the separation between operational observability and AI evaluation.
When an application uses the Quarkus OpenTelemetry extension, every database query, HTTP request, security check, and business method generates a span. Exporting this complete distributed trace to backend tools like Grafana or Grafana Tempo is critical for debugging infrastructure health, memory usage, and thread pools.
However, evaluation platforms like Langfuse only care about AI-specific lifecycle events: prompt inputs, completions, token usage, tool invocations, and agent steps. Sending general database and HTTP spans to an AI evaluation tool wastes storage and clutters the UI.
The Quarkus Langfuse extension addresses this by acting as an intelligent trace filter:
-
All distributed tracing data continues flowing to standard OpenTelemetry backends.
-
The Langfuse extension intercepts the trace and exports only the AI-related spans and metadata to Langfuse.
-
OpenTelemetry trace IDs are preserved across both systems, allowing developers inspecting an anomaly in Langfuse to copy the trace ID and jump directly into Grafana Tempo to examine raw application logs and surrounding infrastructure spans.
A Three-Tiered Evaluation Strategy
Eric demonstrated a cumulative three-tiered approach to evaluating AI systems in production and development:
Tier 1: Real-Time Interaction Evaluation (LLM-as-a-Judge)
Every individual request and response sent through LangChain4j is captured as an observation in Langfuse. An asynchronous evaluator assesses the quality of the completion:
-
Independent model families: To prevent evaluation bias, the evaluator uses a different model family than the application. For example, an application using OpenAI ChatGPT can be evaluated by an LLM-as-a-judge running Google Gemini 2.5 Flash.
-
Scoring rubrics: The evaluator compares the prompt input against the model output, returning a quantitative score (0.0 to 1.0) along with written reasoning explaining why the score was assigned.
-
Pluggable evaluators: Teams can configure semantic similarity evaluators, regex checks, or custom scoring logic depending on domain requirements.
-
Automatic dataset recording: Langfuse automatically buckets interactions by AI service or agent name, creating a continuous historical dataset of real production inputs and outputs.
Tier 2: Session and Conversation Level Evaluation
Individual interactions can score well in isolation while the broader conversation fails from a user perspective. For example, an LLM might politely explain policy limitations on each prompt, receiving high individual relevance scores, while the user becomes increasingly frustrated.
Langfuse groups related interactions into sessions:
-
When a session concludes, a session-level evaluator analyzes the entire multi-turn exchange.
-
In Eric’s insurance claim demo, a sentiment analysis evaluator identified escalating customer frustration and assigned a negative conversation score, despite individual turns scoring above 90%.
-
Evaluators also generate structured explanations, giving engineers clear diagnostic reasoning alongside quantitative metrics.
Tier 3: Drift Detection and CI/CD Quality Gating
Because production evaluations run asynchronously after the fact, they cannot directly prevent bad responses from reaching users. However, the accumulated production datasets can be used to prevent regressions during pre-release testing and CI/CD pipelines.
Eric demonstrated how to convert asynchronous evaluation data into synchronous build gates:
-
LangChain4j evaluation strategy: LangChain4j provides an evaluation framework where custom strategies can query Langfuse datasets by service name.
-
Historical baseline comparison: During a test run, the test harness pulls historical sample sets and baseline scores from Langfuse, sends test prompts to the current application, and evaluates the new outputs.
-
Threshold enforcement: If the average evaluation score falls below a configured threshold (such as 90%), the test triggers an output guardrail failure and halts the build.
-
Validating changes: This mechanism detects regressions caused by prompt template modifications, RAG pipeline updates, or underlying model upgrades (such as migrating from GPT-4 to GPT-5 or Gemini 2.5 to 3.5).
Designing a Modular Java API and Swappable Providers
While exploring Langfuse integration, Eric identified architectural limitations in the upstream auto-generated Java SDK:
-
The official SDK hardcodes dependencies on Jackson 2 (which conflicts with Quarkus 4 and Jackson 3) and OkHttp.
-
The low-level generated API exposes raw protocol details rather than developer-friendly abstractions.
To resolve these issues, Eric created a modular architecture within the Quarkus extension:
-
Generating clean data models from the OpenAPI specification while decoupling the HTTP transport layer to leverage native Quarkus REST capabilities.
-
Building a higher-level Java API focused on core concepts: evaluations, datasets, sessions, and observations.
-
Contributing improvements and decoupling proposals back to the upstream Langfuse project.
Looking forward, Eric and the Quarkus team are exploring a standardized, provider-agnostic AI evaluation layer. Similar to how Quarkus abstracts caching or messaging, a common evaluation API could allow developers to write evaluation logic once and swap between Langfuse, Arize Phoenix, LangSmith, or other providers without rewriting application code.
Looking Ahead: Complex Workflows and Asynchronous Agents
The episode wrapped up with a preview of upcoming work extending evaluation patterns to complex, long-running agent workflows:
-
Asynchronous claims processing with Quarkus Flow: Eric is building an email-driven insurance claims intake system. Instead of blocking on web forms, an AI agent extracts claim details and photos from incoming emails, requests missing information through follow-up emails, and coordinates with human adjusters.
-
Multi-day session evaluation: By mapping an entire business workflow to a single Langfuse session, teams can evaluate complex multi-step processes that span days or weeks of asynchronous interactions.
-
Operational kill switches and guardrails: Exploring hybrid remediation strategies, such as using Langfuse webhooks to automatically trigger feature flags and kill switches when evaluation scores drop in production.
Key Takeaways
-
LLM non-determinism breaks traditional testing: Green operational metrics and HTTP 200 responses do not guarantee that AI responses are accurate, helpful, or safe.
-
Evaluations differ from unit tests: While unit tests assert deterministic exact matches, evaluations measure qualitative attributes, semantic relevance, and domain alignment.
-
The AI testing pyramid: Comprehensive AI validation spans deterministic unit tests at the base, reproducibility and flow verification in the middle, and qualitative judgment (LLM-as-a-judge) at the top.
-
Automatic Dev Services: The Quarkus Langfuse extension automatically provisions PostgreSQL, Redis, ClickHouse, and Langfuse containers in dev mode.
-
Intelligent trace filtering: OpenTelemetry exports full infrastructure traces to backends like Grafana Tempo, while the Langfuse extension forwards only AI-relevant spans.
-
LLM-as-a-judge evaluation: Using an independent model family (such as Gemini evaluating OpenAI completions) provides unbiased scoring rubrics and diagnostic explanations.
-
Session-level observability: Evaluating full multi-turn conversations captures aggregate user sentiment that individual prompt evaluations miss.
-
Automated dataset collection: Production interactions are automatically bucketed by service name in Langfuse, creating reference datasets for offline experimentation and CI testing.
-
CI/CD drift detection: Output guardrails in LangChain4j can query historical Langfuse benchmarks and fail builds if model changes cause quality regressions.
-
Decoupled architecture: Keeping AI observability and evaluation logic inside framework extensions ensures business application code remains clean and vendor-neutral.
Conclusion
Integrating generative AI into enterprise applications requires moving beyond basic health checks and embracing continuous, multi-tiered evaluation. By pairing LangChain4j with Langfuse and OpenTelemetry, Quarkus provides developers with automated local development environments, lean trace filtering, and robust quality gates that catch regressions before code reaches production. Next week, Quarkus Insights returns with Maria Arias de Reyna Dominguez presenting the latest updates on Project Leyden.
