Step 07 - Testing, Evaluation, and Observability
In Step 06, we connected the planner to weather and points-of-interest tools through MCP so it could use their results when recommending a trip. Even with those details, the model still has to choose activities that fit the customer’s request. It might, for example, recommend three days of sightseeing in Rome’s historical center when the family specifically asked to visit coastal towns. Miles of Smiles needs to catch that mismatch before relying on customer feedback.
We need a way to check whether changes to our prompts, skills, or models improve the results, or at least don’t make them worse. Evaluation gives us a repeatable way to make that comparison.
Evaluating with Langfuse
Evaluation gives the planner a known trip request and checks the resulting plan against a description of what a satisfactory answer should contain. For a family asking for three days around Rome with coastal towns, we would expect a complete three-day itinerary with activities that fit the family and include a coastal visit. Repeating this request after a prompt or model change lets us compare the plans against the same expectations.
Langfuse is a platform for recording AI application runs and evaluating their results. We’ll keep our trip requests and expectations in a dataset there, then compare runs as experiments. The Quarkus Langfuse extension connects our application to Langfuse, sends planning traces, and lets the application set up the dataset and evaluator. This gives us the plan, its score, and the calls that produced it in one place.
flowchart LR
sample["Same trip request<br/>and expectations"] --> first["First planning run"]
sample --> second["Run after a change"]
first --> compare["Compare plans<br/>and scores"]
second --> compareJudge models
Some requirements can be checked directly, such as whether a three-day trip has three itinerary days. Others need judgment. A plan might mention the coast without offering a useful coastal visit. A judge model reads the completed plan and uses a rubric, or scoring instructions, to assess it against the request. It returns a score with an explanation that we can check against the plan.
We’ll also add a judge to the vehicle budget guardrail. This judge checks a recommendation while the planner is working and can ask the vehicle agent to try again. The Langfuse judge scores a completed plan during an evaluation run, so its verdict helps us assess the planner without changing that plan.
Testing with scripted responses
Before asking a judge to assess a live plan, we need to know that our checks work and that the workflow passes the right information between agents. Plans with known good and bad contents let us test the structural checks. Scripted model responses then give the workflow predictable replies while the MCP agents call the Trip Intelligence server. This lets us check whether weather and points of interest reach the itinerary planner without model variation obscuring a wiring problem.
flowchart LR
fixtures["Known plans"] --> structure["Do the checks catch defects?"]
scripts["Scripted replies + MCP data"] --> workflow["Does the workflow pass data correctly?"]
live["Live model + saved request"] --> quality["Does the plan meet the request?"]Understanding scores and traces
A score is a starting point for reviewing a plan. The judge’s explanation tells us which requirement it thought the plan missed, and a trace shows the agent calls and MCP tool results from that planning run. If the score is low, we’ll compare the explanation with the plan, then use the trace to find where the recommendation came from. After a change, we can compare two experiments to see how their plans and scores differ.
Prepare the working copy
Stop dev mode in your Step 06 working copy, then copy section-3/step-07/trip-planner/pom.xml to trip-planner/pom.xml in your working copy. Keep any local model-provider dependencies you added earlier. Use the completed Step 07 project for comparison if you get stuck.
Remove the Step 06 tests from your working copy’s trip-planner directory: src/test/java/com/tripplanner/agentic/, src/test/java/com/tripplanner/mcp/, and src/test/frontend/. These tests remain available in Step 06. The Step 07 test configuration selects the evaluation suite.
Then follow the page from the vehicle guardrail changes onwards. The first dev mode start after the POM change takes longer than usual, because Dev Services pulls and starts the Langfuse containers.
Copy section-3/step-07 to a working directory and open that copy. Apply your model-provider settings. Every file on this page is already there, so you can follow along and join the hands-on part at Run the offline tests.
You still need a container runtime for Dev Services, and the live evaluation runs need the Trip Intelligence MCP server from Step 06 running on port 8085. The judge in Langfuse calls OpenAI, so dev mode needs OPENAI_API_KEY set for the quality run. If you use another provider, read the note in Configure the Langfuse evaluator first.
The paths below are relative to your working copy’s trip-planner directory.
The updated POM adds the LangChain4j evaluation module for loading samples and running checks. It also adds OpenTelemetry and the Langfuse extension to record planning runs and send them to Langfuse.
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-opentelemetry</artifactId>
</dependency>
<dependency>
<groupId>io.quarkiverse.langfuse</groupId>
<artifactId>quarkus-langfuse</artifactId>
<version>${quarkus-langfuse.version}</version>
</dependency>
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-junit</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.smallrye.reactive</groupId>
<artifactId>smallrye-reactive-messaging-in-memory</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.quarkiverse.langchain4j</groupId>
<artifactId>quarkus-langchain4j-testing-evaluation-junit5</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.awaitility</groupId>
<artifactId>awaitility</artifactId>
<scope>test</scope>
</dependency>
Maven runs the fixture-based tests with ./mvnw test. The evals profile selects the live integration tests through Failsafe, so we’ll use it when we reach the MCP and model evaluations.
Check vehicle budgets with a judge model
An economy-budget customer should not receive an expensive vehicle recommendation. The existing brand list catches an obvious luxury brand such as Ferrari, but can miss an expensive model from another manufacturer. We’ll add a judge to the guardrail so it can check the vehicle before the workflow continues and ask the vehicle agent to try again when needed. This judge acts during planning. The Langfuse judge we’ll configure later scores a completed plan for an evaluation run; its score does not change the customer’s plan.
The budget judge returns a verdict and a reason, which we’ll represent with a Java record.
Create src/main/java/com/tripplanner/guardrails/BudgetVerdict.java:
package com.tripplanner.guardrails;
/**
* Structured verdict returned by the LLM-based budget guardrail judge.
*
* @param appropriate {@code true} if the vehicle is a reasonable choice for the given budget tier
* @param reason a short explanation of the verdict, used for logging and UI display
*/
public record BudgetVerdict(boolean appropriate, String reason) {}
Create src/main/java/com/tripplanner/guardrails/VehicleBudgetJudge.java:
package com.tripplanner.guardrails;
import dev.langchain4j.service.SystemMessage;
import dev.langchain4j.service.UserMessage;
import io.quarkiverse.langchain4j.RegisterAiService;
import jakarta.enterprise.context.ApplicationScoped;
/**
* LLM-as-judge that evaluates whether a vehicle recommendation is appropriate
* for the given budget tier. Used by {@link TripAppropriatenessGuardrail} in
* place of a hardcoded brand allowlist.
*
* <p>Runs on a fast, cheap model (configured as {@code judgeModel}) to keep
* latency low. Returns a structured {@link BudgetVerdict} with a boolean verdict
* and a short explanation suitable for logging.
*/
@ApplicationScoped
@RegisterAiService(modelName = "judgeModel")
public interface VehicleBudgetJudge {
@SystemMessage("""
You are a vehicle budget compliance judge.
Your only job is to decide whether a vehicle recommendation is appropriate for a given budget tier.
Economy budgets (roughly €500–€1000 total or €30–€50/day) require affordable, mass-market vehicles.
Premium or luxury vehicles — regardless of brand — are not appropriate for economy budgets.
Be strict: if there is any doubt, mark the vehicle as not appropriate.
Respond only with a JSON object matching the BudgetVerdict schema.
""")
@UserMessage("""
Budget tier: {budget}
Vehicle type: {vehicleType}
Vehicle model: {vehicleModel}
Vehicle reasoning: {reasoning}
Is this vehicle appropriate for the stated budget tier?
""")
BudgetVerdict evaluate(String budget, String vehicleType, String vehicleModel, String reasoning);
}
The judge reads the budget tier together with the vehicle type, model, and the recommending agent’s reasoning. Its prompt asks it to reject a recommendation when it is uncertain whether the vehicle fits an economy budget.
@RegisterAiService connects this service to the named judgeModel. The service also needs @ApplicationScoped because the guardrail runs on Quarkus Flow executor threads, where there is no HTTP request context. The default @RequestScoped service would fail there with RequestScoped context was not active.
Update the highlighted lines in src/main/java/com/tripplanner/guardrails/TripAppropriatenessGuardrail.java to inject and use the judge:
package com.tripplanner.guardrails;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.tripplanner.model.TripPlan.VehicleRecommendation;
import dev.langchain4j.data.message.AiMessage;
import dev.langchain4j.guardrail.OutputGuardrail;
import dev.langchain4j.guardrail.OutputGuardrailRequest;
import dev.langchain4j.guardrail.OutputGuardrailResult;
import io.quarkus.logging.Log;
import jakarta.enterprise.context.ApplicationScoped;
import jakarta.inject.Inject;
import java.io.UncheckedIOException;
import java.util.Locale;
import java.util.Set;
@ApplicationScoped
public class TripAppropriatenessGuardrail implements OutputGuardrail {
private static final Set<String> SMALL_VEHICLE_KEYWORDS = Set.of(
"sports car", "sport car", "coupé", "coupe", "convertible", "2-seater", "two-seater", "roadster");
// Fast pre-check for obvious supercars — avoids a judge call for clear-cut cases.
private static final Set<String> OBVIOUS_LUXURY_BRANDS = Set.of(
"ferrari", "porsche", "lamborghini", "maserati", "bentley", "rolls-royce", "aston martin", "mclaren");
// Brands detected by the user preferences ANNOTATE path (unchanged from earlier steps).
private static final Set<String> LUXURY_BRANDS = Set.of(
"ferrari", "porsche", "lamborghini", "maserati", "bentley", "rolls-royce", "aston martin", "mclaren",
"land rover", "range rover", "jaguar", "bmw", "mercedes", "audi", "lexus");
@Inject
ObjectMapper objectMapper;
@Inject
VehicleBudgetJudge budgetJudge;
@Override
public OutputGuardrailResult validate(OutputGuardrailRequest guardrailRequest) {
String text = guardrailRequest.responseFromLLM().aiMessage().text();
if (text == null || text.isBlank()) {
Log.info("🛡️ [TripAppropriatenessGuardrail] SKIP — No text content; tool-call content was not validated");
return success();
}
VehicleRecommendation vehicle;
try {
vehicle = objectMapper.readValue(TripSafetyGuardrail.extractJson(text), VehicleRecommendation.class);
} catch (Exception e) {
Log.info("🛡️ [TripAppropriatenessGuardrail] SKIP — Response is not valid JSON; suitability was not validated");
return success();
}
if (vehicle == null || isBlank(vehicle.type()) || isBlank(vehicle.model())) {
Log.info("🛡️ [TripAppropriatenessGuardrail] SKIP — Vehicle type or model is missing; suitability was not validated");
return success();
}
var variables = guardrailRequest.requestParams().variables();
String budget = (String) variables.get("budget");
String tripType = (String) variables.get("tripType");
String preferences = variables.get("preferences") instanceof String s ? s : "";
int travelers;
try {
// Prompt variables may contain text or numeric method arguments.
travelers = Integer.parseInt(String.valueOf(variables.get("travelers")));
} catch (NumberFormatException e) {
Log.info("🛡️ [TripAppropriatenessGuardrail] SKIP — Traveler count is missing or invalid; suitability was not validated");
return success();
}
if (budget == null || budget.isBlank() || tripType == null || tripType.isBlank()) {
Log.info("🛡️ [TripAppropriatenessGuardrail] SKIP — Trip variables are missing or incomplete; suitability was not validated");
return success();
}
String vehicleType = vehicle.type().toLowerCase(Locale.ROOT);
String vehicleModel = vehicle.model().toLowerCase(Locale.ROOT);
String reasoning = vehicle.reasoning() == null ? "" : vehicle.reasoning();
// For economy budgets: fast-path for obvious supercars, then LLM judge for everything else.
if (budget.toLowerCase(Locale.ROOT).contains("economy")) {
boolean inappropriate;
if (isObviousLuxuryBrand(vehicleModel)) {
inappropriate = true;
} else {
try {
BudgetVerdict verdict = budgetJudge.evaluate(budget, vehicleType, vehicleModel, reasoning);
inappropriate = !verdict.appropriate();
Log.infof("🛡️ [TripAppropriatenessGuardrail] %s — Budget judge verdict for '%s': %s",
inappropriate ? "JUDGE-FAIL" : "JUDGE-PASS", vehicleModel, verdict.reason());
} catch (Exception e) {
Log.infof("🛡️ [TripAppropriatenessGuardrail] JUDGE-ERROR — Budget judge threw an exception for '%s' — treating as appropriate: %s", vehicleModel, e.getMessage());
inappropriate = false;
}
}
if (inappropriate) {
Log.infof("🛡️ [TripAppropriatenessGuardrail] REPROMPT — Vehicle '%s' does not match economy budget — asking model to retry with an affordable option", vehicleModel);
return reprompt("The vehicle recommendation does not fit the economy budget. "
+ "Please recommend an affordable, budget-friendly vehicle instead.",
"The previous vehicle recommendation does not fit the economy budget. "
+ "Respond ONLY with a valid JSON vehicle recommendation object (with fields: type, model, reasoning) "
+ "for an affordable, budget-friendly, non-luxury vehicle suitable for this trip. Do not explain or acknowledge — just output the JSON.");
}
}
if (travelers >= 4 && isSmallVehicle(vehicleType)) {
String reason = "Original recommendation '" + vehicleType + "' is too small for " + travelers + " travelers";
Log.infof("🛡️ [TripAppropriatenessGuardrail] REWRITE — %s; returned a generic category recommendation", reason);
return successWith(AiMessage.from(toJson(genericVehicle(travelers, tripType, reason))));
}
// If the user asked for a luxury brand but the output doesn't contain it, the model
// already overrode the request on its own. Annotate the result so the UI can show why.
String requestedBrand = requestedLuxuryBrand(preferences);
if (requestedBrand != null && !vehicleModel.contains(requestedBrand)) {
String reason = "Requested brand '" + requestedBrand + "' is not suitable for this trip; a more appropriate vehicle was selected";
var annotated = new VehicleRecommendation(vehicle.type(), vehicle.model(), vehicle.reasoning(), reason);
Log.infof("🛡️ [TripAppropriatenessGuardrail] ANNOTATE — Preferences mentioned '%s' but output is '%s' — %s", requestedBrand, vehicleModel, reason);
return successWith(AiMessage.from(toJson(annotated)));
}
Log.info("🛡️ [TripAppropriatenessGuardrail] PASS — Budget judge approved the vehicle recommendation");
return success();
}
private boolean isSmallVehicle(String vehicleType) {
return SMALL_VEHICLE_KEYWORDS.stream().anyMatch(vehicleType::contains);
}
private boolean isObviousLuxuryBrand(String vehicleModel) {
return OBVIOUS_LUXURY_BRANDS.stream().anyMatch(vehicleModel::contains);
}
private String requestedLuxuryBrand(String preferences) {
if (preferences == null || preferences.isBlank()) return null;
String lower = preferences.toLowerCase(Locale.ROOT);
return LUXURY_BRANDS.stream().filter(lower::contains).findFirst().orElse(null);
}
private VehicleRecommendation genericVehicle(int travelers, String tripType, String reason) {
String replacement = switch (tripType.toLowerCase(Locale.ROOT)) {
case "adventure" -> "SUV";
case "business" -> "Estate";
default -> "MPV";
};
return new VehicleRecommendation(
replacement,
(replacement.equals("MPV") ? "Family MPV" : replacement) + "; specific model subject to availability.",
"Vehicle corrected by guardrail: original recommendation was too small for "
+ travelers + " travelers. Suggested category: " + replacement
+ ". Confirm seating, luggage capacity, price, and availability with the rental provider.",
reason);
}
private String toJson(VehicleRecommendation vehicle) {
try {
return objectMapper.writeValueAsString(vehicle);
} catch (JsonProcessingException e) {
throw new UncheckedIOException(e);
}
}
private static boolean isBlank(String value) {
return value == null || value.isBlank();
}
}
For economy budgets, the guardrail still rejects brands in OBVIOUS_LUXURY_BRANDS without calling the judge. Other recommendations go to the judge, whose verdict determines whether the guardrail asks the vehicle agent for an affordable alternative.
Add the judgeModel configuration to src/main/resources/application.properties:
# Model used by the vehicle budget judge
quarkus.langchain4j.judgeModel.chat-model.provider=openai
quarkus.langchain4j.openai.judgeModel.api-key=${OPENAI_API_KEY}
quarkus.langchain4j.openai.judgeModel.chat-model.model-name=gpt-4o-mini
quarkus.langchain4j.openai.judgeModel.chat-model.temperature=0
quarkus.langchain4j.openai.judgeModel.timeout=30
The budget judge uses its own model configuration, so you can change it independently of the agents that generate the plan.
Define acceptable plans for sample requests
The Rome example gives us a repeatable question: does a plan for a three-day family trip include a coastal town? A sample saves the planner inputs alongside a description of an acceptable result. We can then change the planner and compare each new plan against the same requirements.
Create src/main/resources/evaluation/samples.yaml:
# Evaluation samples for the trip planner (step 07).
#
# The parameters are the planner inputs, in planTrip order: destination,
# start date, days, travel style, travelers, budget, interests.
# The expected-output describes what an acceptable plan for that request must
# contain. Dev mode seeds these samples into the Langfuse dataset
# trip-plan-samples, where the judge compares a real plan against the
# expected-output using rubric.txt. The invariant strategy only uses the
# parameters.
- name: "rome-family-three-days"
parameters:
- "Rome"
- "2027-07-10"
- "3"
- "family"
- "2"
- "moderate"
- "coastal towns"
expected-output: >-
A three-day family road trip starting in Rome in July for two travelers.
The vehicle suits two people with luggage and is not a luxury or sports
car. The itinerary has days 1 to 3, each with family-friendly activities,
and includes at least one coastal town. The cost total is a euro amount
that is reasonable for a moderate budget.
tags:
- "smoke"
- "mcp"
- name: "barcelona-family-three-days"
parameters:
- "Barcelona"
- "2027-07-12"
- "3"
- "family"
- "2"
- "moderate"
- "beaches"
expected-output: >-
A three-day family road trip starting in Barcelona in July for two
travelers. The vehicle suits two people with luggage and is not a luxury
or sports car. The itinerary has days 1 to 3, each with family-friendly
activities, and includes time at the beach. The cost total is a euro
amount that is reasonable for a moderate budget.
tags:
- "smoke"
- "mcp"
- name: "rome-family-seven-days"
parameters:
- "Rome"
- "2027-07-10"
- "7"
- "family"
- "2"
- "moderate"
- "historic sites"
expected-output: >-
A seven-day family road trip starting in Rome in July for two travelers.
The vehicle suits two people with luggage and is not a luxury or sports
car. The itinerary has days 1 to 7, includes historic sites, and leaves
room for rest. The cost total is a euro amount that is reasonable for a
moderate budget.
tags:
- "smoke"
In rome-family-three-days, the parameters are the trip request and expected-output describes what a satisfactory plan contains, including family-friendly activities and a coastal town. Several itineraries could meet those requirements, so the sample does not prescribe exact wording or stops.
The Java checks use the request parameters to check details such as the number of days. The Langfuse judge compares the generated plan with expected-output.
The samples belong in the main resources because dev mode reads them at startup and copies them into the trip-plan-samples dataset in Langfuse. Edit the YAML file to keep changes in your project, then restart dev mode to update the dataset.
Write the evaluation rubric
Java can check whether the itinerary has three days, but assessing whether its activities suit a family requires judgment. We’ll give the Langfuse evaluator a rubric that tells it how to compare the plan with the sample’s requirements.
Create src/main/resources/evaluation/rubric.txt:
You are evaluating a trip plan produced by a trip planning assistant.
The trip request and the plan are given as JSON. The expected output
describes what an acceptable plan for the same trip request must contain.
Check the plan against each requirement in the expected output. Wording,
order, and extra detail in the plan do not matter.
Trip request: {{input}}
Plan to evaluate: {{output}}
Expected output: {{ground_truth}}
For the Rome sample, Langfuse replaces {{ input }} with the saved trip request, {{ output }} with the generated plan JSON, and {{ ground_truth }} with the requirements in expected-output. The rubric asks the judge to check each requirement without penalizing different wording, order, or extra detail. The result is a score with an explanation of any missed requirements.
Configure the Langfuse evaluator
Starting dev mode will copy the YAML samples into a Langfuse dataset and register the rubric as an evaluator. When the quality test runs a sample, Langfuse receives the completed plan through its trace, applies the rubric, and records a score. LangfuseEvaluationSetup prepares the dataset, model connection, and evaluator through the extension’s LangfuseOperations client.
Create src/main/java/com/tripplanner/evaluation/LangfuseEvaluationSetup.java:
package com.tripplanner.evaluation;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
import com.langfuse.api.LangfuseApiException;
import com.langfuse.api.model.BooleanEvaluationRuleFilter1;
import com.langfuse.api.model.CreateDatasetItemRequest;
import com.langfuse.api.model.CreateDatasetRequest;
import com.langfuse.api.model.CreateEvaluationRuleRequest;
import com.langfuse.api.model.CreateLlmAsJudgeEvaluatorRequest1;
import com.langfuse.api.model.CreateScoreConfigRequest;
import com.langfuse.api.model.EvaluationRuleBooleanFilterOperator;
import com.langfuse.api.model.EvaluationRuleEvaluatorAssignmentInput;
import com.langfuse.api.model.EvaluationRuleFilter;
import com.langfuse.api.model.EvaluationRuleOptionsFilterOperator;
import com.langfuse.api.model.EvaluatorChatPromptInput;
import com.langfuse.api.model.EvaluatorModelConfig;
import com.langfuse.api.model.EvaluatorOutputDefinition;
import com.langfuse.api.model.LlmAdapter;
import com.langfuse.api.model.PromptVariableMappingInput;
import com.langfuse.api.model.PromptVariableMappingSource;
import com.langfuse.api.model.PublicEvaluatorNumericScore1;
import com.langfuse.api.model.ScoreConfigDataType;
import com.langfuse.api.model.StringOptionsEvaluationRuleFilter1;
import com.langfuse.api.model.UpsertLlmConnectionRequest;
import io.quarkiverse.langfuse.api.LangfuseOperations;
import io.quarkus.logging.Log;
import io.quarkus.runtime.StartupEvent;
import jakarta.enterprise.event.Observes;
import jakarta.inject.Inject;
import jakarta.inject.Singleton;
import org.eclipse.microprofile.config.inject.ConfigProperty;
import java.io.IOException;
import java.io.InputStream;
import java.io.UncheckedIOException;
import java.nio.charset.StandardCharsets;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.function.Supplier;
/**
* Sets up the trip plan judge inside Langfuse when dev mode starts. Every step
* looks up or upserts by name, so a restart leaves one of each. A failing step
* logs a warning and never stops the application from starting.
*/
@Singleton
public class LangfuseEvaluationSetup {
static final String DATASET = "trip-plan-samples";
// Langfuse names each score after the evaluator that produced it.
static final String SCORE = "plan-quality";
static final String PROVIDER = "openai";
@ConfigProperty(name = "trip.evaluation.langfuse-setup.enabled", defaultValue = "false")
boolean enabled;
@ConfigProperty(name = "quarkus.langchain4j.openai.api-key")
String openAiKey;
@ConfigProperty(name = "quarkus.langchain4j.openai.judgeModel.chat-model.model-name")
String judgeModel;
@Inject
LangfuseOperations langfuse;
void onStart(@Observes StartupEvent event) {
if (!enabled) {
return;
}
step("LLM connection", this::createLlmConnection);
step("score config", this::createScoreConfig);
String datasetId = step("dataset", this::createDataset);
String evaluatorId = step("evaluator", this::createEvaluator);
if (datasetId != null && evaluatorId != null) {
step("evaluation rule", () -> createEvaluationRule(evaluatorId, datasetId));
}
}
String createLlmConnection() {
return langfuse.llmConnections().upsert(UpsertLlmConnectionRequest.builder()
.provider(PROVIDER)
.adapter(LlmAdapter.OPENAI)
.secretKey(openAiKey)
.customModels(List.of(judgeModel))
.build()).getId();
}
String createScoreConfig() {
return langfuse.scoreConfigs().createIfAbsent(CreateScoreConfigRequest.builder()
.name(SCORE)
.dataType(ScoreConfigDataType.NUMERIC)
.minValue(0.0)
.maxValue(1.0)
.description("How well a trip plan meets the expected output of its sample. 1 = every requirement met.")
.build()).getId();
}
String createDataset() {
String datasetId = langfuse.datasets().createIfAbsent(CreateDatasetRequest.builder()
.name(DATASET)
.description("Trip requests from evaluation/samples.yaml with the plan each one should produce")
.build()).getId();
// The sample name is the item id, so a restart updates the items in place.
for (Sample sample : readSamples()) {
langfuse.datasetItems().create(CreateDatasetItemRequest.builder()
.datasetName(DATASET)
.id(sample.name())
.input(sample.input())
.expectedOutput(sample.expectedOutput())
.build());
}
return datasetId;
}
String createEvaluator() {
return langfuse.evaluators().createIfAbsent(CreateLlmAsJudgeEvaluatorRequest1.builder()
.type(CreateLlmAsJudgeEvaluatorRequest1.TypeEnum.LLM_AS_JUDGE)
.name(SCORE)
.description("Trip plan quality: does the plan meet the expected output of its sample?")
.prompt(new EvaluatorChatPromptInput(readResource("evaluation/rubric.txt")))
.modelConfig(EvaluatorModelConfig.builder().provider(PROVIDER).model(judgeModel).build())
.variableMapping(variableMapping())
.outputDefinition(new EvaluatorOutputDefinition(PublicEvaluatorNumericScore1.builder()
.dataType(PublicEvaluatorNumericScore1.DataTypeEnum.NUMERIC)
.minValue(0.0)
.maxValue(1.0)
.scoreValueInstructions("A number from 0 to 1, where 1 means the plan meets every requirement in the expected output.")
.scoreReasoningInstructions("One sentence that names any requirement the plan misses.")
.build()))
.build()).getLlmAsJudgeEvaluator1().getId();
}
// Judge the root span of every experiment item that runs against the samples dataset.
String createEvaluationRule(String evaluatorId, String datasetId) {
return langfuse.evaluationRules().createIfAbsent(CreateEvaluationRuleRequest.builder()
.name(DATASET + " experiments")
.enabled(true)
.sampling(1.0)
.filter(List.of(
new EvaluationRuleFilter(BooleanEvaluationRuleFilter1.builder()
.type(BooleanEvaluationRuleFilter1.TypeEnum.BOOLEAN)
.column("isExperimentItemRootSpan")
.operator(EvaluationRuleBooleanFilterOperator.EQUAL)
.value(true)
.build()),
new EvaluationRuleFilter(StringOptionsEvaluationRuleFilter1.builder()
.type(StringOptionsEvaluationRuleFilter1.TypeEnum.STRING_OPTIONS)
.column("datasetId")
.operator(EvaluationRuleOptionsFilterOperator.ANY_OF)
.value(List.of(datasetId))
.build())))
.evaluatorAssignments(List.of(EvaluationRuleEvaluatorAssignmentInput.builder()
.evaluatorId(evaluatorId)
.variableMapping(variableMapping())
.build()))
.build()).getId();
}
// {{input}} and {{output}} come from the span, {{ground_truth}} from the dataset item.
static List<PromptVariableMappingInput> variableMapping() {
return List.of(
mapping("input", PromptVariableMappingSource.INPUT),
mapping("output", PromptVariableMappingSource.OUTPUT),
mapping("ground_truth", PromptVariableMappingSource.EXPECTED_OUTPUT));
}
static PromptVariableMappingInput mapping(String variable, PromptVariableMappingSource source) {
return PromptVariableMappingInput.builder().variable(variable).source(source).build();
}
static List<Sample> readSamples() {
try (InputStream yaml = resource("evaluation/samples.yaml")) {
return new ObjectMapper(new YAMLFactory()).readValue(yaml, new TypeReference<List<Sample>>() {
});
} catch (IOException e) {
throw new UncheckedIOException(e);
}
}
static String readResource(String path) {
try (InputStream in = resource(path)) {
return new String(in.readAllBytes(), StandardCharsets.UTF_8);
} catch (IOException e) {
throw new UncheckedIOException(e);
}
}
static InputStream resource(String path) throws IOException {
InputStream in = Thread.currentThread().getContextClassLoader().getResourceAsStream(path);
if (in == null) {
throw new IOException(path + " not found on the classpath");
}
return in;
}
static <T> T step(String name, Supplier<T> action) {
try {
T id = action.get();
Log.infof("Langfuse evaluation setup: %s ready (%s)", name, id);
return id;
} catch (LangfuseApiException e) {
Log.warnf("Langfuse evaluation setup: %s failed (HTTP %d): %s", name, e.getStatusCode(), e.getServerMessage());
} catch (RuntimeException e) {
Log.warnf("Langfuse evaluation setup: %s failed: %s", name, e.getMessage());
}
return null;
}
/** One entry in samples.yaml. The parameters follow the planTrip argument order. */
record Sample(String name, List<String> parameters,
@JsonProperty("expected-output") String expectedOutput, List<String> tags) {
Map<String, String> input() {
String[] keys = {"destination", "startDate", "days", "tripType", "travelers", "budget", "preferences"};
Map<String, String> input = new LinkedHashMap<>();
for (int i = 0; i < keys.length; i++) {
input.put(keys[i], parameters.get(i));
}
return input;
}
}
}
At startup, onStart() checks whether evaluation setup is enabled. When it is, the class creates the model connection and the trip-plan-samples dataset, then registers the rubric as an evaluator named plan-quality. Its evaluation rule selects experiment items from that dataset and supplies the request, generated plan, and expected output to the rubric.
The evaluator returns a score between 0 and 1, where 1 means the plan meets every requirement, with a sentence explaining any missed requirements. The test will retrieve this score by its plan-quality name, so the evaluator and test must use the same name.
The setup reuses existing objects and updates dataset items by sample name, so restarting it against the same Langfuse instance does not add duplicate samples. If a setup call fails, the application logs a warning and continues starting. The quality test needs both the dataset and evaluator, so we’ll check those startup messages before running it.
Add the setup flag to src/main/resources/application.properties to enable this initialization in dev mode:
# The trip plan judge runs inside Langfuse on OpenAI with the judgeModel above; dev mode sets it up at startup.
trip.evaluation.langfuse-setup.enabled=false
%dev.trip.evaluation.langfuse-setup.enabled=true
Using a provider other than OpenAI
The judge runs in the Langfuse worker container and calls the model through the LLM connection, so it needs an OpenAI API key even when your planner uses another provider. If your provider has an OpenAI-compatible endpoint, you can point the connection at it instead, either by editing the LLM connection in the Langfuse UI or by setting a base URL in createLlmConnection(). For Ollama, the base URL is http://host.docker.internal:11434/v1, because localhost inside the container is the container itself. If neither works for you, skip the quality run. The offline suite and the composition run work with any provider.
Check plan structure
Before judging the recommendations, we need to know whether a plan is complete enough to evaluate. A missing itinerary day or invalid cost is a definite error, so Java can catch it without asking a model. We’ll first test those checks against plans with known contents. Once they work, we’ll test the workflow and then the quality of a live plan.
The supplied evaluation code loads the samples, checks plans, runs the planner, and retrieves scores. Copy it into the working project before running the tests.
For the hands-on route, copy these paths from section-3/step-07/trip-planner to the same paths in your working copy:
src/test/java/com/tripplanner/evaluation/(the invariant strategy, the dataset loader, the experiment runner, and all the tests)src/test/resources/evaluation/known-bad.yaml(plans with deliberate defects)src/test/resources/application.properties(test profiles for the offline, composition, and quality runs)src/main/java/com/tripplanner/agentic/TracingExecutorSetup.java(keeps one planning run in one trace)
TripPlanText converts plans to JSON for evaluation and parses that JSON when the structural checks need to inspect a field.
A plan needs a complete itinerary and a valid cost before we assess its recommendations. TripPlanInvariantStrategy checks those fields in Java and implements EvaluationStrategy<String> so the evaluation module can run it against each sample.
public EvaluationResult evaluate(EvaluationSample<String> sample, TripPlan plan) {
if (plan == null) return EvaluationResult.failed(0, "planner returned no plan");
int requestedDays = Integer.parseInt(sample.parameters().get(2).toString());
if (requestedDays < 1 || requestedDays > 30) {
throw new IllegalArgumentException("Sample duration must be between 1 and 30 days");
}
List<String> errors = new ArrayList<>();
if (plan.vehicle() == null) {
errors.add("missing vehicle");
} else {
requireText(plan.vehicle().type(), "vehicle type", errors);
requireText(plan.vehicle().model(), "vehicle model", errors);
requireText(plan.vehicle().reasoning(), "vehicle reasoning", errors);
}
requireText(plan.routeOverview(), "route", errors);
if (plan.itinerary() == null || plan.itinerary().size() != requestedDays) {
errors.add("itinerary must contain " + requestedDays + " days");
}
var numbers = new HashSet<Integer>();
if (plan.itinerary() != null) {
for (var day : plan.itinerary()) {
if (day == null) {
errors.add("missing itinerary entry");
continue;
}
if (!numbers.add(day.day())) errors.add("duplicate day " + day.day());
if (day.day() < 1 || day.day() > requestedDays) errors.add("invalid day " + day.day());
requireText(day.title(), "day title", errors);
requireText(day.description(), "day description", errors);
requireText(day.overnightStop(), "overnight stop", errors);
}
}
for (int day = 1; day <= requestedDays; day++) {
if (!numbers.contains(day)) errors.add("itinerary missing day " + day);
}
if (plan.costs() == null) {
errors.add("missing costs");
} else {
String total = plan.costs().total();
if (total == null || !total.matches("\\s*(?:€|EUR)?\\s*\\d+(?:[.,]\\d+)*(?:\\s*(?:€|EUR))?\\s*")) {
errors.add("cost total must be a non-negative EUR amount");
}
}
return errors.isEmpty() ? EvaluationResult.passed(1)
: EvaluationResult.failed(0, String.join("; ", errors));
}
The requested duration comes from the sample’s third parameter. The strategy collects every problem it finds instead of stopping at the first one:
- The vehicle must have a type, model, and reasoning, and the plan needs a route overview.
- The itinerary must have exactly the requested number of days, each with a title, description, and overnight stop.
- Day numbers must be unique, fall between 1 and the requested duration, and leave no gaps.
- The total cost must be a plain, non-negative euro amount.
A plan with no problems scores 1, and anything else scores 0 with the list of problems as the reason.
To check that the strategy detects defects, known-bad.yaml contains plans with a missing vehicle, a gap in the itinerary, a negative or non-numeric cost, and an empty output. TripPlanInvariantStrategyTest checks that each fails for the expected reason and that a complete plan passes.
Run the offline tests
TripPlanEvaluationHarnessTest checks that the harness loads the samples and applies the requested duration to each fixture plan. It also checks that a one-day plan fails a request for a longer trip and that the results can be saved as a JSON report. Maven runs this test alongside the invariant strategy tests by default.
Run the default test suite from your trip-planner directory:
These tests tell us whether the structural checks recognize known good and bad plans. If a deliberately broken plan passes, or a valid fixture fails, investigate the checks before relying on their results. They do not tell us whether the agents exchange the right information during a real planning run, which is what we’ll test next.
Test the workflow with scripted responses
Now we can test whether the agents pass the right data through the workflow. The composition test uses scripted model responses while the two @McpClientAgent subagents call the running Trip Intelligence server. With model responses fixed, a failed assertion points to the workflow or its MCP data rather than a change in the model’s wording.
TripPlannerCompositionLiveIT has two tests:
- The first plans a Rome trip, checks that the weather and points of interest from the MCP server reached the itinerary planner’s prompt, and runs the invariant strategy on the saved output.
- The second plans two trips in a row and checks that each plan keeps its own request’s destination.
The test’s mcp profile disables Langfuse because these assertions use the plan and the captured prompts. Before each test, it probes the MCP server on port 8085 and skips if the server isn’t running the sunny fixture.
With the MCP server running, run the composition IT from your trip-planner directory:
Evaluating recommendation quality
The composition test confirms that the workflow connects its parts, but scripted replies cannot tell us whether the configured model recommends suitable activities. The quality test generates a new plan for the Rome sample, checks its structure, and asks Langfuse to assess its recommendations against the saved requirements.
Before planning, the test checks that the MCP server is running the sunny fixture and that the trip-plan-samples dataset exists in Langfuse. If the dataset is missing, the test fails and tells you to start dev mode first, since dev mode is what creates it. The test then loads the rome-family-three-days item from the dataset and passes it to a small helper bean that runs the planner:
assertTrue(TripPlannerCompositionLiveIT.probeSunnyServer(),
"Trip Intelligence MCP server on :8085 is not running the sunny fixture");
var loader = new LangfuseDatasetSampleLoader(langfuse);
assertTrue(loader.datasetId(DATASET).isPresent(),
"dataset " + DATASET + " not found in Langfuse: start dev mode first");
var item = loader.load(SAMPLE);
var result = runner.run("step-07-" + Instant.now(), item);
The runner records the planning calls in a trace and returns the plan together with its trace id. The test checks the plan’s structure before waiting for the judge’s score:
// Langfuse scores the experiment item asynchronously.
NumericScoreV31 score = Awaitility.await()
.atMost(Duration.ofSeconds(90))
.pollInterval(Duration.ofSeconds(3))
.until(() -> planQualityScore(result.traceId()), Optional::isPresent)
.orElseThrow();
assertTrue(score.getValue() >= THRESHOLD,
"plan-quality " + score.getValue() + " is below " + THRESHOLD + ": " + score.getComment());
assertFalse(score.getComment() == null || score.getComment().isBlank(),
"the judge must explain its score");
Langfuse evaluates the plan asynchronously in its worker container, so the test polls for the trace’s plan-quality score for up to 90 seconds. It passes when the structural checks succeed, the score is at least 0.7, and the judge has supplied a reason.
The test’s evals profile connects to the Langfuse instance started by dev mode, where the dataset and evaluator have been created. It uses the Dev Services credentials and disables the generic OTLP exporter because the Langfuse extension exports the traces itself.
%evals.quarkus.langfuse.devservices.enabled=false
%evals.quarkus.langfuse.public-key=quarkus
%evals.quarkus.langfuse.secret-key=quarkuslangfuse
%evals.quarkus.otel.exporter.otlp.enabled=false
Keep dev mode running while you evaluate plans and inspect their traces. The test is skipped unless you pass its Langfuse URL through quarkus.langfuse.base-url.
Start dev mode in your working project with ./mvnw quarkus:dev (.\mvnw.cmd quarkus:dev on Windows), with OPENAI_API_KEY set, and keep the MCP server running. Open the Dev UI at http://localhost:8080/q/dev, go to Dev Services, and copy the quarkus.langfuse.base-url value from the Langfuse entry.
The same page lists the Langfuse login, [email protected] with the password quarkuslangfuse. The Extensions page also has a Quarkus Langfuse card with a direct link to the Langfuse UI.
Check the dev mode log for the Langfuse evaluation setup messages before running the test. Each completed setup operation logs a ready message. If the log reports a warning, resolve the reported setup failure before continuing.
In a second terminal, run the quality IT from your trip-planner directory, using the URL you copied:
Complete quality IT
package com.tripplanner.evaluation;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertTrue;
import com.langfuse.api.model.NumericScoreV31;
import com.langfuse.api.model.ScoreV3;
import io.quarkiverse.langfuse.api.LangfuseOperations;
import io.quarkiverse.langfuse.api.ScoreFieldGroup;
import io.quarkiverse.langfuse.api.ScoreFilter;
import io.quarkus.test.junit.QuarkusTest;
import io.quarkus.test.junit.QuarkusTestProfile;
import io.quarkus.test.junit.TestProfile;
import jakarta.inject.Inject;
import org.awaitility.Awaitility;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.condition.EnabledIfSystemProperty;
import java.time.Duration;
import java.time.Instant;
import java.util.Optional;
/**
* Runs one sample from the Langfuse dataset as an experiment item and waits
* for the Langfuse evaluator to score it.
*
* <p>Start dev mode first: it seeds the {@code trip-plan-samples} dataset and
* creates the evaluator. Then run {@code ./mvnw verify -Pevals
* -Dquarkus.langfuse.base-url=<dev mode Langfuse URL>} with a real LLM key and
* the MCP server running on :8085.
*/
@QuarkusTest
@TestProfile(TripPlanQualityEvaluationLiveIT.LiveProfile.class)
@EnabledIfSystemProperty(named = "quarkus.langfuse.base-url", matches = ".+")
class TripPlanQualityEvaluationLiveIT {
static final String DATASET = "trip-plan-samples";
static final String SAMPLE = "rome-family-three-days";
static final double THRESHOLD = 0.7;
@Inject
LangfuseOperations langfuse;
@Inject
TripPlanExperimentRunner runner;
@Test
void langfuseScoresTheExperimentItem() {
assertTrue(TripPlannerCompositionLiveIT.probeSunnyServer(),
"Trip Intelligence MCP server on :8085 is not running the sunny fixture");
var loader = new LangfuseDatasetSampleLoader(langfuse);
assertTrue(loader.datasetId(DATASET).isPresent(),
"dataset " + DATASET + " not found in Langfuse: start dev mode first");
var item = loader.load(SAMPLE);
var result = runner.run("step-07-" + Instant.now(), item);
assertFalse(result.traceId().matches("0+"), "the planning run must have a valid trace id");
// Deterministic gate, checked locally before waiting for the judge.
var invariant = new TripPlanInvariantStrategy().evaluate(item.sample(), result.output());
assertTrue(invariant.passed(), "plan must pass the invariants: " + invariant.explanation());
// Langfuse scores the experiment item asynchronously.
NumericScoreV31 score = Awaitility.await()
.atMost(Duration.ofSeconds(90))
.pollInterval(Duration.ofSeconds(3))
.until(() -> planQualityScore(result.traceId()), Optional::isPresent)
.orElseThrow();
assertTrue(score.getValue() >= THRESHOLD,
"plan-quality " + score.getValue() + " is below " + THRESHOLD + ": " + score.getComment());
assertFalse(score.getComment() == null || score.getComment().isBlank(),
"the judge must explain its score");
}
Optional<NumericScoreV31> planQualityScore(String traceId) {
var filter = ScoreFilter.builder()
.traceId(traceId)
.name("plan-quality")
.fields(ScoreFieldGroup.DETAILS)
.build();
return langfuse.scores().matching(filter).findAll().stream()
.map(ScoreV3::getActualInstance)
.filter(NumericScoreV31.class::isInstance)
.map(NumericScoreV31.class::cast)
.findFirst();
}
public static class LiveProfile implements QuarkusTestProfile {
@Override
public String getConfigProfile() {
return "evals";
}
}
}
Inspect the score and the planning trace
A passing test tells us that the plan met the structural checks and scored at least 0.7. To understand which requirements affected the score, we need to read the judge’s explanation alongside the plan.
Open Langfuse from the Quarkus Langfuse card, log in, and go to Tracing. Open the trip-plan-evaluation trace, select its root span in the tree, and select the Scores tab.
Read the plan-quality score and its explanation first. Compare any missed requirement with the saved sample and the generated plan before deciding whether the planner needs a change. A low score can also reflect a mistaken judgment, so the explanation matters as much as the number.
The trace shows the calls that produced the plan. The MCP weather and points-of-interest calls feed the vehicle advisor and itinerary planner; the evaluators and any vehicle revision follow, then the cost estimator calls its pricing tool. The header shows latency, cost, and token counts for the run. The score is on the root span, where the experiment records the completed plan.
Open the score’s comment and find any requirement the judge says was missed. Check the plan against that requirement. If the plan is wrong, inspect the relevant agent’s prompt and response. For a recommendation that depends on weather or points of interest, check the MCP tool result as well. If the explanation misreads the plan, review the rubric before treating the score as evidence of a regression.
Keep parallel calls in one trace
The vehicle and itinerary research run in parallel. By default, the executor used for those branches does not carry the current OpenTelemetry context to the new threads, so one planning run would appear as several unrelated traces. The TracingExecutorSetup copied earlier wraps that executor and carries the context into each branch.
@ApplicationScoped
public class TracingExecutorSetup {
void onStart(@Observes StartupEvent event, ManagedExecutor managedExecutor) {
Executor tracing = Context.taskWrapping(managedExecutor);
ExecutorProvider.set(() -> tracing);
}
void onStop(@Observes ShutdownEvent event) {
ExecutorProvider.set(null);
}
}
At startup, Context.taskWrapping() wraps the Quarkus managed executor that LangChain4j uses for parallel branches. Their agent calls then appear under the same planning trace you just inspected.
How the trace becomes a dataset experiment
The experiment name is step-07- followed by the current time, so each run of the IT shows up as a separate run of the dataset in Langfuse. The helper bean runs the planner inside a root span:
@WithSpan("trip-plan-evaluation")
public Result run(String experimentName, DatasetSample item) {
Span span = Span.current();
span.setAttribute("langfuse.experiment.id", experimentName);
span.setAttribute("langfuse.experiment.name", experimentName);
span.setAttribute("langfuse.experiment.dataset.id", item.datasetId());
span.setAttribute("langfuse.experiment.item.id", item.itemId());
span.setAttribute("langfuse.experiment.item.root_observation_id", span.getSpanContext().getSpanId());
span.setAttribute("langfuse.experiment.item.expected_output", item.sample().expectedOutput());
span.setAttribute("langfuse.observation.input", item.inputJson());
span.setAttribute(AttributeKey.stringArrayKey("langfuse.trace.tags"), List.of("evaluation"));
span.setAttribute("langfuse.trace.metadata.sample_id", item.itemId());
span.setAttribute("langfuse.trace.metadata.model", modelName);
Parameters p = item.sample().parameters();
TripPlan plan = tripPlannerSystem.planTrip(p.get(0), p.get(1), p.get(2), p.get(3),
p.get(4), p.get(5), p.get(6));
String output = TripPlanText.render(plan);
span.setAttribute("langfuse.observation.output", output);
return new Result(plan, output, span.getSpanContext().getTraceId());
}
@WithSpan("trip-plan-evaluation") opens a span for the whole planning run, so every agent, guardrail, and MCP call is recorded beneath it. The attributes on that span turn the trace into an experiment item:
langfuse.experiment.dataset.idandlangfuse.experiment.item.idtie the trace to the dataset item it came from.langfuse.experiment.idandlangfuse.experiment.nameboth get the experiment name, which groups the trace with the other items of the same run.langfuse.experiment.item.root_observation_idis the span’s own id, which tells Langfuse where the result of the run is.langfuse.experiment.item.expected_outputcarries the sample’s expected output, which the judge reads as{{ ground_truth }}.langfuse.observation.inputandlangfuse.observation.outputhold the trip request and the rendered plan, which become{{ input }}and{{ output }}.langfuse.trace.tagstags the trace withevaluation, and the twolangfuse.trace.metadata.*attributes record the sample id and the planner’s model name, so you can filter traces by them when you run the IT several times.
Compare plans after a change
To compare results, we need a second evaluation of the same request. Change a planner prompt or skill, then rerun the quality test with dev mode still running. Keep the sample requirements and rubric unchanged so both plans are assessed against the same criteria.
Go to Datasets, open trip-plan-samples, and select the Experiments tab. Select two runs and click Compare.
Each quality test adds an experiment for the rome-family-three-days item. The comparison shows both generated plans and their scores beside the request and expected output. Open the speech-bubble icon next to each score to read the judge’s reasoning. Compare the explanations with the plans to see whether the change addressed the requirement you were interested in. A single pair of runs is only a starting point because both generation and judging can vary between runs.
Adjust the rubric
If the judge repeatedly misinterprets a requirement, try making the rubric more specific. Open Evaluators and select plan-quality.
The evaluator’s prompt contains the rubric from rubric.txt. You can edit it here and rerun the quality test to inspect the judge’s explanation with the revised instructions. Changing the rubric also changes the basis for the score, so treat this as an adjustment to the evaluation itself.
Copy any rubric changes you want to keep into src/main/resources/evaluation/rubric.txt. Dev Services removes the Langfuse containers when dev mode stops, and the next start recreates the evaluator from that file.
Inspect model costs and latency
Open the project dashboard to inspect model costs and timing across the recorded runs. The trace names identify the agents and tools, which helps locate the calls responsible for a slow or expensive plan.
Scroll down to the latency charts and hover over a trace name. The tooltip shows the agent class and method so you can locate the corresponding code.
Further reading
Eric Deandrea’s non-deterministic-no-problem project uses the same Quarkus Langfuse extension and goes further than this step. It detects drift in a running application with a guardrail and scores whole sessions as well as single traces. The Langfuse documentation on running experiments through OpenTelemetry describes the langfuse.experiment.* attributes the quality run sets.
You can now rerun a trip request after changing the planner and inspect how the generated plan meets its requirements. This completes Section 3. Continue to the conclusion to review the workshop.