What would stop you releasing an AI agent to customers? An answer based on a restricted document should be reviewed as a permission failure, even if a model gives the response a high quality score.
I'd begin with the mistakes the team cannot accept and agree on examples with the people responsible for the task. Some checks have exact results, such as whether the answer cites an allowed source. Others need a person or another AI model to judge whether the retrieved information actually supports the answer.
What an evaluation score needs to tell you
A release review and live monitoring answer different questions. Before release, the team needs to know how a candidate performs on cases it can inspect and repeat. After release, monitoring helps find failures in the requests customers are making now, and the record of changes lets the team investigate how those results move over time.
Databricks documents scoring selected traces in the interface and repeating evaluations over a dataset, with assessments attached to individual traces. That lets a reviewer investigate the answers behind an aggregate score.
I would inspect a permission failure separately from the agent's overall score. Different wording may be acceptable, but a citation to a document the user cannot access gives the team a specific reason to stop a release.
Before comparing scores, I'd separate pass, fail, error and missing assessment. A failed answer has been assessed and found unacceptable; an error or missing assessment means the required evidence is incomplete. I would hold the release until each required assessment produced a usable result.
The published local MLflow 3.17.2 experiment used saved answers to synthetic support tickets and a Boolean scorer. Its report describes a rise from 0.75 to 1.00 after the scorer errored on the two answers it should have failed, while those answers remained unchanged. This was a local test of saved answers; it did not evaluate a deployed Databricks agent. TechFabric's work on evaluating trained models also describes agreeing on the examples and pass conditions before promotion.
How a trace explains an agent's answer
A trace records the operations behind one request, including the retrieved material and tool calls, with their inputs, outputs and errors. Each operation is a separate recorded step, called a span (Databricks tracing concepts).
For an agent answering from documents, start by looking at the passage it received. Did it contain what the answer needed? When it did, an unsupported response points towards answer generation; when it didn't, changing the answer prompt alone may leave the retrieval problem in place.
The trace also gives a permission check something concrete to inspect. Compare the requested documents with the set the application allowed that user to access, using the application's verified identity and access rules. That check answers a different question from whether a model considers the response well written. When access and retrieval checks pass, I'd examine the answer and its assessment.
A recorded tool call establishes what the agent requested. Confirming that the action reached another system still requires evidence from that system.
Which checks belong in code
I'd write a code check whenever the requirement has an exact pass condition. For a document-answering agent, these checks can cover the response format as well as the information it used:
| Requirement | What the code checks |
|---|---|
| Required output format | The response parses into the expected structure |
| Permitted citation | The cited document belongs to the user's allowed set |
| Expected tool arguments | The recorded call uses the expected values |
MLflow's custom code-based scorers can apply Python logic to evaluation inputs, outputs and traces. Include a reason with a failed assessment when someone will need to investigate it, because a rejected document reference is more useful to that person than an unexplained failed score.
A model judge is another AI model assessing the response, interpreting criteria such as whether the answer addresses the question or whether retrieved material supports its claims. MLflow's built-in judge reference lists, for each judge, whether it requires ground truth and whether it requires traces. Choose the criterion, then give the judge the evidence it requires.
For a document-access check, I wouldn't pay a model to compare references the application can check exactly. I'd reserve that cost for interpreting answers and repeating disputed judgments, while keeping permission failures separate from any score for the wording.
How to check a model judge
In our publishing workflow on 6 November 2026, the draft-checking tool blocked a recommendation about code checks and model judges as unconfirmed experience, although its explanation said the statement was supported.
| Part of the review record | Observed result |
|---|---|
| Tool | TechFabric's check_draft publishing check |
| Statement being checked | A recommendation to use code for exact requirements and model judges for interpretation |
| Blocking classification | First-hand (status fail) |
| Explanation | The recommendation was supported and presented no factual problem |
| Decision for this work | Keep the checks unchanged and request maintainer review of the criterion |
The record alone cannot establish whether the model or the surrounding workflow caused the discrepancy, so I would investigate why the verdict disagreed with its explanation before treating the result as release evidence.
I'd want to see where a judge disagrees with the people who understand the task before letting it influence a release. An answer it approves and a domain reviewer rejects is the case I'd inspect first: what did that person see that the judge missed? The reviewers would need agreed criteria, including when an answer should acknowledge missing information. MLflow supports aligning judges with human feedback. I would keep some reviewed examples aside while refining a judge, then use them to see whether the revision agrees with people on answers it wasn't adjusted against.
Repeat the judging on fixed answers to see how much the scores vary. If you rerun the agent at the same time, you mix changes in its responses with changes in the evaluator's assessment, making a small score movement harder to investigate before calling it an improvement.
If the judge changes, the scores may move while the agent's answers stay the same. I would re-score those saved answers with the revised judge before attributing any difference to the agent. MLflow supports registering and versioning custom and built-in LLM judges in an experiment and loading a specific version. This registration/versioning mechanism does not cover code-based scorers or Guidelines judges. For a comparison, I would record the judge's definition and model alongside the test set and agent version.
How live failures improve the next evaluation
A fixed test set gives a team cases it can repeat before release. Live monitoring exposes requests that set did not anticipate. Databricks production monitoring, currently in Beta, assesses a configurable sample of incoming traces in the background and attaches feedback to them. Choose the sample with the cost of judging and the consequences of a missed failure in mind.
Send flagged live failures for human review before adding them to the next evaluation. Once a reviewer has established what the agent should have done, the confirmed case can join the versioned test set. Before release, I would run each candidate on that set or score its saved answers and traces, then review its code checks and model judgments together. Sampled monitoring continues after release.
A test set built entirely from past incidents could improve while ordinary answers deteriorate. I'd keep representative requests alongside cases where evidence is missing or the request falls outside the task, and examine each group's results. MLflow's evaluation datasets can hold those inputs with or without expected outputs and support testing known failures.
Where we'd start with one agent task
For a document-answering agent, the first discussion would be about which sources it may use and what evidence makes an answer acceptable. I'd ask the people responsible for that task to describe examples they can review, including requests where the right response is to acknowledge that information is missing.
I'd keep the following fields in the team's review record:
| Field | What it records |
|---|---|
| Test case | Input, user permissions and the risk being tested |
| Expected behaviour | Reference facts when available, acceptable uncertainty and prohibited actions |
| Pass conditions | Required checks and the failures that would stop release |
| Comparison versions | Dataset, agent, prompt and tool versions, plus scorer and judge definitions and model configuration |
| Case outcome | Pass, fail, error or missing assessment, with the answer, trace reference and reason |
Those examples, together with known failures and a record of expected behaviour, would form the first test set. Tracing the agent as it answers them gives the team something concrete to investigate. I'd begin with code checks and add model judging where interpretation is needed, reviewing disagreements with people who understand the task.
Before seeing the next candidate's scores, the team could agree which failures should stop its release and use the saved baseline answers and assessments to compare both candidates under the same conditions. After release, reviewed live failures would extend the set while the ordinary requests remained part of the evaluation.
Frequently asked questions
Can an AI agent be evaluated without a single correct answer?
Yes. Expected behaviour can specify facts an answer must support while allowing different wording. Have reviewers agree on those facts and acceptable uncertainty before asking a model to assess the response.
How do you check whether an LLM judge is reliable?
Agreement with domain reviewers is the first check. I'd compare the judge's assessments with their reviews, paying particular attention to false approvals, where it accepts an answer people consider unacceptable. A material change to the judge would need another review.
Which agent checks can run without labelled answers?
MLflow lists relevance to the question and groundedness in retrieved material among the judgments that do not require a reference answer, although groundedness requires retrieval traces. Those checks still depend on the input and evidence their criterion needs. Confirm that both are present before interpreting a missing or failed assessment.
How should changes to the evaluation dataset affect score comparisons?
Once the test set changes, the earlier and newer scores cover different examples. Re-running both comparison versions against the changed dataset would make their aggregate scores comparable again. Keeping the earlier dataset and results would also help reviewers distinguish a harder test set from a change in the agent, while inspecting the new cases individually.