We broke our own MLflow scorer on purpose, on the two answers it should have failed, and the evaluation score went up from 0.75 to 1.00. The tickets, their labels and the answers were the same in both runs, which our script confirmed from MLflow's own result export. A release rule that read the mean alone would have approved an answer set that still sent two support tickets to the wrong queue.
Earlier this month we designed a release gate for Databricks data quality. The next question was what a gate like that should read when an AI application is evaluated with MLflow, so we built a small local experiment around mlflow.genai.evaluate(). A code-based scorer called route_match checks saved routing answers for eight synthetic support tickets, and in one run it raises RuntimeError('Injected scorer dependency failure') on the two answers that are wrong.
The scorer's exceptions did not stop the evaluation. MLflow recorded each one as an error, left that case out of the mean and returned a normal result.
What MLflow records when a scorer raises an exception
mlflow.genai.evaluate() accepts saved inputs, outputs and expectations, so the experiment needed no model endpoint and no Databricks workspace. The Databricks code-based scorer reference says that when a scorer raises an exception, MLflow captures it as feedback with a null value and the error details, and the evaluation carries on. A normal return from evaluate() tells you the call finished. It doesn't tell you every required assessment succeeded.
We kept three outcomes apart in our results. A wrong route is a failure, because the application answered badly. A broken lookup inside the scorer is an error, because nobody judged the answer at all. If the scorer caught its own exception and returned True, missing evidence would turn into a pass. Returning False would blame the application for a defect in the evaluator, and the report would show a regression that never happened.
Three runs over the same saved answers
The fixture holds ten synthetic support tickets. Eight carry a hand-written routing label: billing, access or technical. The other two, a contract cancellation and a sales quote, belong to separate workflows, so we marked them out of scope before any evaluation ran and kept them in the manifest with their exclusion reasons.
Candidates A and B are saved answer sets written into the fixture, with no trained model behind them. Candidate A routes six of the eight eligible tickets correctly. It sends "The invoice is paid but my account is still locked" to billing, where the label says access, and "The billing page crashes before I can download an invoice" to billing, where the label says technical. Candidate B corrects those two answers and changes nothing else.
We ran three evaluations with MLflow 3.17.2 on Python 3.14.5. The first scored candidate A with a working scorer. The second kept the tickets, labels and candidate A's answers unchanged and made the scorer raise a RuntimeError on the two misrouted cases, c07 and c08. The third scored candidate B with the working scorer. The whole difference between the working and the broken scorer is one flag:
@scorer(name="route_match")
def route_match(inputs, outputs, expectations) -> bool:
if inject_errors and inputs["case_id"] in {"c07", "c08"}:
raise RuntimeError("Injected scorer dependency failure")
return outputs == expectations["label"]
In the complete run the mean is 6/8, which is 0.75, with all eight eligible cases scored. In the broken run MLflow recorded six passing scores and two error assessments, and route_match/mean came back as 6/6, which is 1.00, while coverage fell from 100% to 75%. Candidate B passed all eight at full coverage.
Candidate A's score rose by 0.25 while its two wrong answers stayed exactly as they were.
We measured this for a Boolean scorer and the default mean in MLflow 3.17.2, on a Windows workstation. We haven't run it in a Databricks workspace or checked another MLflow version, so we'd repeat the broken run after any upgrade before relying on the same behaviour.
Coverage counted against a frozen manifest
For each required scorer we built the expected set of case IDs from the manifest, then reconciled the returned assessments against it by case ID, never by row order. Only the manifest can exclude a case. An empty result is never read as "not applicable", and a duplicate, unexpected or changed case ID invalidates the report, so the denominator can't shift without anyone noticing. Every case ends in one of five statuses.
| Status | Meaning |
|---|---|
| Pass | A valid Boolean score is true. |
| Fail | A valid Boolean score is false. |
| Error | The assessment carries the evaluator's error details. |
| Missing | An eligible case has no usable score and no recorded error. |
| Not applicable | The manifest excludes the case, with a reason written before the run. |
For this binary check, coverage is (pass + fail) / eligible and quality is pass / (pass + fail). A failing score counts toward coverage because it tells us something about the answer. Errors and missing assessments don't. With no eligible cases or no valid scores the ratio is undefined, and the release is held. We count coverage separately for every required scorer. A formatting check that always completes could otherwise cover for a correctness scorer that rarely does. Our example demands 100% coverage. A team that accepts less needs minimums per language or ticket category as well, or the unscored cases can pile up in one group.
The release rule and its 0.90 threshold
The rule in our example requires full coverage, no errors or missing assessments, a reconciled ledger and quality of at least 0.90. With eight eligible cases that means all eight must pass, because 7/8 is 0.875. We chose those numbers for the example. MLflow doesn't set a release threshold.
release = (
ledger_valid
and eligible > 0
and scored == eligible
and errors == 0
and missing == 0
and quality is not None
and quality >= 0.90
)
The rule held candidate A twice: once for quality when the evaluation was complete, and once for incomplete evidence when the scorer broke. It approved candidate B.
A repaired scorer gets a full rerun on the frozen candidate. The original error report stays on file, and the repaired run records the scorer's revision. Rerunning only the cases that errored, or comparing two candidates scored on different subsets, leaves a comparison nobody can check.
Checks on the gate and the evidence it keeps
The example runs 23 checks, and every one passed. Beyond the three runs above, they remove a result row or a single assessment, inject a malformed score, duplicate a result, an assessment or the manifest, add an unexpected case ID or a changed input, and empty the eligible set. Each of those either holds the release or rejects the ledger.
Each run saves the fixture, the resolved dependency versions, per-case statuses, raw MLflow feedback, the metrics and the reason behind every decision. Run IDs and trace IDs change on a rerun; the scores and decisions don't.
Eight synthetic tickets test the gate. They don't measure how accurately a real application routes tickets, which takes a representative dataset and explicit pass requirements for the cases that matter most. A production evaluation also records the application and prompt revisions, the scorer configuration and the dataset version. Subjective scorers need calibrating against human-reviewed examples, because full coverage shows that the evaluation ran, and nothing about whether the scorer measures the right thing.
Break one scorer on purpose before the next release
Take the evaluation your release already reads, make one scorer raise an exception on two cases you know are wrong, and run it again. If the reported score goes up and the release would still go out, add a coverage count against a frozen list of case IDs before the next candidate is evaluated.
Frequently asked questions
Does MLflow fail the evaluation when a scorer raises an exception?
No. In our run with MLflow 3.17.2, both RuntimeErrors became error assessments with a null value and error details, and evaluate() returned normally. The Databricks scorer reference documents the same behaviour, so the release rule has to look for errors itself.
Why not catch the exception inside the scorer and return False?
Returning False marks the application's answer as wrong when the evaluator is what broke, so an evaluator bug shows up as an application regression. Returning True is worse, because missing evidence becomes a pass. When the scorer lets the exception through, MLflow keeps the error, the report can show quality and coverage side by side, and the rule can hold on either.
Can a release go out with less than 100% coverage?
Yes, if the team sets that minimum before the run and writes it into the rule. Set it per scorer and per group that matters, such as language or ticket category, so unscored cases can't gather where the application is weakest. Our example kept 100% because it has only eight eligible cases.
Does the same rule work for LLM judges?
The coverage count reads the ledger, so an errored judge assessment is counted the same way as an errored code-based scorer. Our experiment used a deterministic scorer and made no model calls. Judges raise questions this run didn't test, such as calibration against human-reviewed examples and variation from one run to the next.