# How we block AI releases when MLflow scorers fail

> A scorer that crashed on two wrong answers raised our MLflow evaluation score from 0.75 to 1.00. We built a release rule that counts coverage first.

Published: 2026-10-29
Author: Ihor Seleznov
Tags: Databricks, AI/ML, Knowledge Share
Canonical: https://www.techfabric.com/blog/how-we-block-ai-releases-when-mlflow-scorers-fail

---

We broke our own MLflow scorer on purpose, on the two answers it should have failed, and the evaluation score went up from 0.75 to 1.00. The tickets, their labels and the answers were the same in both runs, which our script confirmed from MLflow's own result export. A release rule that read the mean alone would have approved an answer set that still sent two support tickets to the wrong queue.

Earlier this month we [designed a release gate for Databricks data quality](https://www.techfabric.com/blog/databricks-data-quality-gates). The next question was what a gate like that should read when an AI application is evaluated with MLflow, so we built a small local experiment around `mlflow.genai.evaluate()`. A code-based scorer called `route_match` checks saved routing answers for eight synthetic support tickets, and in one run it raises `RuntimeError('Injected scorer dependency failure')` on the two answers that are wrong.

The scorer's exceptions did not stop the evaluation. MLflow recorded each one as an error, left that case out of the mean and returned a normal result.

![An evaluation tray contains blue approved tiles and amber empty slots behind a closed release barrier.](/blog-media/how-we-block-ai-releases-when-mlflow-scorers-fail/hero.png)

## What MLflow records when a scorer raises an exception

`mlflow.genai.evaluate()` accepts [saved inputs, outputs and expectations](https://mlflow.org/docs/latest/api_reference/python_api/mlflow.genai.html#mlflow.genai.evaluate), so the experiment needed no model endpoint and no Databricks workspace. The Databricks [code-based scorer reference](https://docs.databricks.com/aws/en/mlflow3/genai/eval-monitor/custom-scorer-reference#error-handling) says that when a scorer raises an exception, MLflow captures it as feedback with a null value and the error details, and the evaluation carries on. A normal return from `evaluate()` tells you the call finished. It doesn't tell you every required assessment succeeded.

We kept three outcomes apart in our results. A wrong route is a failure, because the application answered badly. A broken lookup inside the scorer is an error, because nobody judged the answer at all. If the scorer caught its own exception and returned `True`, missing evidence would turn into a pass. Returning `False` would blame the application for a defect in the evaluator, and the report would show a regression that never happened.

## Three runs over the same saved answers

The fixture holds ten synthetic support tickets. Eight carry a hand-written routing label: `billing`, `access` or `technical`. The other two, a contract cancellation and a sales quote, belong to separate workflows, so we marked them out of scope before any evaluation ran and kept them in the manifest with their exclusion reasons.

Candidates A and B are saved answer sets written into the fixture, with no trained model behind them. Candidate A routes six of the eight eligible tickets correctly. It sends "The invoice is paid but my account is still locked" to billing, where the label says access, and "The billing page crashes before I can download an invoice" to billing, where the label says technical. Candidate B corrects those two answers and changes nothing else.

We ran three evaluations with MLflow 3.17.2 on Python 3.14.5. The first scored candidate A with a working scorer. The second kept the tickets, labels and candidate A's answers unchanged and made the scorer raise a `RuntimeError` on the two misrouted cases, `c07` and `c08`. The third scored candidate B with the working scorer. The whole difference between the working and the broken scorer is one flag:

```python
@scorer(name="route_match")
def route_match(inputs, outputs, expectations) -> bool:
    if inject_errors and inputs["case_id"] in {"c07", "c08"}:
        raise RuntimeError("Injected scorer dependency failure")
    return outputs == expectations["label"]
```

![Three local MLflow runs: candidate A scores 0.75 at full coverage and is held; its broken evaluation scores 1.00 at 75% coverage and is held; corrected candidate B scores 1.00 at full coverage and is approved.](/blog-media/how-we-block-ai-releases-when-mlflow-scorers-fail/coverage-results.png "Three local runs with MLflow 3.17.2: the broken scorer lifted the mean to 1.00 while coverage fell to 75%.")

In the complete run the mean is 6/8, which is 0.75, with all eight eligible cases scored. In the broken run MLflow recorded six passing scores and two error assessments, and `route_match/mean` came back as 6/6, which is 1.00, while coverage fell from 100% to 75%. Candidate B passed all eight at full coverage.

Candidate A's score rose by 0.25 while its two wrong answers stayed exactly as they were.

We measured this for a Boolean scorer and the default mean in MLflow 3.17.2, on a Windows workstation. We haven't run it in a Databricks workspace or checked another MLflow version, so we'd repeat the broken run after any upgrade before relying on the same behaviour.

## Coverage counted against a frozen manifest

For each required scorer we built the expected set of case IDs from the manifest, then reconciled the returned assessments against it by case ID, never by row order. Only the manifest can exclude a case. An empty result is never read as "not applicable", and a duplicate, unexpected or changed case ID invalidates the report, so the denominator can't shift without anyone noticing. Every case ends in one of five statuses.

| Status | Meaning |
|---|---|
| Pass | A valid Boolean score is true. |
| Fail | A valid Boolean score is false. |
| Error | The assessment carries the evaluator's error details. |
| Missing | An eligible case has no usable score and no recorded error. |
| Not applicable | The manifest excludes the case, with a reason written before the run. |

```mermaid
%% caption: An errored case drops out of the MLflow mean, so the release rule counts coverage against the manifest before it reads quality.
flowchart TD
  A["Manifest of eligible case IDs"] --> B["mlflow.genai.evaluate()"]
  B --> C["Scorer returns True or False"]
  B --> D["Scorer raises: null value and error details"]
  C --> E["Counted in route_match/mean"]
  D --> F["Left out of the mean, marked Error in the ledger"]
  E --> G["Release rule: coverage first, then quality"]
  F --> G
  class G accent
```

For this binary check, coverage is (pass + fail) / eligible and quality is pass / (pass + fail). A failing score counts toward coverage because it tells us something about the answer. Errors and missing assessments don't. With no eligible cases or no valid scores the ratio is undefined, and the release is held. We count coverage separately for every required scorer. A formatting check that always completes could otherwise cover for a correctness scorer that rarely does. Our example demands 100% coverage. A team that accepts less needs minimums per language or ticket category as well, or the unscored cases can pile up in one group.

## The release rule and its 0.90 threshold

The rule in our example requires full coverage, no errors or missing assessments, a reconciled ledger and quality of at least 0.90. With eight eligible cases that means all eight must pass, because 7/8 is 0.875. We chose those numbers for the example. MLflow doesn't set a release threshold.

```python
release = (
    ledger_valid
    and eligible > 0
    and scored == eligible
    and errors == 0
    and missing == 0
    and quality is not None
    and quality >= 0.90
)
```

The rule held candidate A twice: once for quality when the evaluation was complete, and once for incomplete evidence when the scorer broke. It approved candidate B.

A repaired scorer gets a full rerun on the frozen candidate. The original error report stays on file, and the repaired run records the scorer's revision. Rerunning only the cases that errored, or comparing two candidates scored on different subsets, leaves a comparison nobody can check.

## Checks on the gate and the evidence it keeps

The example runs 23 checks, and every one passed. Beyond the three runs above, they remove a result row or a single assessment, inject a malformed score, duplicate a result, an assessment or the manifest, add an unexpected case ID or a changed input, and empty the eligible set. Each of those either holds the release or rejects the ledger.

Each run saves the fixture, the resolved dependency versions, per-case statuses, raw MLflow feedback, the metrics and the reason behind every decision. Run IDs and trace IDs change on a rerun; the scores and decisions don't.

Eight synthetic tickets test the gate. They don't measure how accurately a real application routes tickets, which takes a representative dataset and explicit pass requirements for the cases that matter most. A production evaluation also records the application and prompt revisions, the scorer configuration and the dataset version. Subjective scorers need calibrating against human-reviewed examples, because full coverage shows that the evaluation ran, and nothing about whether the scorer measures the right thing.

## Break one scorer on purpose before the next release

Take the evaluation your release already reads, make one scorer raise an exception on two cases you know are wrong, and run it again. If the reported score goes up and the release would still go out, add a coverage count against a frozen list of case IDs before the next candidate is evaluated.

## Frequently asked questions

### Does MLflow fail the evaluation when a scorer raises an exception?

No. In our run with MLflow 3.17.2, both `RuntimeError`s became error assessments with a null value and error details, and `evaluate()` returned normally. The Databricks scorer reference documents the same behaviour, so the release rule has to look for errors itself.

### Why not catch the exception inside the scorer and return False?

Returning `False` marks the application's answer as wrong when the evaluator is what broke, so an evaluator bug shows up as an application regression. Returning `True` is worse, because missing evidence becomes a pass. When the scorer lets the exception through, MLflow keeps the error, the report can show quality and coverage side by side, and the rule can hold on either.

### Can a release go out with less than 100% coverage?

Yes, if the team sets that minimum before the run and writes it into the rule. Set it per scorer and per group that matters, such as language or ticket category, so unscored cases can't gather where the application is weakest. Our example kept 100% because it has only eight eligible cases.

### Does the same rule work for LLM judges?

The coverage count reads the ledger, so an errored judge assessment is counted the same way as an errored code-based scorer. Our experiment used a deterministic scorer and made no model calls. Judges raise questions this run didn't test, such as calibration against human-reviewed examples and variation from one run to the next.