Evaluation and quality
Put a number on whether the new version is better
A Databricks adoption accelerator for quality engineering, experimentation, and governed delivery.
- Role in the family
- Evaluation and quality
- Databricks surfaces
- MLflowUnity CatalogDeltaLakeflowModel ServingAI Gateway
- Status
- Live across a typed API, a CLI, a studio UI, Cloudflare edge workers for assignment and ingestion, and Temporal for durable workflows.
- Documentation
- experiments.techfabric.com
The problem
Teams ship a model or an agent and cannot say whether the new version is better than the old one. Evaluation lives in notebooks, experiment assignment is ad hoc, and nothing blocks a regression from reaching production.
How it works
Built on Databricks SQL, Delta, Lakeflow, Unity Catalog, managed MLflow, Model Serving, AI Gateway, Apps, and Lakebase. Databricks stays authoritative for compute, data, lineage, and governance; Experiments adds A/B assignment, cross-workload evidence, and production quality gates. Native MLflow runs link into the quality centre while their traces and artifacts stay in Databricks.
What it does
Quality gates that block
A gate is a policy checkpoint, so a failed evaluation stops the promotion rather than filing a warning somebody reads later.
Signed manifests at the edge
Experiment assignment is served from a signed, key-verified manifest at the edge.
Durable exposure ingestion
Exposure and conversion events queue, retry, and land as idempotent batches for Auto Loader.
Linked MLflow evidence
Native runs are referenced, not copied. Datasets, traces, and lineage stay in Databricks.
You can answer whether the new version is better, mechanically. Assignment, evaluation, and quality gates run against your own Databricks data, so shipping a model or agent change stops depending on somebody's judgement call.
Everyone has an opinion. Nobody has a score.
A model or an agent changed. Whether it improved is answered in a meeting, by the person most confident in the room.
The evaluation is a notebook
It exists, one person runs it, and the results live in their output cells. Nothing about that survives them going on holiday.
Ground truth is somebody's memory
What a good answer looks like was agreed verbally and has drifted since. Two reviewers score the same output differently and both are being reasonable.
A regression ships
The prompt changed, the demo looked fine, and the thing that got worse was not the thing anybody checked.
Without a suite that runs on every change, every prompt edit is a matter of opinion, and the loudest opinion wins by default rather than by evidence.
The shift
Three ways to decide whether it got better
Two of them are opinions with extra steps.
Look at some outputs
Fast, and unrepeatable
A notebook one person runs
Real, and not durable
A scored suite in the pipeline
Repeatable, and it blocks
What gets installed
The apparatus that turns a judgement into a number somebody can argue with.
Ground truth you own
A dataset of what a good answer looks like, versioned, in your workspace, that outlives whoever assembled it.
Deterministic scorers
The checks that do not need a model to run, so a regression in the obvious things is caught for nothing.
Model-based judges
For the answers where correctness is a matter of degree, scored consistently rather than by whoever is reviewing.
A/B on the real thing
Two versions against the same suite and the same data, so the comparison is between them rather than between two demos.
Gates in the delivery path
A failed evaluation stops a promotion. The suite is a checkpoint rather than a report.
The history, kept
Every run scored and retained, so the question of when a thing got worse has an answer with a date on it.
Every change
Scored before it can be promoted
We needed it before we sold it
Experiments came out of not being able to answer whether our own agents had improved. It is free and open source under Apache-2.0 and the ground truth stays in your workspace.
FAQ
TechFabric Experiments, answered
Does this replace MLflow?
No. Databricks stays authoritative for compute, data, lineage and governance. Native MLflow runs are referenced rather than copied, so datasets, traces and artifacts stay where they are, and Experiments adds assignment, gates and cross-workload evidence on top.
How do we know a new version is actually better?
Assignment, evaluation and quality gates run against your own Databricks data, so the comparison is mechanical. A failed evaluation blocks the promotion. When the thing being scored is a Genie space that executives have already stopped trusting, the named engagement is Genie Accuracy at /databricks/genie-accuracy.
Can we run experiments on agents as well as models?
Yes. The same gates apply to an agent version as to a model version, which is what lets you ship agent changes at a pace without guessing at the effect.