We changed what total revenue meant in a Databricks metric view, and Genie's built-in benchmark still scored all three revenue questions Good. Then we ran the same change through a golden set kept outside Genie, and it flagged six answers for review.
The difference between the two runs was 259,716 dollars of revenue. If your team signs off a Genie agent on the Benchmarks tab alone, that's the kind of change it won't tell you about, so it's worth knowing why and what to run next to it.
How we set up the test
We kept the setup small. The data is 5,000 synthetic orders, each with a region, a channel, a status (completed, cancelled or refunded) and an amount.
On top of it sits a Unity Catalog metric view with three measures, total_revenue, order_count and avg_order_value. One Genie Agent reads that metric view and nothing else, and it has no instructions to start with.
The golden set is ten questions a business user would actually type, from "What was our total revenue?" to "Revenue and number of orders by region for store orders". A notebook sends each one to Genie through the Conversation API as a new conversation, takes the SQL Genie wrote, runs it, and compares the result with what we recorded earlier. mlflow.genai.evaluate logs every run, so each one ends up as an MLflow run with its scores.
Why the built-in benchmark stayed green
Genie Agents have a Benchmarks tab. You give it a question and a ground-truth SQL answer, it asks the agent, and it scores the agent's answer against that SQL.
We added three revenue questions from the golden set with their SQL and ran the benchmark while revenue counted completed orders only. Total revenue came back as 1,052,434.66 from Genie and from the ground truth, so it scored Good, 3 out of 3.
Then we switched total_revenue back to every order, whatever its status, and ran the benchmark again a minute later. Genie said 1,312,150.88, the ground truth said 1,312,150.88, and it scored Good again, 3 out of 3.
The benchmark re-executes the ground-truth SQL when it runs. The expected answer lives as a query, so when the metric moves the expectation moves with it. Genie's own text answers did describe each definition correctly ("completed orders only" one time, "includes all statuses" the next), but nothing in the score changed.
That's fine for what the tab is for. When you're tuning instructions and want to know whether Genie writes the right query, re-running the SQL is exactly what you want. It just can't tell you that last week's number and this week's number are different.
A golden set with stored results and a definition hash
Our golden set stores the result of each expected query once, as a snapshot, in a Delta table. Next to it goes a hash of the metric view definition as Unity Catalog stores it, plus the comments on the source table's columns, since Genie reads those too.
For each golden set question, Genie writes the SQL, the notebook runs it and compares the result with the snapshot, and then it checks whether the hash changed. Every run gives each question one of three verdicts:
- PASS: Genie's result matches the snapshot.
- REVIEW: definition changed: it doesn't match, and the hash has changed since the snapshot. Someone changed the metric, and a person should decide whether that was intended.
- REGRESSION: it doesn't match, and the hash is the same. The definition didn't move, so something else broke.
We compare results instead of SQL text. Genie wrote the same intent differently from run to run. For "What was revenue in March 2026?" it used a date range in one run and order_month = '2026-03-01 00:00:00' in another, and now and then it added IS NOT NULL filters, an ORDER BY or a HAVING. A text diff would have failed all of those. The comparison sorts rows and rounds numbers to two decimals, so row order doesn't matter.
The part that talks to Genie is short:
def ask_genie(question: str) -> dict:
out = {"genie_sql": None, "rows": None, "def_hash": definition_hash()}
msg = w.genie.start_conversation_and_wait(space_id, question)
for att in msg.attachments or []:
if att.query and att.query.query:
out["genie_sql"] = att.query.query
if out["genie_sql"]:
out["rows"] = normalise(spark.sql(out["genie_sql"]).toPandas())
return out
match = out["rows"] is not None and out["rows"] == golden["expected_rows"]
same_def = out["def_hash"] == golden["def_hash"]
verdict = "PASS" if match else ("REVIEW: definition changed" if not same_def else "REGRESSION")
We changed revenue twice, once to completed orders only and once back to every order, which is the change the benchmark scored Good. Both times the harness gave six REVIEW and four PASS. The six were every question that touches revenue. The four were the order counts and the average order value, which the change didn't touch. After the first change we agreed it was intended, re-recorded the snapshot, and the next run was 10 out of 10.
One instruction broke seven answers, including a count
The hash only sees the metric view and the column comments. So we tried a change it can't see: one line in the agent's General Instructions.
When asked about revenue, count completed orders only (status = 'completed').
The metric view still said revenue meant every order. Genie followed the instruction anyway and added WHERE status = 'completed' to every revenue question. Seven of ten answers came back as REGRESSION, with the hash unchanged on all ten.
The one we didn't expect was "How many orders were placed?". The question says nothing about revenue, and Genie still wrote SELECT MEASURE(order_count) ... WHERE status = 'completed'. The instruction leaked into a plain count. Order count by channel, average order value and the cancelled-orders count were left alone.
We removed the instruction and ran the set again. It came back 10 out of 10.
What the harness costs and what it still misses
The snapshot is the point, and it's also the cost. After every change you approve, somebody has to re-record the golden set, and that's a person deciding, not a job you can schedule.
It's slow, too. Each question is a full Genie conversation, and a run of ten took about five minutes, so a set of a hundred is a run you'd schedule and check later.
A snapshot also only holds while the data under it stays put. Our orders didn't change between runs. For a question over live data, like this month's revenue, the stored number goes stale on its own, and re-running the expected SQL against current data is the right check. That's the case the built-in benchmark covers, and there the hash is what still tells you the definition moved.
The hash tells you what kind of change you're looking at only for the definitions it covers. An edited instruction, a new SQL example or a model change all show up as REGRESSION, because only a re-run sees them. That's still useful, since REGRESSION means "look now", but the verdict won't tell you which of those it was.
Genie also isn't deterministic on unchanged definitions. In the very first baseline run, "How many orders were cancelled?" came back with an extra status column next to the count, so the comparison failed and the run scored 9 out of 10. Repeated on the same data, it came back as a plain count and scored 10 out of 10. A single red answer is a reason to look, and two baseline runs before you trust the set are worth the five minutes.
Where to start with your own Genie agent
Take ten to twenty questions the business already asks, run the queries you trust, and store their results with a hash of the definitions they depend on, outside Genie. Re-run that set after every change to a metric, an instruction or the model, and keep the Benchmarks tab for what it's good at, which is tuning how Genie writes a query.
If the set grows past what one notebook should carry, that's the work behind Genie Accuracy and TechFabric Experiments. Andrew Ripley wrote about the other half of the problem, two teams meaning different things by the same metric, in Conflicting metric definitions in Databricks Genie.
Frequently asked questions
Do Databricks Genie benchmarks catch metric definition changes?
Not when the ground-truth answer is SQL against the changed metric. The benchmark runs that SQL again at scoring time, so the expected number changes along with Genie's answer. In our test both runs scored 3 out of 3 Good with totals 259,716 apart.
How do I regression test a Genie agent?
Keep a set of real questions with their expected results stored as data, outside Genie, and send each question through the Conversation API on every change. Compare the result rows rather than the SQL text, and record which definition the expectations were taken against so you can tell an intended change from a broken answer.
Can a Genie instruction override a Unity Catalog metric view?
In our test it did. A one-line instruction to count completed orders only made Genie add a status filter on top of a metric view that defined revenue over every order. It also added that filter to a plain order count, which the instruction never mentioned.
Why does Genie generate different SQL for the same question?
Genie writes a new query for every conversation, and the same intent can come out as a date range or an exact timestamp, with or without extra filters and sorting. Across our two baseline runs that changed the result once in twenty answers, when it added an extra column, which is enough to fail a strict comparison.
When is Genie's built-in benchmark the right check?
When you're tuning an agent and want to know whether Genie writes the right query, and when the question runs over data that keeps changing. In both cases you want the expected SQL run against today's data, which is what the benchmark does. Pair it with a check on the definition itself, so a changed metric doesn't pass as a correct answer.