Skip to content
TechFabric

The cost dashboard said the workspace was fine

Ihor Seleznov8 min read

The finance partner sent a screenshot. The line was up and to the right, and the dashboard had a green badge next to "within forecast." She asked why, if the workspace was healthy, three teams were still complaining that jobs failed on Monday mornings.

We had read-only access by Wednesday: system tables where they had them, job run history, cluster events, Unity Catalog grants. Nobody gave us production data, and nobody needed to.

The green badge was telling the truth it was built to tell. Spend matched the forecast. The forecast had been written against a cluster estate that nobody had cleaned since the pilot.

What the dashboard cannot see

An all-purpose cluster sized for a load that never arrived will sit in a cost report as a successful reservation. A job that retries four times and then succeeds looks like a completed run. A schedule whose owner left in April still fires, still bills, and still writes to a table that two dashboards have quietly stopped trusting.

Unity Catalog coverage that stops halfway looks fine in the catalogue browser. You click a schema and you see grants. You miss the three schemas that were never migrated off hive_metastore, and the service principal that was granted ALL PRIVILEGES because someone had a demo on a Thursday.

I have lost count of how many times the first useful finding was a job that should have been serverless and was still pinned to a cluster whose autoscale floor was a guess from the original workshop.

Two weeks, then a scorecard

That work is now a named engagement, the Databricks Health Check. Two weeks, a fixed fee, and a written scorecard whether or not you do anything about it. We rank findings by severity and put an estimated saving against each one, with the reasoning shown, then a thirty, sixty and ninety day order.

The fee stays fixed if the workspace is actually healthy, which does happen. We would rather hand a CFO a scorecard that says so than invent a remediation. We do not need write access to run it. If security wants a named scope first, we write one.

What we do with the findings

The straightforward ones your team can take: turn this job serverless, kill that schedule, tighten this grant. The scorecard is written so that is possible without us.

The ones that keep coming back need a different kind of attention. A job that fails on Monday because an upstream load slipped, and the person who used to rerun it by hand is on another project. A warehouse that is the right size on Wednesday and the wrong size on month-end. Those workloads need a declared idea of healthy, and an intervention that leaves a record.

That is Fabric Radar. Monitors carry service level objectives. When one is breached, a restart or a reroute runs as a governed action: a policy check, an audit trail, a Temporal workflow so the fix survives if the worker dies at two in the morning. We do not point Radar at a workspace we have not looked at. The Health Check is the look.

A thing I still get wrong

I used to open these with the most expensive cluster, because it makes a good chart, and it is often the finding that matters least. The expensive cluster is usually owned. Someone can tell you why it exists, even if the why is two years old.

The cheap, forgotten schedule is the one that has been writing slightly wrong data into a gold table since February, and the dashboard that reads it has been "a bit off" for long enough that people stopped filing tickets. Cost is the reason we get invited. Drift is usually the reason we stay.

If you already know it is a mess

Send us the screenshot anyway. The scorecard turns a feeling about the bill into an order of work. Book the Health Check, or talk to an engineer if you want to know whether two weeks is even the right shape.