From a six-line ticket to embedded analytics on Databricks
Andrii's title slide. The talk was in Ukrainian; this is his English version of the deck.
Eight steps, most of them rebuilt at least once
The eight steps the project went through. Most of them were built twice.
Six lines in the ticket, a seventh in conversation
What the customer asked for, plus the line that came up on a call afterwards: they wanted to see a row they'd written a minute ago.
"Real time" and "no impact" cannot both be true
You can't have live numbers and zero load on the app at the same time. With no data team and no ETL budget, they built the cheapest version first and measured.
Why Databricks, not a BI tool on a read replica
Why Databricks and not a BI tool on a read replica: the customer wants AI next, and Unity Catalog lets dashboards, Genie and models share one set of tables and permissions.
Step one: read the live database in place
Step one. Lakehouse Federation reads the app's Azure SQL database in place, so there was a demo within days and no pipeline to build.
Pushdown is per table, not per join
A view filtered one side of a join and not the other, so Azure SQL sent back 6.46 million rows in production to show about 800.
Measure server-side: wall clock lies by 1.5 to 2 s
Timing through the REST API adds 1.5 to 2 seconds. Read durations from query history instead.
Step two: one dataset per widget hurt the database
Step two. One dashboard fired 285 queries because every chart had its own dataset. One shared dataset per dashboard fixed it.
Four Lakeview traps that cost us a day each
Four AI/BI dashboard quirks that each cost the team a day.
Step three: pages, not dashboards
Step three. Five reports became pages on one dashboard, so switching between them doesn't reload anything.
Step four: the requirement nobody wrote down
Step four. Embedded viewers have no Databricks login, so tenant isolation lives in the dashboard SQL, keyed on a value from the embed token.
One JOIN, three ways to get zero rows silently
Three ways that one join quietly returns nothing, including case-sensitive ids.
One character took every board down
The outage. A pipe character in the tenant key blanked every dashboard for every customer.
Scope version: bust one tenant's cache only
The scope version doubles as a cache key, so changing one tenant's access doesn't clear the cache for anyone else.
Step five: materialize all or nothing
Step five. With everything in Delta a typical query took about half a second. With a Delta fact and federated lookups it was nearly four times slower.
What it bought: a full dashboard load, replayed
A full dashboard load went from about two minutes to 20 seconds.
Things we tried so you do not have to
Six things they tried that don't work, so you don't have to.
Step six: business hours as architecture
Step six. Refresh every 30 minutes during the customer's working hours, about $43 a month.
Step seven: the Refresh button Databricks lacks
Step seven. A Refresh button that actually runs the refresh job, and doesn't report success when the job was skipped.
Step eight: filtering past NOT_SUPPORTED
Step eight. Filters passed through the embed URL, after the SDK route returned NOT_SUPPORTED.
Serverless cost: the idle tail is the whole story
Most of the serverless bill was the warehouse sitting idle before it shut down.
The ownership transfer that caused an outage
Transferring ownership of a view took away access the dashboards relied on. Let a group own production.
Shipping dashboards like code
Dashboards deploy like code, with a smoke test that opens every one.
The ticket, answered honestly
The original six lines, and how well each one got answered.
Still scaffolding: the steps ahead of us
What's still to do, including Genie for customers and moving app data to Lakebase.
Six rules we would give ourselves on day one
The six rules Andrii would give himself on day one.
Thank you
Questions, and a link to the written series on the blog.
The session came out of work we do for clients. If any of it maps onto something you are deciding, these are the places to start.
Unity Catalog consulting
Most Unity Catalog rollouts stop halfway. The catalogues exist, half the estate is still external tables against the old metastore, and the one thing everybody remembers is the service principal with access to everything because a deadline was coming.
