Skip to content

Unity Catalog before the first pipeline

Sushruth Aeluguri3 min read

The pilot had a bronze layer, a dashboard, and a Genie space that demoed well on a Thursday. It didn't have Unity Catalog.

Someone had created the schemas in hive_metastore because that was the default in the workspace they were given. The pipeline wrote a Delta table, the graph looked good on a slide, and they kept going. Six weeks later security asked who could see the customer table, and the honest answer was anyone with workspace access. That's an answer that ends a rollout.

I have the same conversation about agents. Draw the permissions boundary before you hand anything a service principal, because once an agent can reach production data without a grant model you can defend, you've got an incident with a date on it. For a lakehouse the version is shorter, and I still have to say it every time. Grants, then data.

Why people skip it

Unity Catalog looks like ceremony when you're trying to prove the platform. A workshop cluster, a sample dataset, a notebook that lands one table, and you can have a graph on a slide by Friday. Putting the catalogue, the schemas, the storage credentials and the grants in place first looks like you haven't started, which is the opposite of how a security review will read that week.

I've watched teams retrofit Unity Catalog onto a lakehouse that already had six months of tables in it. Moving the metadata is doable. Reconstructing who was supposed to see what isn't, because that knowledge lived in the people who ran the notebooks, and two of them have moved on.

Workshop cluster, sample dataset, one table

Graph on a slide by Friday

Six months of tables

Retrofit Unity Catalog

Who was supposed to see what?

Catalogue, schemas, storage credentials and grants first

Notebook lands the first table

Putting the catalogue, schemas, storage credentials and grants in place first saves reconstructing who was supposed to see what after six months of tables.

Six weeks, with the scope fixed

The Lakehouse Launchpad is how we stand Databricks up when the job is a foundation. Six weeks of identity and environment strategy, Unity Catalog structure, ingestion through Lakeflow or Jobs, bronze silver and gold for the domains you named at the start, the dashboards people asked for, a Genie space with real definitions, and runbooks your team can use.

The date holds if access holds. If credentials and source-system owners take three weeks to appear, the clock reflects that, and we say so during scoping. If the discovery before the build shows six weeks isn't honest for the estate, we rescope before anyone starts.

Where the Launchpad replaces a warehouse, TechFabric Airlift does the migration underneath. Profiling, conversion, reconciliation, a reversible cutover. Where the output is an application in the workspace, TechFabric Runway is how that app gets preview environments and a gate on promotion.

The thing that actually stops a handover

I used to worry about slow pipelines. The conversation that kills a programme is "the only person who understands the grants is on holiday." A catalogue that matches how the teams actually work, with lineage intact as data moves, is what lets us leave. The runbooks are a deliverable for that reason.

Plenty of clients keep us on for the next programme. That should be a decision somebody makes, not a dependency created by a hive_metastore nobody documented.

If you already have a lakehouse and you're wondering whether to rebuild it, look at the Health Check first. The Launchpad is for standing something up.

Week one

We won't land a sample dataset in a demo schema and call it a platform, or wire Genie to tables whose definitions still live in a Slack thread, or grant a service principal ALL PRIVILEGES so the first job runs.

I'm usually in the first of those conversations when the work includes an agent. Even when it doesn't, the first artefact is the same. A catalogue structure you can defend, and grants that match the organisation you actually have.

If that's the job, scope a Launchpad. If you're still leaving a warehouse, start with the sprint.

Frequently asked questions

What if the pilot is already running in hive_metastore?

Moving the metadata into Unity Catalog is doable. Reconstructing who was meant to see what isn't, because that knowledge lives in the people who ran the notebooks, so I'd write it down now while they're still around.

Does the six-week date hold if credentials and source-system owners are slow?

The clock moves with them. If credentials and source-system owners take three weeks to appear, that gets said during scoping, so nobody finds out in week four. It also means the date is something you can influence, because naming the owners and getting credentials ready before the start is what keeps it.

Should we rebuild an existing lakehouse with the Launchpad?

Not as a first move. The Launchpad is for standing something up, and for a lakehouse that already exists I'd look at the Health Check first.