Skip to content
TechFabric

Unity Catalog before the first pipeline

Troy Busot8 min read

The pilot had a bronze layer, a dashboard, and a Genie space that demoed well on a Thursday. It did not have Unity Catalog.

Someone had created the schemas in hive_metastore because that was the default in the workspace they were given. The pipeline wrote a Delta table, the graph looked good on a slide, and they kept going. Six weeks later security asked who could see the customer table. The honest answer was anyone with workspace access. That is an answer that ends a rollout.

I have the same conversation about agents. Draw the permissions boundary before you give anything a service principal, because once an agent can reach production data without a grant model you can defend, you have an incident with a date on it. For a lakehouse the version is shorter and I still have to say it every time: grants, then data.

Why people skip it

Unity Catalog looks like ceremony when you are trying to prove the platform. A workshop cluster, a sample dataset, a notebook that lands one table, and you can have a graph on a slide by Friday. Putting the catalogue, the schemas, the storage credentials and the grants in place first looks like you have not started, which is the opposite of how a security review will read that week.

I have watched teams retrofit Unity Catalog onto a lakehouse that already had six months of tables in it. Moving the metadata is doable. Reconstructing who was supposed to see what is not, because that knowledge lived in the people who ran the notebooks, and two of them have moved on.

Six weeks, with the scope fixed

The Lakehouse Launchpad is how we stand Databricks up when the job is a foundation. Six weeks of identity and environment strategy, Unity Catalog structure, ingestion through Lakeflow or Jobs, bronze silver and gold for the domains you named at the start, dashboards people asked for, a Genie space with real definitions, and runbooks your team can use.

The date holds if access holds. If credentials and source-system owners take three weeks to appear, the clock reflects that, and we say so during scoping. If the discovery before the build shows six weeks is not honest for the estate, we rescope before anyone starts.

Where the Launchpad replaces a warehouse, Fabric Airlift does the migration underneath: profiling, conversion, reconciliation, a reversible cutover. Where the output is an application in the workspace, Fabric Runway is how that app gets preview environments and a gate on promotion.

The thing that actually stops a handover

I used to worry about slow pipelines. The conversation that kills a programme is "the only person who understands the grants is on holiday." A catalogue that matches how the teams actually work, with lineage intact as data moves, is what lets us leave. The runbooks are a deliverable for that reason. Plenty of clients keep us on for the next programme. That should be a decision, not a dependency created by a hive_metastore nobody documented.

If you already have a lakehouse and you are wondering whether to rebuild it, look at the Health Check first. The Launchpad is for standing something up.

Week one

We will not land a sample dataset in a demo schema and call it a platform, or wire Genie to tables whose definitions still live in a Slack thread, or grant a service principal ALL PRIVILEGES so the first job runs.

I am usually in the first of those conversations when the work includes an agent. Even when it does not, the first artefact is the same: a catalogue structure you can defend, and grants that match the organisation you actually have.

If that is the job, scope a Launchpad. If you are still leaving a warehouse, start with the sprint.