Building applications on Lakebase: Postgres, Databricks Apps and AI in one place

The first Gilbert Databricks User Group session, at the TechFabric office, built an application on Lakebase live from a script.

Every number on the slides was measured in the room that evening. Branching the whole database reached READY in 4.5 seconds. A cold query cost 2,979 ms against 1,650 ms warm, a synced table came online 51 seconds after the call, and writes from the app reached Delta inside 10 to 15 seconds.

Nobody typed a database password all evening. This project sets enable_pg_native_login to false, so there isn't one to type, and a person at psql and an application holding client credentials both arrive as the same one-hour OAuth token.

Lakebase is managed Postgres inside a Databricks project, with lakehouse tables synced into it and the operational data sitting beside the analytics under one set of permissions. That collapses a stack most teams run as three systems and two copy jobs. We built an application on it live, from a script, and worked through four parts of it: signing in with Databricks identity, branching and recovery, a Data API straight over Postgres, and moving data between the lakehouse and the application in both directions.

PostgreSQL with Databricks identity

The schema is ordinary Postgres, foreign keys and check constraints and all, and every client we pointed at it connected unmodified. Postgres reads the token as a role, so GRANT decides the rest exactly as it always has, and the credential you would normally be rotating and eventually leaking isn't there to manage.

Scale-to-zero is the other thing people want a number for. Cold, the first query took 2,979 ms. Warm it was 1,650 ms, so roughly 1.3 seconds of that is the endpoint waking up, and five idle minutes later it goes back to zero on its own.

Safe branching and recovery

create-branch came back READY in 4.5 seconds with its own read-write endpoint and logical_size_bytes at zero. Pages are shared copy-on-write with the parent, so a branch costs you only what you write into it.

Then we broke things on it. One fingerprint query run against both branches showed a schema change, a price change and a DELETE that production never saw. Recovery is the same trick pointed at a clock. source_branch_time snaps to the last committed write at or before the moment you name, source_branch_lsn is the exact version of the same idea, and we used it on an UPDATE that had lost its WHERE. Repairing from there is ordinary Postgres, a COPY out and an UPDATE ... FROM back in.

A secure Data API

REST straight over Postgres, with nothing to build or host in between. The only server-side code we wrote was the token exchange, so the browser never held the secret. Filters and ordering ride in as query parameters, foreign keys come back as embedded objects with no join written anywhere, and responses ran 384 to 542 ms.

Every request runs as whoever is calling, which makes GRANT and row-level security the entire authorization model. One policy on demo_app.orders gave three callers three different answers. The owner saw every status over psql, the service principal got an empty array where it wasn't allowed to read, and posting a cancelled order came back 403 with 42501, because a policy that would hide a row won't let you create it either.

Lakehouse and application data together

postgres create-catalog took 5.7 seconds and turns the database into a Unity Catalog catalog, after which a SQL warehouse reads live Postgres rows in place with no copy anywhere. Where we wanted a curated Delta table instead, create-synced-table in SNAPSHOT mode came online 51 seconds after the call and re-snapshotting runs about 33 seconds.

Traffic goes the other way too. Writes from the application reach Delta through the change data feed in 10 to 15 seconds with the transaction id on every row, and the synced copy lives in its own lakehouse schema so the feed on demo_app doesn't replicate straight back out again.

Where the seams are

history_retention_duration was seven days here, and past that there's no moment left to branch from, so it's worth settling on the number before somebody needs it. The repair we ran assumes nothing else is writing, which is true of a demo and won't be true of your production database. REPLICA IDENTITY FULL has to be set before you enable the change data feed, because a table without it gets skipped, and flipping it afterwards isn't reliably picked up.

Who was in the room

Application developers, data engineers and architects from around the Phoenix metro, a good number with laptops open, following the script as it ran. Some are already on Databricks and some are still deciding. Questions ran about half an hour at the end.

The group meets at the TechFabric office in Gilbert.