Skip to content
TechFabric

Nobody wanted to run the migration on a Friday

Raja Vemuri5 min read

Most teams I have worked with have a migration sitting in review right now that nobody wants to be the one to merge.

It is usually small. An index, a dropped column, a type change. The reason it sits there is that staging has a few thousand rows and production has tens of millions, and the only honest thing anyone can say in review is that it looks fine. So it waits for a window, and the windows are Tuesday mornings, and everything else is also waiting for Tuesday morning.

The cost of that is not the delay. It is that people start batching. If a change has to wait for a window anyway, you may as well put three changes in the window, and now the thing you are shipping is bigger and harder to reason about than any of its parts. I have watched a two-line migration become a six-table release for exactly this reason and everyone involved thought they were being careful.

What branching changes

Lakebase is Postgres, so your application code does not change and the things you know still apply. Storage and compute are separate underneath, and the storage is copy-on-write and versioned. That means a branch is a pointer rather than a copy. You get production data in about a second, it costs almost nothing to hold, and you delete it when you are done.

The comparison people reach for is a snapshot restore, and it is worth being clear about why that is different. A restore on RDS or Cloud SQL provisions an instance and rehydrates storage, which is minutes to hours, and costs a second full copy of your data the whole time it exists. Fine for recovery. You are never going to run it on every pull request.

Once branching is there, CI can create a branch, run the migration against real volume, time it, run the tests, and throw the branch away. The pull request carries a number. Nine minutes of lock time turns up in review instead of in an incident channel at eleven at night.

Each engineer also gets their own copy, which quietly removes the other thing that slows teams down. Nobody is scheduling around the shared staging database, because there is no shared staging database.

What it does not do

Branching does not make a bad migration good. If the schema change loses data it will lose data on a branch too. You find out on a Tuesday afternoon rather than during a release, which is worth a lot, and it is a different thing from safety.

Knowing a migration takes nine minutes is also the start of the work rather than the end of it. You still have to decide whether that is a concurrent index build, a batched backfill, or a rethink. Same engineering as before.

And I would not move a system of record onto Lakebase this quarter. We have run it and I like it. It has not been generally available long enough for me to tell you what it does at three in the morning in year two, and anyone who says otherwise is guessing.

The thing I did not expect was the effect on appetite. When trying an index strategy costs a second and a deletion, people try four. The migration queue drains because the reason it existed went away, and nobody made a decision about it.