Lakeflow, source to gold table
Lakeflow pipelines your own team can run and explain
We build and fix data engineering on Databricks so the nightly loads finish, a bad row stops before it reaches a dashboard, and anyone can trace a number back to the system it came from.
That's Lakeflow underneath: Connect and Auto Loader to land data, Lakeflow pipelines (once Delta Live Tables) with data quality rules, Unity Catalog lineage, and Lakeflow Jobs to run it.
- Who it is for
- Data and analytics leaders running pipelines on Databricks, or moving onto it
- Bench
- 115+ engineers, 80 Databricks-certified
Your pipelines mostly run. One job half-finishes most Tuesdays, the silver tables get rebuilt by hand when a source adds a column, and two dashboards disagree because nobody wrote down the grain. The usual first step is a two-week Health Check that reads the estate as it stands and ends in a written scorecard.
What we look at first
- Which jobs matter, which run on all-purpose compute, and which nobody has opened in a year
- Whether each source can be read incrementally, or has to be read in full every time
- Where bad rows go today, and whether anyone finds out before a report does
- The grain of every table a dashboard reads from
- Whether lineage resolves from the dashboard back to the source system
- What last month's bill paid for, from system.billing.usage
What we build
- Ingestion through Lakeflow Connect where a managed connector covers the source, and Auto Loader for files
- Lakeflow pipelines with an expectation on every silver table, written as in the SQL below
- Changes from operational databases applied with AUTO CDC, keeping history where somebody needs it
- Medallion tables in Unity Catalog, with the grain of each gold table written down
- Lakeflow Jobs triggered on a schedule, a file arriving or a table updating, each with retries and an owner
- Lineage and cost queries your team can run when somebody asks
Which Lakeflow piece does which job
Lakeflow is Databricks' name for the data engineering half of the platform. It's four products, with Auto Loader working beside them, and people tend to blur them together.
Lakeflow Connect is the managed connectors for SaaS apps and databases, each at its own release stage.
Auto Loader is for files. It notices new ones in cloud storage, adds a column when a source introduces one (the stream stops once and picks it up on restart), and keeps values that don't fit the schema in _rescued_data. For most workloads the docs now recommend its file events mode over plain directory listing.
Lakeflow pipelines is what Delta Live Tables became: declarative streaming tables and materialized views, with expectations for data quality and AUTO CDC for change feeds. Lakeflow Jobs runs everything on a schedule, on a file arriving, on a table updating or continuously. Lakeflow Designer, a visual canvas that writes code you can keep in Git, has been generally available since June 2026, and we'd use it where an analyst owns the logic.
Checked on 12 October 2026 against Lakeflow Connect, Auto Loader file detection modes, where Delta Live Tables went, Lakeflow Jobs, Lakeflow Designer and Designer's general availability.
A silver table with its rules written in
This is the shape of most silver tables we write. The bronze table streams files from a Unity Catalog volume.
The silver table drops a row with no order ID, because a downstream join can't use it, and stops the whole update on a negative amount, because that means the source changed meaning and somebody needs to know today. A constraint with no ON VIOLATION clause records the failure and keeps the row, which is the right choice while a rule is new.
The second half applies a change feed with AUTO CDC as slowly changing dimension type 2, so the customer table keeps every version and a report can ask what an account looked like in March.
CREATE OR REFRESH STREAMING TABLE orders_bronze
AS SELECT * FROM STREAM read_files('/Volumes/raw/erp/orders/', format => 'json');
CREATE OR REFRESH STREAMING TABLE orders_silver (
CONSTRAINT order_id_present EXPECT (order_id IS NOT NULL) ON VIOLATION DROP ROW,
CONSTRAINT amount_not_negative EXPECT (amount >= 0) ON VIOLATION FAIL UPDATE
)
AS SELECT * FROM STREAM(orders_bronze);
CREATE OR REFRESH STREAMING TABLE customers;
CREATE FLOW customers_cdc AS AUTO CDC INTO customers
FROM STREAM(customers_changes)
KEYS (customer_id)
SEQUENCE BY updated_at
STORED AS SCD TYPE 2;Checked on 12 October 2026 against CREATE STREAMING TABLE, expectations and AUTO CDC INTO.
Issued by Databricks to the engineers
Work we have published on this
Automotive / Fintech
Location Services
The warehouse and ETL conversion off Synapse and SQL Server ran on TechFabric Airlift, each wave with signed certificates and a cutover that could be reversed.
B2B ecommerce
DealerIQ
An operational store moved off Cosmos DB and SQL Server onto Lakebase, beside the lakehouse, with Unity Catalog holding the grants.
Manufacturing
Business Central and six other sources landed and layered as 12 bronze, 15 silver and 10 gold tables, built and tested on sample data before any live system was connected.
What makes it faster
Where the pipelines come from SSIS, Informatica, Data Factory or Synapse, Airlift does the conversion. It composes Databricks Lakebridge, keeps a record of every change so a reviewer checks the translation, signs each converted object, and gives every wave a tested way back.
About TechFabric AirliftWhere this goes next
Data engineering services
How an engagement runs, and the case studies
Data integration
Getting ERP and CRM data in first
Databricks MLOps
What the gold tables feed next
Unity Catalog consulting
Grants and lineage for the tables
Pipelines, in the Field Guide
Lakeflow pipelines chapter and verse
The medallion architecture, in the Field Guide
Bronze, silver and gold, and when to skip one
FAQ
Databricks data engineering: common questions
What is Lakeflow?
Databricks' name for its data engineering products together: Lakeflow Connect for managed ingestion from SaaS apps and databases, Lakeflow pipelines for declarative transformation with data quality rules, Lakeflow Jobs for orchestration, and Lakeflow Designer, a visual canvas, generally available since June 2026. We build on all four, and most of the engineering sits in the pipelines and the jobs.
Is Delta Live Tables the same as Lakeflow pipelines?
Yes, renamed twice. Delta Live Tables became Lakeflow Declarative Pipelines in June 2025, then Lakeflow Spark Declarative Pipelines in November 2025, and the documentation now just says Lakeflow pipelines. The SQL on this page is the current syntax, AUTO CDC included.
When do you use Lakeflow Connect, and when do you build the pipeline?
We use a managed connector when it covers the source and the fields we need, because then the ingestion is setup rather than code somebody has to own. We build when there's no connector, as with Business Central, when the connector doesn't expose a table we need, or when the source only drops files, which is Auto Loader's job.
How do you handle data quality in Databricks pipelines?
With expectations in the pipeline definition, one per rule, as in the SQL above. For rules that keep or drop a failing row, the pass and fail counts land in the pipeline's event log, so a quality report is a query.
A rule that fails the update records no counts; it stops the run, and the failed update is the alert. We start a new rule as record-only, watch it for a few runs, and tighten it once we know what it catches.
How do you show where a number came from?
From the lineage Unity Catalog records as jobs run, as long as every job reads tables by name. A job that reads a table by its storage path loses column lineage, which is the gap we find most often, so we name every table and hand over the lineage query with the pipelines.
The rest of databricks
Databricks consulting
Data and engineering leaders choosing a Databricks partner
Hire Databricks engineers
Engineering and data leaders who need Databricks capacity quickly
Lakebase implementation
Data and product leaders putting an operational workload on Databricks
Unity Catalog consulting
Platform and security leads who have to defend the setup to somebody else
Databricks cost optimization
Data platform leads whose Databricks bill is outgrowing the work
Data integration
Data, finance and operations leaders whose numbers live in five systems
Databricks MLOps
ML leads and heads of data science whose models stop at the notebook