# Databricks data engineering

> We build and fix data engineering on Databricks so the nightly loads finish, a bad row stops before it reaches a dashboard, and anyone can trace a number back to the system it came from. That's Lakeflow underneath: Connect and Auto Loader to land data, Lakeflow pipelines (once Delta Live Tables) with data quality rules, Unity Catalog lineage, and Lakeflow Jobs to run it.

For: Data and analytics leaders running pipelines on Databricks, or moving onto it
Canonical: https://www.techfabric.com/databricks/data-engineering

---

Lakeflow pipelines your own team can run and explain

Your pipelines mostly run. One job half-finishes most Tuesdays, the silver tables get rebuilt by hand when a source adds a column, and two dashboards disagree because nobody wrote down the grain. The usual first step is a two-week Health Check that reads the estate as it stands and ends in a written scorecard.

## What we look at

- Which jobs matter, which run on all-purpose compute, and which nobody has opened in a year
- Whether each source can be read incrementally, or has to be read in full every time
- Where bad rows go today, and whether anyone finds out before a report does
- The grain of every table a dashboard reads from
- Whether lineage resolves from the dashboard back to the source system
- What last month's bill paid for, from system.billing.usage

## What you get

- Ingestion through Lakeflow Connect where a managed connector covers the source, and Auto Loader for files
- Lakeflow pipelines with an expectation on every silver table, written as in the SQL below
- Changes from operational databases applied with AUTO CDC, keeping history where somebody needs it
- Medallion tables in Unity Catalog, with the grain of each gold table written down
- Lakeflow Jobs triggered on a schedule, a file arriving or a table updating, each with retries and an owner
- Lineage and cost queries your team can run when somebody asks

## Which Lakeflow piece does which job

Lakeflow is Databricks' name for the data engineering half of the platform. It's four products, with Auto Loader working beside them, and people tend to blur them together.

Lakeflow Connect is the managed connectors for SaaS apps and databases, each at its own release stage.

Auto Loader is for files. It notices new ones in cloud storage, adds a column when a source introduces one (the stream stops once and picks it up on restart), and keeps values that don't fit the schema in `_rescued_data`. For most workloads the docs now recommend its file events mode over plain directory listing.

Lakeflow pipelines is what Delta Live Tables became: declarative streaming tables and materialized views, with expectations for data quality and AUTO CDC for change feeds. Lakeflow Jobs runs everything on a schedule, on a file arriving, on a table updating or continuously. Lakeflow Designer, a visual canvas that writes code you can keep in Git, has been generally available since June 2026, and we'd use it where an analyst owns the logic.

Checked on 12 October 2026 against: [Lakeflow Connect](https://docs.databricks.com/aws/en/ingestion/lakeflow-connect/), [Auto Loader file detection modes](https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/file-detection-modes), [where Delta Live Tables went](https://docs.databricks.com/aws/en/ldp/where-is-dlt), [Lakeflow Jobs](https://docs.databricks.com/aws/en/jobs/), [Lakeflow Designer](https://docs.databricks.com/aws/en/designer/what-is-lakeflow-designer), [Designer's general availability](https://docs.databricks.com/aws/en/release-notes/product/2026/june#lakeflow-designer-is-now-generally-available).

## A silver table with its rules written in

This is the shape of most silver tables we write. The bronze table streams files from a Unity Catalog volume.

The silver table drops a row with no order ID, because a downstream join can't use it, and stops the whole update on a negative amount, because that means the source changed meaning and somebody needs to know today. A constraint with no ON VIOLATION clause records the failure and keeps the row, which is the right choice while a rule is new.

The second half applies a change feed with AUTO CDC as slowly changing dimension type 2, so the customer table keeps every version and a report can ask what an account looked like in March.

```
CREATE OR REFRESH STREAMING TABLE orders_bronze
AS SELECT * FROM STREAM read_files('/Volumes/raw/erp/orders/', format => 'json');

CREATE OR REFRESH STREAMING TABLE orders_silver (
  CONSTRAINT order_id_present EXPECT (order_id IS NOT NULL) ON VIOLATION DROP ROW,
  CONSTRAINT amount_not_negative EXPECT (amount >= 0) ON VIOLATION FAIL UPDATE
)
AS SELECT * FROM STREAM(orders_bronze);

CREATE OR REFRESH STREAMING TABLE customers;

CREATE FLOW customers_cdc AS AUTO CDC INTO customers
FROM STREAM(customers_changes)
KEYS (customer_id)
SEQUENCE BY updated_at
STORED AS SCD TYPE 2;
```

Lakeflow pipelines SQL. Syntax checked against the CREATE STREAMING TABLE and AUTO CDC references on 12 October 2026.

Checked on 12 October 2026 against: [CREATE STREAMING TABLE](https://docs.databricks.com/aws/en/ldp/developer/ldp-sql-ref-create-streaming-table), [expectations](https://docs.databricks.com/aws/en/ldp/expectations), [AUTO CDC INTO](https://docs.databricks.com/aws/en/ldp/developer/ldp-sql-ref-apply-changes-into).

## Questions

### What is Lakeflow?

Databricks' name for its data engineering products together: Lakeflow Connect for managed ingestion from SaaS apps and databases, Lakeflow pipelines for declarative transformation with data quality rules, Lakeflow Jobs for orchestration, and Lakeflow Designer, a visual canvas, generally available since June 2026. We build on all four, and most of the engineering sits in the pipelines and the jobs.

### Is Delta Live Tables the same as Lakeflow pipelines?

Yes, renamed twice. Delta Live Tables became Lakeflow Declarative Pipelines in June 2025, then Lakeflow Spark Declarative Pipelines in November 2025, and the documentation now just says Lakeflow pipelines. The SQL on this page is the current syntax, AUTO CDC included.

### When do you use Lakeflow Connect, and when do you build the pipeline?

We use a managed connector when it covers the source and the fields we need, because then the ingestion is setup rather than code somebody has to own. We build when there's no connector, as with Business Central, when the connector doesn't expose a table we need, or when the source only drops files, which is Auto Loader's job.

### How do you handle data quality in Databricks pipelines?

With expectations in the pipeline definition, one per rule, as in the SQL above. For rules that keep or drop a failing row, the pass and fail counts land in the pipeline's event log, so a quality report is a query. A rule that fails the update records no counts; it stops the run, and the failed update is the alert. We start a new rule as record-only, watch it for a few runs, and tighten it once we know what it catches.

### How do you show where a number came from?

From the lineage Unity Catalog records as jobs run, as long as every job reads tables by name. A job that reads a table by its storage path loses column lineage, which is the gap we find most often, so we name every table and hand over the lineage query with the pipelines.