# Databricks Data Quality Gates: From Failed Checks to Blocked Releases

> Turn Databricks quality checks into release gates. Validate a fixed candidate, block publication on failure, and preserve the previous approved data.

Published: 2026-10-19
Author: Igor Seleznov
Tags: Databricks, Knowledge Share
Canonical: https://www.techfabric.com/blog/databricks-data-quality-gates

---

![A closed amber inspection gate holds back a new batch of data while previously approved blue data remains available.](/blog-media/databricks-data-quality-gates/hero-4.png)

A failed data quality check should prevent rejected data from reaching consumers. In Databricks, that takes an explicit release boundary, where you validate a fixed candidate, fail on a critical violation, and publish only after validation succeeds.

Here, a **release** means making new data available to a reporting table or application.

## A failed expectation has a scope

Lakeflow expectations offer three actions. `warn` retains invalid rows and records metrics, `drop` removes them, and `fail` stops the offending flow. Choose from the data contract. Dropping an optional event may be acceptable; dropping an order can leave a financial total incomplete.

In triggered mode, a failed expectation rolls back that flow's update while other flows can continue independently. Continuous mode also stops dependent flows. Neither behavior turns a separate validation table into a general release gate. See [expectation behavior](https://docs.databricks.com/aws/en/ldp/expectations).

Databricks warns that reading a validation table does not make downstream datasets wait for its result. Its recommended pattern puts validation and downstream processing in separate pipelines, connected through job tasks. See [validation tables and pipeline control flow](https://docs.databricks.com/aws/en/ldp/expectation-patterns#validation-tables-and-pipeline-control-flow).

```mermaid
flowchart LR
%% caption: Reading a validation table does not make downstream datasets wait for its result. The recommended pattern puts validation and downstream processing in separate pipelines, connected through job tasks.
  subgraph read["Reading a validation table"]
    A["Validation table"] -.-> B["Downstream datasets do not wait for its result"]
  end
  subgraph tasks["Separate pipelines, connected through job tasks"]
    C["Validation"] --> D["Downstream processing"]
  end
```

## Define the blocking rules

Consider four synthetic orders totaling 100,000 cents, with an independent source manifest that supplies the expected count and total.

| Contract | Injected defect |
|---|---|
| IDs are required and unique | Replace an ID with an existing ID |
| Amounts are positive and at most 100,000 cents | Set an amount to NULL |
| Row count matches the manifest | Remove an order |
| Total matches the manifest | Change an amount |

Every violation blocks publication. Count and total reconciliation fit this example because the transformation preserves orders, while filtering and aggregation need different expectations. These checks cannot prove every field is correct, because matching counts don't prove identity and equal totals can hide offsetting errors.

Aggregate checks belong in a validation dataset. This Lakeflow SQL rejects duplicate or missing IDs.

```sql
CREATE OR REFRESH MATERIALIZED VIEW release_checks (
  CONSTRAINT unique_required_ids EXPECT (
    actual_rows = distinct_ids
  ) ON VIOLATION FAIL UPDATE
) AS
SELECT COUNT(*) AS actual_rows,
       COUNT(DISTINCT order_id) AS distinct_ids
FROM dq_gate_demo.staging.orders_r101;
```

`COUNT(DISTINCT order_id)` excludes NULL, so missing IDs also break equality. An empty batch passes this rule; count and total checks against the source manifest reject it.

## Connect validation to publication

![Two pipeline tasks validate and publish the same frozen candidate. A failed validation leaves the previous release available.](/blog-media/databricks-data-quality-gates/release-workflow-4.png)

For a triggered or scheduled job, configure two separate pipeline tasks.

1. **Validate** reads the frozen candidate and fails the update on any blocking violation.

2. **Publish** depends directly on Validate, sets **Run if** to **All succeeded**, and reads the same candidate.

If validation fails, publication is marked `Upstream failed`. Validation must actually execute and succeed. Excluded dependencies receive special handling, so skipping validation is not a pass. See [task dependencies](https://docs.databricks.com/aws/en/jobs/run-if). This pattern requires validation to complete; [continuous jobs run pipelines differently](https://docs.databricks.com/aws/en/jobs/tasks/pipeline).

Task order does not freeze data. Materialize a candidate under a release-specific name, make both tasks read it, and prevent changes until publication completes, because a release-specific name alone does not enforce immutability.

Keep consumers on the previous release until validation passes. Restrict publication to the gated workflow, because another schedule or writer can bypass it. Serialize releases so an older approved candidate cannot overwrite a newer one.

## Prove that failure prevents publication

![Local SQLite demonstration: the clean candidate publishes R101; a duplicate ID blocks publication and keeps R100 unchanged.](/blog-media/databricks-data-quality-gates/release-results-3.png)

The companion demonstration runs locally on SQLite, and it doesn't execute Databricks or Lakeflow. Each scenario starts with R100, which holds two orders totaling 90,000 cents.

The clean candidate publishes R101 with four orders totaling 100,000 cents. Replacing one ID with a duplicate leaves both count and total unchanged, but fails uniqueness, so publication doesn't run. The demonstration asserts that R100's exact rows remain unchanged.

Missing-order, NULL-amount and empty-batch cases also block publication, while another case changes the source after validation and confirms that publication reads the frozen candidate.

Repeat the clean and duplicate cases in a Databricks test workspace, and record the candidate identity, rule version, job run ID, named validation error, publication task state, and consumer rows before and after. A permissions failure proves nothing about the quality rule. Failed `fail` updates do not record [expectation metrics](https://docs.databricks.com/aws/en/ldp/expectations#expectation-tracking-metrics); inspect failure details.

The gate is proven when valid data advances, a critical defect prevents publication from running, and consumers retain the previous approved data.

## Frequently asked questions

### Does a `fail` expectation alone stop downstream tables from updating?

No. In triggered mode, a failed expectation rolls back only that flow's update, and other flows can continue independently. Continuous mode also stops dependent flows, but reading a validation table still doesn't make downstream datasets wait for its result. Put validation and publication in separate pipelines, connected through job tasks.

### Why do both tasks read a frozen candidate?

Task order doesn't freeze data. If the source changes after validation, publication could read rows that never passed the checks. Materialize the candidate under a release-specific name, make both tasks read it, and prevent changes until publication completes.

### Do count and total checks prove the data is correct?

No. Matching counts don't prove identity, and equal totals can hide offsetting errors. A duplicate ID leaves both count and total unchanged, so uniqueness needs its own rule.

### Was this gate run on Databricks?

No. The companion demonstration runs locally on SQLite and doesn't execute Databricks or Lakeflow. Repeat the clean and duplicate cases in a Databricks test workspace before relying on the gate.