# From Notebook Assertions to Reliable Databricks CI

> Make Databricks test failures reach CI. Structure repeatable pytest checks, handle empty suites, wait for jobs, and verify that failures block promotion.

Published: 2026-10-22
Author: Ihor Seleznov
Tags: Databricks, Knowledge Share
Canonical: https://www.techfabric.com/blog/from-notebook-assertions-to-reliable-databricks-ci

---

![A red failure signal passes through four gates labelled pytest, Notebook, Job and CI.](/blog-media/from-notebook-assertions-to-reliable-databricks-ci/hero.png)

A Databricks notebook can finish successfully even when pytest reports failed tests. The missing step can be small: a notebook that calls `pytest.main()` and never checks its return code. When that happens, a broken data rule can move into production while every stage of the release pipeline reports green.

Reliable CI needs the whole failure path to work. A failed test must fail the notebook task, the Databricks job and the required CI check before code advances to the next environment.

This extends [our guide to pytest checks inside Databricks](https://www.techfabric.com/blog/pytest-for-databricks-data-quality-checks) with fixed inputs, explicit comparisons and a runner that propagates failure. Five local Python cases verify the pytest handoff and its limits; the Spark test and Databricks/GitHub configuration remain templates requiring workspace validation.

## Move the transformation out of the notebook

Keep orchestration in notebook cells. Put transformation logic in importable Python modules, and pass its inputs explicitly. A test shouldn't depend on a temporary view or variable left by an earlier interactive cell. Databricks [recommends keeping Python functions and their unit tests outside notebooks](https://docs.databricks.com/aws/en/notebooks/testing).

Start with a small layout:

```text
orders_ci/
  transformations.py
  test_runner.py
  notebooks/run_tests.py
  tests/conftest.py
  tests/test_paid_orders.py
  databricks.yml
```

Suppose paid orders may have a zero amount, while NULL and negative amounts must be excluded. Cancelled orders don't belong in the output.

```python
# transformations.py
from pyspark.sql import functions as F

def paid_orders(df):
    return df.filter(
        (F.col("status") == "PAID")
        & (F.col("amount_cents") >= 0)
    ).select("order_id", "amount_cents")
```

The function neither reads nor writes a table. That keeps this test about transformation behavior; table access and publication need separate integration checks.

## Make the expected result independent

Use a tiny, fixed fixture with an expected output written independently of the transformation. Reusing the production filter to build the expected DataFrame would repeat the same mistake on both sides.

Define a session-scoped `spark` fixture in `tests/conftest.py`. In Databricks, `getOrCreate()` can reuse the existing session; local execution needs a compatible Spark and Java environment.

```python
# tests/conftest.py
import pytest
from pyspark.sql import SparkSession

@pytest.fixture(scope="session")
def spark():
    return SparkSession.builder.getOrCreate()
```

```python
# tests/test_paid_orders.py
from pyspark.testing import assertDataFrameEqual
from transformations import paid_orders

def test_paid_orders(spark):
    source = spark.createDataFrame([
        (1, 1250, "PAID"),
        (2, 0, "PAID"),
        (3, None, "PAID"),
        (4, -1, "PAID"),
        (5, 500, "CANCELLED"),
    ], "order_id long, amount_cents long, status string")
    expected = spark.createDataFrame(
        [(1, 1250), (2, 0)],
        "order_id long, amount_cents long",
    )
    assertDataFrameEqual(
        paid_orders(source), expected,
        checkRowOrder=False, rtol=0, atol=0,
        ignoreNullable=False,
    )
```

This example uses the [PySpark 4.0 comparison interface](https://spark.apache.org/docs/4.0.1/api/python/reference/api/pyspark.testing.assertDataFrameEqual.html). It compares values and schema, including nullability, without requiring a row order. Integer cents need exact equality. Floating-point results may need a justified tolerance; ordered output needs an explicit ordering contract.

Pin a Databricks Runtime that supports this PySpark 4.0 comparison API, and install a compatible pytest version as a task dependency. The local probe versions below are evidence of that probe, not a validated Databricks dependency set.

## Turn pytest's result into a task failure

Calling `pytest.main()` from Python [returns an exit code instead of raising SystemExit](https://docs.pytest.org/en/stable/how-to/usage.html#calling-pytest-from-python-code). Printing a failure summary, or catching an `AssertionError` and only logging it, doesn't establish a failing task.

![The same pytest failure produces two process outcomes: ignored return code, exit 0; checked return code, exit 1.](/blog-media/from-notebook-assertions-to-reliable-databricks-ci/false-success.png)

Databricks' notebook example checks the return code with an assertion. A reusable runner can make the handoff explicit with an exception:

```python
# test_runner.py
from pathlib import Path
import sys
import pytest

def run_tests(suite, report):
    sys.dont_write_bytecode = True
    report = Path(report)
    report.parent.mkdir(parents=True, exist_ok=True)
    code = pytest.main([
        str(suite), "-q", "-p", "no:cacheprovider",
        f"--junitxml={report}",
    ])
    if code != 0:
        raise RuntimeError(f"pytest exit code {int(code)}; report: {report}")
```

Don't catch that exception and continue. Code 1 means tests failed; code 5 means [no tests were collected](https://docs.pytest.org/en/stable/reference/exit-codes.html). Both must reject this required suite. A zero exit code still doesn't prove coverage: selected tests can all be skipped, or discovery can miss part of the suite. This minimal runner doesn't reject all-skipped suites; add a policy checking required test names, executed counts and allowed skips before relying on it as a release gate.

Use a fresh job execution for each candidate. Pytest warns that repeated `pytest.main()` calls in one Python process can reuse imported modules and miss file changes.

## Wait for the job, then decide whether code advances

Call the runner with explicit suite and report paths. Write JUnit XML to a location CI can retrieve, such as a permitted Unity Catalog volume. Retrieve diagnostics after failure without clearing the failed status. Startup and permission errors may prevent report creation; an uploaded report alone does not make a check blocking.

Use a Databricks test job for runtime-dependent behavior and integration checks against isolated test data. Give each concurrent run its own writable schema or namespace. Verify access using the job's execution identity, along with the intended reads and writes. A passing DataFrame test doesn't establish Unity Catalog permissions.

For a configured Declarative Automation Bundle with a `ci` target and a **single-task** test job named `quality_tests`, the blocking shell step is:

```bash
set -euo pipefail
databricks bundle validate -t ci
databricks bundle deploy -t ci
databricks bundle run -t ci quality_tests
```

Deploy the candidate commit to the **test target** before running its job. Configuration validation doesn't execute the tests. For a job, `bundle run` waits by default; don't use [`--no-wait`](https://docs.databricks.com/aws/en/dev-tools/cli/bundle-commands#databricks-bundle-run) for this blocking check. An asynchronous wrapper must poll the terminal result and reject anything other than the required success.

Keep the test job's outcome unambiguous. In a multi-task job, a successful cleanup task can leave the run [Succeeded with failures](https://docs.databricks.com/aws/en/jobs/monitor#how-databricks-determines-job-run-status) because job status depends on the final, or leaf, tasks. If you extend the graph, verify that every required test task succeeded; a broad job-success classification is insufficient.

In GitHub Actions, make promotion depend on the quality job with `needs: quality` and require the `databricks-quality` check in repository merge rules. Keep promotion conditional on success; reserve `always()` for diagnostics. Avoid `continue-on-error` on the blocking step. Before enabling [GitHub OIDC federation](https://docs.databricks.com/aws/en/dev-tools/auth/provider-github), configure the service principal and federation policy for the repository and environment.

```mermaid
flowchart LR
%% caption: A failed test must fail the notebook task, the Databricks job and the required CI check before code advances to the next environment.
  A["Failed test"] --> B["Notebook task"] --> C["Databricks job"] --> D["Required CI check"]
```

## Test the failure path deliberately

First confirm the clean fixture passes. Then change `>= 0` to `> 0`: the valid zero-amount order should disappear and the expected-output test should fail. Run a separate empty-suite case to catch broken discovery.

![For PAID orders, changing amount_cents >= 0 to amount_cents > 0 incorrectly removes the valid zero-amount order.](/blog-media/from-notebook-assertions-to-reliable-databricks-ci/zero-amount-boundary.png)

For each case, record the commit, runtime, selected tests, pytest result, job run ID and CI conclusion. A permission error isn't evidence that the zero-amount rule works. Find the intended comparison failure in the report.

| Case | Required outcome |
|---|---|
| Clean fixture | Expected rows match; the quality check succeeds. |
| Zero-amount regression | The named test fails; the notebook task and job fail; CI blocks promotion. |
| No tests collected | Code 5 fails the runner and the required CI check. |

The local probe ran on Python 3.14.5 with pytest 9.0.2. Its five cases isolate the handoff to a fresh Python process:

| Local case | pytest code | Python process code |
|---|---|---|
| Passing test, result checked | 0 | 0 |
| Failing test, result checked | 1 | 1 |
| Empty suite, result checked | 5 | 1 |
| All tests skipped, result checked | 0 | 0 |
| Failing test, result ignored | 1 | 0 |

The final row reproduces the false success. These are local Python results, not Spark, Databricks Jobs or GitHub Actions results. Repeat the full sequence in the test workspace before relying on the release check.

## Frequently asked questions

### Is a notebook assertion enough for CI?

An unhandled assertion can fail execution, but CI must also observe that failure and block promotion. If a notebook launches pytest, check its return code explicitly. Logging an error and continuing hides the failure from the caller.

### Should every test run in Databricks?

No. Run independent Python tests close to the code. Spark tests need a compatible Spark environment, while checks of permissions, deployed resources and workspace behavior need the target Databricks environment. Keep the evidence for those layers separate.

### Does an uploaded JUnit report make a check blocking?

No. The report explains the outcome. The test runner, job result and CI dependency determine whether code can advance. Verify both with a deliberate failure.