A pipeline that breaks will tell you it broke. A pipeline that is starved will not.
It reads an empty directory, writes nothing, marks itself complete, and the run history stays a clean column of green. Whatever depends on it keeps serving the last numbers it had. If nobody downstream happens to know what the number should be doing, this can go on for a while.
I bring this up because most pipeline monitoring answers one question, which is whether the job ran. That question has a satisfying yes or no and it is usually not the one that matters. The useful questions are about what arrived rather than what executed. Did anything land. Was it roughly the volume it normally is. Is the newest row newer than it was yesterday.
None of those look like failures to a scheduler.
Three checks
If a team has nothing, I would put these in before anything else, and I would not build a framework first.
Freshness. Every table anything depends on gets a maximum age, written down in units the business cares about. Six hours for some things, a day for others, longer at month end when a source is legitimately slow. The number matters less than having one somebody can argue with.
Volume. Yesterday's row count against a trailing average, inside a band. Not an exact match, because real data moves and a tight band teaches everyone to ignore the alert within two weeks. You are looking for the source that sent four rows instead of four hundred thousand.
Schema. Something that notices when a column arrives, disappears or changes type, before whatever reads it notices. People push back on this one because it fires when an upstream team adds a field they are proud of. Fire it anyway.
That is most of it. There is a lot more you can do and most teams are not doing these three.
Unity Catalog already knows half of this
If you are on Databricks, some of the work is done and sitting in system tables.
Lineage tells you what reads what, so you do not have to decide by hand which tables matter enough to watch. Audit logs tell you when a table was last written and by which pipeline. Put those together and the alert becomes specific: this table is stale, eleven things read it, two of them are dashboards the leadership team opens on Monday.
Lakeflow expectations cover the row-level version. Constraints declared next to the transformation, dropping bad records or failing the run. Five expectations a team understands beat a quality framework nobody maintains after the person who chose it moves on.
A couple of things I had backwards
For years I thought of this as a testing problem. Write assertions, run them after the load, alert on failures. Which is fine, and it completely misses the case above, because zero rows satisfy every constraint you can write about the shape of a row. The check that catches a starved pipeline is a check about nothing being there, and it did not occur to me for a long time that nothing was a state worth alerting on.
I also used to route all of these the same way. A freshness alert on a daily table that pages someone at two in the morning about a file that will not arrive until nine is an alert people learn to silence, and then they silence the ones that matter too. Most freshness failures are a working-hours problem.
The last one is less obvious and I only started doing it recently: keep the passes, not just the failures. A check that has been quietly succeeding for eight months is telling you something every day. When somebody eventually asks how long the number has been wrong, you want a table you can query rather than an inference from when the alerts started.