For Data platform leads whose pipelines break more often than they ship
Data engineering
TechFabric builds the data engineering layer on Databricks: ingestion, Lakeflow pipelines, medallion modelling, and the tests and cost controls around them. We take on ETL estates built on SSIS, Informatica and Data Factory, and we design for idempotent, replayable jobs so a failure at three in the morning recovers rather than restarting.
Data still stuck in an ERP or CRM? Data integration into Databricks
Lakehouses fail in the middle
Pipelines that hold, tables people trust, and a bill that stops surprising you.
Most lakehouses do not fail on the hard problems. They fail because nobody owns the boring middle: the job that half-runs at 03:00 and leaves a partial table, the definition two teams both use correctly and cannot share, the cluster left on a default that quietly doubles the bill.
We build the ingestion, the Lakeflow jobs and the modelled layers underneath, with the medallion boundaries drawn on purpose rather than inherited from a tutorial. Where a job must not half-complete we put Temporal under it, because a retry that replays from the start is not a recovery.
A good share of this work arrives as an ETL estate somebody else built. SSIS packages, Informatica mappings, Azure Data Factory and Synapse pipelines, some of them ten years old and none of them documented.
We move those onto Lakeflow pipelines and Lakeflow Jobs through TechFabric Airlift, which composes Databricks Lakebridge, carries executable playbooks for twenty-seven source platforms, and wraps every conversion in a signed certificate and a cutover you can reverse. It starts with the two-week Migration Readiness Sprint at /databricks/migration-readiness, so the scope is written down before anyone commits.
- Ingestion and Lakeflow jobs with schema evolution handled rather than hoped for
- Bronze, silver and gold boundaries drawn deliberately, with the grain written down
- Idempotent, replayable jobs, so a failed run is recoverable instead of a restart from zero
- SSIS, Informatica and Data Factory pipelines converted with a record of what changed, so a reviewer checks the translation rather than trusts it
- Conformed customer and order tables, so every report built on the lakehouse agrees on one customer and one order
- Cost sitting in the design: cluster policies, file sizing, and the queries that actually pay
Work we have published on this
Automotive / Fintech
Location Services
Synapse and SQL Server onto Databricks through TechFabric Airlift, cut over in waves that could each be reversed. The rules engine that validates a repossession assignment on arrival kept its shape in the move and now reads governed lakehouse data.
B2B ecommerce
DealerIQ
Cosmos DB and SQL Server onto Lakebase, converted with Airlift and cut over in reversible waves. The sales intelligence that matches customer equity to cars on the lot now reads from the same governed store as the analytics.
Sports media
BZZR
Click analytics on Databricks for a sports streaming platform, beside a content pipeline that transcribes partner video and mines it for key terms to build a sports graph.
Manufacturing
Leadership reporting for a plant in its first year
Seven sources, Business Central among them, through 12 bronze, 15 silver and 10 gold tables to nine leadership dashboards and a Genie space, built and tested on sample data and not yet connected to the plant's live systems.
How an engagement works
01
Talk to an engineer
A real conversation about your initiative with a senior engineer who has built this before. Not a sales call. What you are trying to build, what has been tried, and what is realistic.
02
Discovery and scoping
Two to three weeks to clarify requirements, evaluate where AI fits, and define realistic scope. On AI work this is also where success gets defined precisely enough to score, because a goal nobody can measure cannot be hillclimbed. You get a plan you can act on before committing to a larger engagement.
03
The right team, daily demos
We put the team the work actually needs on it and show you running software every day. Built with the same rigor as any enterprise system: tested, monitored, documented.
04
Production and beyond
Deployed and running under real load, handling real business processes. Ongoing support and team continuity for whatever comes next.
How the pipelines are laid out
Data comes in one of three ways. Salesforce, Workday and SQL Server have managed Lakeflow Connect connectors, so landing them is mostly setup. Business Central has none, so we build that pipeline and somebody owns it. Files dropped into cloud storage go through Auto Loader. When a source adds a column, the stream stops once and picks it up on restart, and values that don't fit the schema go into a _rescued_data column instead of being lost.
From bronze on, the work is Lakeflow pipelines. Each silver table carries expectations, and each expectation says what a bad row does: expect records it and keeps going, expect_or_drop drops the row, expect_or_fail stops the update. That choice is the data quality policy, written where the pipeline runs.
Changes from operational databases are applied with AUTO CDC, as type 1 when only the latest value matters or type 2 when somebody needs the history.
Lakeflow Jobs runs the whole thing on a schedule, when a file arrives, or when an upstream table updates. Unity Catalog records table and column lineage as the jobs run, and system.access.table_lineage keeps a rolling year of it, which is what answers an auditor asking where a number came from.
The Lakeflow products one by one, with a pipeline written out in SQL, are on Databricks data engineering.
Checked on 12 October 2026 against Lakeflow pipelines, pipeline expectations, AUTO CDC, Auto Loader schema evolution, Lakeflow Jobs triggers, Unity Catalog lineage and the medallion architecture.
Four ways a data engineering engagement runs
01
Two weeks reading what's actually running
For an estate already on the platform, the Databricks Health Check reads cost, job reliability and Unity Catalog coverage and ends in a ranked scorecard. Before a move, the Migration Readiness Sprint inventories the estate, which jobs matter and which tables nobody has queried in a year, and ends in a wave plan. Both are two weeks at a fixed fee.
02
Six weeks to a governed foundation
The Lakehouse Launchpad, for a team starting fresh or stalled after a pilot, is a six-week build after two to three weeks of discovery. Unity Catalog first, ingestion through Lakeflow or Jobs, bronze, silver and gold for the domains you named on day one, dashboards for the people who asked, and runbooks your team can use without us.
03
A migration in waves you can reverse
SSIS, Informatica, Data Factory and Synapse estates move through TechFabric Airlift, which composes Databricks Lakebridge and adds a signed certificate per converted object and a tested way back for every wave. Location Services and DealerIQ, now TotalLoop both moved this way.
04
A team on your backlog
Engineers who take the pipeline work your own team can't get to, in your workspace and your repository. A team with a lead who owns the outcome, or individual engineers under your lead, as forward-deployed teams describes.
Questions to ask of your own pipelines
Most of what we find in the first two weeks comes from asking these.
Data quality and cost
- When a bad load arrives, does it stop at silver, or reach a dashboard first? The expectation on each silver table decides that, if one exists.
- Where did the column the source added last month go? Auto Loader adds it to the schema when the stream restarts and keeps values that don't fit in
_rescued_data; a hand-written job often drops both without a word. - Can somebody state the grain of every gold table in one sentence?
- How much of last month's spend was all-purpose compute?
system.billing.usagegrouped bysku_nameanswers it. - When a job fails at three in the morning, does the rerun pick up where it stopped, or start again and write twice?
- When a source stops sending, who's told, and how long before a report goes quiet?
- After a table is rebuilt, does its lineage still resolve in
system.access.table_lineage?
Checked on 12 October 2026 against billing usage system table.
Migrations to Databricks
Off Snowflake, Synapse, Teradata and SQL Server, onto Lakehouse and Lakebase, with a cutover you can reverse.
Data science & AI
Context stores, memory, retrieval and governed agents that survive production.
AI/BI & Genie dashboards
Genie answers a business question in English, and the answer holds up when somebody checks it against finance.
Machine learning
Models that reach an endpoint, get retrained on a schedule, and can be rolled back by somebody who was not there.
Data & AI governance
Unity Catalog designed so grants hold, lineage survives a refactor, and an agent inherits permissions instead of routing around them.
Forward-deployed teams
Product, design and engineering people who sit inside your business, find the real problem, and ship it.
Product development
Full product delivery: multi-tenant architecture, operator consoles and the data layer under them. On Azure, AWS, Google Cloud or Cloudflare, and as a Databricks App where the product belongs next to the lakehouse.
APIs & durable systems
Long-running operations that survive restarts and partial failure. Temporal under the lakehouse jobs, agent runs and approvals that must not half-complete.
FAQ
Data engineering, answered
Where does data engineering stop and governance start?
In practice they overlap, and pretending otherwise is how grants get retrofitted. We design the catalogue and the schema shape at the same time as the pipeline, so permissions and lineage are a property of the model rather than something added once the tables already exist.
The separate governance line at /services/data-ai-governance exists for the cases where the estate is already built and the problem is defending it.
Can you take over pipelines somebody else wrote?
Yes, and it is common. The first work is usually reading what is actually running rather than what the documentation claims: which jobs matter, which tables nobody has queried in a year, and where the grain silently changes.
That inventory is the same one the Migration Readiness Sprint at /databricks/migration-readiness produces, and it is worth having before anyone proposes a rewrite.
Do you do ETL development?
Yes. It's most of what data engineering is once the slides are put away: ingestion, transformation into tables people can query, and the jobs that keep both running.
On Databricks that's Lakeflow pipelines and Lakeflow Jobs, with Temporal underneath the flows that cross systems and must not half-complete. Getting data out of an ERP or a CRM in the first place, Business Central included, has its own page at /databricks/data-integration.
Can you move SSIS, Informatica or Data Factory pipelines to Databricks?
Yes, and those three plus Synapse pipelines are the estates we see most. The conversion runs through TechFabric Airlift, which composes Databricks Lakebridge and adds what Lakebridge doesn't do: a record of every change so a reviewer can check the translation, a signed certificate per object, and a cutover with a tested way back.
The Migration Readiness Sprint comes first, so you know the package count and the ones that will fight before you commit.
Should we hire ETL developers or bring in a data engineering consultancy?
If you need one more pair of hands on a pipeline your team already owns, hire. If the pipelines are the problem, you want a team that has shipped together and that builds the application and agent layer above the tables from the same people, because a platform nobody builds on doesn't get used.
We do both, and either way the engagement is scoped so you can judge us on something small before anything larger.
Do you do data engineering off Databricks?
Yes. Databricks is where 80 of our engineers are certified and where we've gone deepest, and where a system has to reach past the workspace we build on Azure, AWS, Google Cloud and Cloudflare too. What decides the design isn't the platform. It's whether the numbers have to reconcile and who has to be able to defend them.
How do you keep Databricks costs from running away?
By treating cost as a design constraint rather than a monthly surprise. Cluster policies, sensible file sizes, and knowing which queries are actually paying for themselves. The two-week Databricks Health Check at /databricks/health-check is the named engagement when the bill is the question.
Do we need Temporal for this, or is Lakeflow enough?
Lakeflow is enough for most pipelines. Temporal earns its place when a process spans systems and must not half-complete, or when a human approval sits in the middle of it and the workflow has to survive the wait. We are a Temporal partner and we use it where that property is worth the extra moving part, not everywhere.
What do data engineering consulting services include?
The pipelines, and the parts around them that decide whether anyone trusts what comes out. Most buyers expect ingestion and Lakeflow pipelines.
Fewer expect the grain of every gold table to be written down, or data quality expectations that stop a bad load before it reaches a dashboard, and those two are where most arguments between reports start. Where an old ETL estate is in the way, moving it is part of the job.
What is a medallion architecture, and do we need one?
It's the layering Databricks recommends: bronze keeps data as it arrived, raw and unvalidated, silver is the same data validated, cleaned and joined, and gold is business tables shaped for one use, often aggregated, for reporting and machine learning.
You don't need three layers for every source, and we'll say so when a table can go straight to silver. What you do need is to know the grain of every gold table, because that's the line two reports disagree on.
Is Delta Live Tables still a product?
Under another name. Delta Live Tables became Lakeflow Declarative Pipelines in June 2025, and the current Databricks documentation calls it Lakeflow pipelines.
Existing pipelines keep running and the old dlt Python module still imports, so nothing has to be rewritten for the rename. APPLY CHANGES is now AUTO CDC, with the same syntax. We write new pipelines against the current names.
How long does a data engineering engagement take?
The first piece of work is short on purpose. A Databricks Health Check or a Migration Readiness Sprint is two weeks at a fixed fee and ends in a written document.
The Lakehouse Launchpad is a six-week build to a governed foundation, after two to three weeks of discovery, with layers for the domains you named. A migration runs in waves after that, each one with its own cutover and a way back, so the length follows the inventory rather than a guess.