For Data platform leads whose pipelines break more often than they ship
Data engineering
TechFabric builds the data engineering layer on Databricks: ingestion, Lakeflow pipelines, medallion modelling, and the tests and cost controls around them. We take on ETL and data integration estates built on SSIS, Informatica and Data Factory, and we design for idempotent, replayable jobs so a failure at three in the morning recovers rather than restarting.
Lakehouses fail in the middle
Pipelines that hold, tables people trust, and a bill that stops surprising you.
Most lakehouses do not fail on the hard problems. They fail because nobody owns the boring middle: the job that half-runs at 03:00 and leaves a partial table, the definition two teams both use correctly and cannot share, the cluster left on a default that quietly doubles the bill.
We build the ingestion, the Lakeflow jobs and the modelled layers underneath, with the medallion boundaries drawn on purpose rather than inherited from a tutorial. Where a job must not half-complete we put Temporal under it, because a retry that replays from the start is not a recovery.
A good share of this work arrives as an ETL estate somebody else built. SSIS packages, Informatica mappings, Azure Data Factory and Synapse pipelines, some of them ten years old and none of them documented.
We move those onto Lakeflow pipelines and Lakeflow Jobs through TechFabric Airlift, which composes Databricks Lakebridge, carries executable playbooks for twenty-seven source platforms, and wraps every conversion in a signed certificate and a cutover you can reverse. It starts with the two-week Migration Readiness Sprint at /databricks/migration-readiness, so the scope is written down before anyone commits.
- Ingestion and Lakeflow jobs with schema evolution handled rather than hoped for
- Bronze, silver and gold boundaries drawn deliberately, with the grain written down
- Idempotent, replayable jobs, so a failed run is recoverable instead of a restart from zero
- SSIS, Informatica and Data Factory pipelines converted with a record of what changed, so a reviewer checks the translation rather than trusts it
- Data integration across the estate, so the ERP, the CRM and the lakehouse agree on one customer and one order
- Cost sitting in the design: cluster policies, file sizing, and the queries that actually pay
How an engagement works
01
Talk to an engineer
A real conversation about your initiative with a senior engineer who has built this before. Not a sales call. What you are trying to build, what has been tried, and what is realistic.
02
Discovery and scoping
Two to three weeks to clarify requirements, evaluate where AI fits, and define realistic scope. On AI work this is also where success gets defined precisely enough to score, because a goal nobody can measure cannot be hillclimbed. You get a plan you can act on before committing to a larger engagement.
03
The right team, daily demos
We put the team the work actually needs on it and show you running software every day. Built with the same rigor as any enterprise system: tested, monitored, documented.
04
Production and beyond
Deployed and running under real load, handling real business processes. Ongoing support and team continuity for whatever comes next.
Migrations to Databricks
Off Snowflake, Synapse, Teradata and SQL Server, onto Lakehouse and Lakebase, with a cutover you can reverse.
Data science & AI
Context stores, memory, retrieval and governed agents that survive production.
AI/BI & Genie dashboards
Genie answers a business question in English, and the answer holds up when somebody checks it against finance.
Machine learning
Models that reach an endpoint, get retrained on a schedule, and can be rolled back by somebody who was not there.
Data & AI governance
Unity Catalog designed so grants hold, lineage survives a refactor, and an agent inherits permissions instead of routing around them.
Forward-deployed teams
Product, design and engineering people who sit inside your business, find the real problem, and ship it.
Product development
Full product delivery: multi-tenant architecture, operator consoles and the data layer under them. On Azure, AWS, Google Cloud or Cloudflare, and as a Databricks App where the product belongs next to the lakehouse.
APIs & durable systems
Long-running operations that survive restarts and partial failure. Temporal under the lakehouse jobs, agent runs and approvals that must not half-complete.
FAQ
Data engineering, answered
Where does data engineering stop and governance start?
In practice they overlap, and pretending otherwise is how grants get retrofitted. We design the catalogue and the schema shape at the same time as the pipeline, so permissions and lineage are a property of the model rather than something added once the tables already exist.
The separate governance line at /services/data-ai-governance exists for the cases where the estate is already built and the problem is defending it.
Can you take over pipelines somebody else wrote?
Yes, and it is common. The first work is usually reading what is actually running rather than what the documentation claims: which jobs matter, which tables nobody has queried in a year, and where the grain silently changes.
That inventory is the same one the Migration Readiness Sprint at /databricks/migration-readiness produces, and it is worth having before anyone proposes a rewrite.
Do you do ETL development and data integration?
Yes. It's most of what data engineering is once the slides are put away. Ingestion from the systems the business runs on, transformation into tables people can query, and integration so the ERP, the CRM and the lakehouse describe the same customer and the same order.
On Databricks that's Lakeflow pipelines and Lakeflow Jobs, with Temporal underneath the flows that cross systems and must not half-complete.
Can you move SSIS, Informatica or Data Factory pipelines to Databricks?
Yes, and those three plus Synapse pipelines are the estates we see most. The conversion runs through TechFabric Airlift, which composes Databricks Lakebridge and adds what Lakebridge doesn't do: a record of every change so a reviewer can check the translation, a signed certificate per object, and a cutover with a tested way back.
The Migration Readiness Sprint comes first, so you know the package count and the ones that will fight before you commit.
Should we hire ETL developers or bring in a data engineering consultancy?
If you need one more pair of hands on a pipeline your team already owns, hire. If the pipelines are the problem, you want a team that has shipped together and that builds the application and agent layer above the tables from the same people, because a platform nobody builds on doesn't get used.
We don't do body-shop staffing, and the engagement is scoped so you can judge us on something small before anything larger.
Do you do data engineering off Databricks?
Yes. Databricks is where 80 of our engineers are certified and where we've gone deepest, and where a system has to reach past the workspace we build on Azure, AWS, Google Cloud and Cloudflare too. What decides the design isn't the platform. It's whether the numbers have to reconcile and who has to be able to defend them.
How do you keep Databricks costs from running away?
By treating cost as a design constraint rather than a monthly surprise. Cluster policies, sensible file sizes, and knowing which queries are actually paying for themselves. The two-week Databricks Health Check at /databricks/health-check is the named engagement when the bill is the question.
Do we need Temporal for this, or is Lakeflow enough?
Lakeflow is enough for most pipelines. Temporal earns its place when a process spans systems and must not half-complete, or when a human approval sits in the middle of it and the workflow has to survive the wait. We are a Temporal partner and we use it where that property is worth the extra moving part, not everywhere.