# Data engineering > TechFabric builds the data engineering layer on Databricks: ingestion, Lakeflow pipelines, medallion modelling, and the tests and cost controls around them. We take on ETL and data integration estates built on SSIS, Informatica and Data Factory, and we design for idempotent, replayable jobs so a failure at three in the morning recovers rather than restarting. For: Data platform leads whose pipelines break more often than they ship Canonical: https://www.techfabric.com/services/data-engineering --- Most lakehouses do not fail on the hard problems. They fail because nobody owns the boring middle: the job that half-runs at 03:00 and leaves a partial table, the definition two teams both use correctly and cannot share, the cluster left on a default that quietly doubles the bill. We build the ingestion, the Lakeflow jobs and the modelled layers underneath, with the medallion boundaries drawn on purpose rather than inherited from a tutorial. Where a job must not half-complete we put Temporal under it, because a retry that replays from the start is not a recovery. A good share of this work arrives as an ETL estate somebody else built. SSIS packages, Informatica mappings, Azure Data Factory and Synapse pipelines, some of them ten years old and none of them documented. We move those onto Lakeflow pipelines and Lakeflow Jobs through TechFabric Airlift, which composes Databricks Lakebridge, carries executable playbooks for twenty-seven source platforms, and wraps every conversion in a signed certificate and a cutover you can reverse. It starts with the two-week Migration Readiness Sprint at /databricks/migration-readiness, so the scope is written down before anyone commits. ## What it includes - Ingestion and Lakeflow jobs with schema evolution handled rather than hoped for - Bronze, silver and gold boundaries drawn deliberately, with the grain written down - Idempotent, replayable jobs, so a failed run is recoverable instead of a restart from zero - SSIS, Informatica and Data Factory pipelines converted with a record of what changed, so a reviewer checks the translation rather than trusts it - Data integration across the estate, so the ERP, the CRM and the lakehouse agree on one customer and one order - Cost sitting in the design: cluster policies, file sizing, and the queries that actually pay ## Questions ### Where does data engineering stop and governance start? In practice they overlap, and pretending otherwise is how grants get retrofitted. We design the catalogue and the schema shape at the same time as the pipeline, so permissions and lineage are a property of the model rather than something added once the tables already exist. The separate governance line at /services/data-ai-governance exists for the cases where the estate is already built and the problem is defending it. ### Can you take over pipelines somebody else wrote? Yes, and it is common. The first work is usually reading what is actually running rather than what the documentation claims: which jobs matter, which tables nobody has queried in a year, and where the grain silently changes. That inventory is the same one the Migration Readiness Sprint at /databricks/migration-readiness produces, and it is worth having before anyone proposes a rewrite. ### Do you do ETL development and data integration? Yes. It's most of what data engineering is once the slides are put away. Ingestion from the systems the business runs on, transformation into tables people can query, and integration so the ERP, the CRM and the lakehouse describe the same customer and the same order. On Databricks that's Lakeflow pipelines and Lakeflow Jobs, with Temporal underneath the flows that cross systems and must not half-complete. ### Can you move SSIS, Informatica or Data Factory pipelines to Databricks? Yes, and those three plus Synapse pipelines are the estates we see most. The conversion runs through TechFabric Airlift, which composes Databricks Lakebridge and adds what Lakebridge doesn't do: a record of every change so a reviewer can check the translation, a signed certificate per object, and a cutover with a tested way back. The Migration Readiness Sprint comes first, so you know the package count and the ones that will fight before you commit. ### Should we hire ETL developers or bring in a data engineering consultancy? If you need one more pair of hands on a pipeline your team already owns, hire. If the pipelines are the problem, you want a team that has shipped together and that builds the application and agent layer above the tables from the same people, because a platform nobody builds on doesn't get used. We do both, and either way the engagement is scoped so you can judge us on something small before anything larger. ### Do you do data engineering off Databricks? Yes. Databricks is where 80 of our engineers are certified and where we've gone deepest, and where a system has to reach past the workspace we build on Azure, AWS, Google Cloud and Cloudflare too. What decides the design isn't the platform. It's whether the numbers have to reconcile and who has to be able to defend them. ### How do you keep Databricks costs from running away? By treating cost as a design constraint rather than a monthly surprise. Cluster policies, sensible file sizes, and knowing which queries are actually paying for themselves. The two-week Databricks Health Check at /databricks/health-check is the named engagement when the bill is the question. ### Do we need Temporal for this, or is Lakeflow enough? Lakeflow is enough for most pipelines. Temporal earns its place when a process spans systems and must not half-complete, or when a human approval sits in the middle of it and the workflow has to survive the wait. We are a Temporal partner and we use it where that property is worth the extra moving part, not everywhere.