We caught a Genie metric change its own benchmark missed
Genie's built-in benchmark re-runs your ground-truth SQL, so a changed metric still scores Good. How a golden set with a definition hash caught it.
Category
Databricks
Pipelines in Lakeflow, governance in Unity Catalog, Postgres in Lakebase, Genie on top, and the migrations that get a team there. Our engineers write down the code, the settings and what surprised them.
69 posts
Closing the AI speed divide with Genie Agents on Databricks
When a decision takes three weeks and a customer's agent takes minutes, the money leaves first. What one Databricks platform and Genie Agents change.
How we block AI releases when MLflow scorers fail
A scorer that crashed on two wrong answers raised our MLflow evaluation score from 0.75 to 1.00. We built a release rule that counts coverage first.
Unity Catalog model names return 404 on /serving-endpoints
A system.ai model name on a Databricks workspace's /serving-endpoints path returns 404 with Unity Gateway enforcement off. Why we route by name, not by URL.
Search that keeps up with a live game and an open loan
We indexed a wine catalog and refinance queues at iLending and Auto Approve. Search can now read a streaming table or a materialized view.
How we test that AUTO CDC keeps the right customer record
A Databricks CDC flow can finish green and still keep a customer's old city or bring a deleted customer back. How we test ordering and deletes before release.
How we catch wrong AI answers before they reach customers
An AI step in a Databricks job can report success while tickets get no category or the wrong one. How we catch both before the results reach the support team.
AI governance for lending and payments agents on Databricks
Lenders ask whether AI agent controls come first or after the build. How AutoFabric puts risk, compliance and team ownership into each action on Databricks.
Real-time loan data with Databricks Zerobus Ingest
Most lenders' fraud checks and examiner answers run on last night's loan data. How Zerobus Ingest gets it there in seconds, and where Kafka still fits.
One grant model for tables, files and models in Unity Catalog
Security reviews stall when nobody can say who reads which table, file or model. How Unity Catalog put all three under one set of grants at Location Services.
Long-running AI agents in production with Temporal and Databricks
An agent that works for ninety seconds in a demo meets rate limits, crashes, deploys and overnight waits in production. How Temporal and Databricks handle each.
Databricks CustomerLake: a customer data platform inside the lakehouse
CustomerLake builds Customer 360 profiles and audiences on data already in Databricks. What's in Beta, and the workloads we'd move to it first.
How we built policy history that survives backdated endorsements
A backdated endorsement can erase what an insurer's data showed before it. We built a Databricks pipeline that keeps both answers for audits and claims.
How we got structured output from Claude on Databricks
Claude Sonnet and Opus 5.5 on Databricks return HTTP 400 for a response_format request. Why we moved Fabric's schemas into the prompt, and what that costs.
Catching a false success in Databricks notebook CI
A test can catch a broken data rule and the release still goes out green. We reproduced that false success and built a runner that stops it.
Databricks and Temporal for durable data and AI systems
When a process stops halfway, the cost shows up weeks later. How we keep business processes and AI agents finishing their work with Temporal and Databricks.
Backend engineering habits on a Databricks lakehouse
A retried Databricks job can double-count a report without failing anything. The backend habits that prevent it, and how our test kit fails jobs on purpose.
How we test models trained on Databricks AI Runtime
How to tell whether a model trained on your own data beats the one in production, what Databricks AI Runtime changes, and when training is worth it.
An MCP server for AI agents working loan files
AI agents need access to loan systems, with limits and a record of all they do. How we built that into AutoFabric, our auto refinance platform.
How we tested data quality rules against twelve defects
A data quality suite can pass every day and still let a duplicate order into the revenue report. We fed our rules twelve defects to find which got through.
How we merged five embedded Databricks dashboards into one
Five client reports as five embedded dashboards meant a new iframe and embed token on every switch. Why we merged them into one AI/BI dashboard, and the cost.
How we designed a release gate for Databricks data quality
A failed Databricks quality check doesn't stop bad data reaching reports on its own. We designed a release gate that does, and tested it on six cases.
Securing long-term agent memory on Databricks
An agent's memory is a write path into production. How we gate memory writes in Fabric, and how we'd carry that to Databricks agent memory.
Azure Databricks private network gateway for serverless compute
Azure Databricks' private network gateway lets serverless reach private databases through one subnet. What it changes for a migration, and what we'd test first.
Governing AI agents with Databricks and Temporal
Enterprise customers worry about AI agents going rogue. The limits we put on our own agents, and how Databricks and Temporal would carry them to clients.
Reading Delta Lake tables from DuckDB through Unity Catalog
We tried reading our own Databricks tables from DuckDB and none were readable. What that means for sharing lakehouse data with partners and AI agents.
Responsive joins Databricks: what Lakebase gains
Responsive, the SlateDB team, is joining Databricks to work on Lakebase. The two Lakebase limits that shape what we build, and which the news could move.
Business Central data mirroring to Fabric, read from Databricks
Microsoft announced Business Central mirroring of any table into Fabric, with no export to maintain. Where it fits a Databricks build, and where it stops.
What Databricks' Funke changes for HL7v2 pipelines
Funke, announced by Databricks on 8 October, parses HL7v2 into Unity Catalog intact. What it changes for clinical pipelines, and what it leaves alone.
How we cut duplicate queries from embedded Databricks dashboards
A client's embedded Databricks reports sent one query to their app database five times per filter click. How we cut that to once, and what the rule costs.
Replit on Databricks: governed apps with Lakebase Postgres
Replit Agent now deploys apps inside Databricks with a Lakebase database behind each. What we'd decide before anyone prompts an internal tool into existence.
Databricks bought Row Zero: spreadsheets on live data
Databricks bought Row Zero, a spreadsheet that reads the lakehouse live. What it changes for month-end exports out of Business Central, and what hasn't shipped.
Databricks Lakebase: managed Postgres next to your lakehouse
When an app's database and the lakehouse disagree, so do the reports. What we measured building on Databricks Lakebase, and which databases we'd move first.
Lakebase branches for test runs, previews and coding agents
A shared test database lets one change break another and holds up releases. How we give each test run its own Lakebase copy, and how we'd do it for AI agents.
AI voice calls for missing loan data in AutoFabric
When a refinance file stalls on one missing detail, an AI phone call can get it in minutes. How we built calls a lender can prove were allowed, now in a demo.
Migrating to Databricks without a cutover spreadsheet
How we move Synapse and Snowflake estates onto Databricks with Lakebridge and Airlift. One manufacturing migration took six weeks instead of six months.
GPT-6: choosing Astra, Sol or Luna and setting effort
GPT-6's three models and its effort setting decide most of what an AI workload costs. How we'd route work across Astra, Sol and Luna, and run it on Databricks.
Meta's ads MCP server on Databricks: what it changes
Meta's ads MCP server is in Databricks Marketplace. What it lets an agent do, how Unity Gateway and Meta rules fence it, and where it does not fit.
Business Central MCP server: query data with no API page
Business Central agents could only read a few standard tables without custom development. Version 29.0 lets them query the rest. How it works, where it stops.
Lakebase vs Snowflake Postgres: branching is the difference
Choosing a managed Postgres for apps beside your data platform? Lakebase against Snowflake Postgres on branching, scale to zero and governance. Lakebase wins.
Unity Catalog for AI agents in practice: four FHIR scenarios
An AI agent can show records a person isn't cleared to see with every row filter working. Four FHIR scenarios on Databricks, with the performance tradeoffs.
Decision models in auto refinance: where Jev and Clef fit
Lenders kick back refinance packages over a mismatched odometer or an expired payoff. Where decision models like Jev and Clef catch it, logged on Databricks.
Business Central reporting: built-in, Power BI or lakehouse
Three ways to report on Dynamics 365 Business Central, what each one is good for, the limits Microsoft documents, and how to tell which one you need.
An AI governance framework is four answers you can query
The EU moved its high-risk AI deadline to December 2027. What NIST AI RMF, ISO 42001 and the EU AI Act ask, mapped to controls you can run on Databricks.
LakeWright.NET carries a SaaS tenant boundary into Databricks
An Apache-2.0 .NET kit that resolves the customer tenant before a request reaches Databricks SQL, Jobs, AI/BI, Genie or Model Serving.
Embedded analytics on Databricks, part 1
One ticket, one platform, and the cheapest first step. Moving a B2B product's client reporting onto Databricks, starting with Lakehouse Federation.
Prove It, a winner in Databricks' Genie-Powered App Challenge
Prove It, by TechFabric engineer Ivan Vydrin, won in the Creativity & Originality track of Databricks' global Genie-Powered App Challenge, judged by Databricks.
Fabric or Databricks is the wrong question on Azure
Microsoft and Databricks shipped the interop this year. For teams already on Azure the question is which platform holds the tables and which one reads them.
The lock-in argument for leaving Snowflake is over
Snowflake and Databricks now read and write each other's Iceberg tables. If a vendor is still selling you a migration on lock-in, they are selling you 2023.
Export to Data Lake stops on 30 November
Export to Data Lake stops permanently on 30 November 2026. Microsoft names two replacements for Dynamics 365 teams. There is a third answer.
Nobody wanted to run the migration on a Friday
Staging has ten thousand rows, production has forty million, and every migration is a guess. What Lakebase branching does to a team's calendar.
Paying frontier prices to rename a variable
Databricks' smart routing sends routine coding-agent work to cheaper models and keeps the agents' traces in Unity Catalog. What it saves, and what to check.
Freshness, volume and schema checks on Databricks
Every job green, nothing failing, and the numbers quietly wrong because a source stopped sending. Freshness, volume and schema checks on system tables.
Conflicting metric definitions in Databricks Genie
Genie fails when two teams mean different things by the same word. Genie Accuracy hardens the semantic layer. TechFabric Experiments keeps the suite running.
Four thousand objects and nobody would name the exclusions
Warehouse migrations slip because the exclusions were never written down. The Migration Readiness Sprint inventories the estate before anyone picks a date.
The cost dashboard said the workspace was fine
A Databricks cost dashboard shows spend. A two-week Health Check finds the spend that bought nothing, from all-purpose clusters to orphaned schedules.
Unity Catalog before the first pipeline
Lakehouse pilots skip grants and then cannot be handed to security. The Lakehouse Launchpad puts Unity Catalog first, with TechFabric Airlift underneath.
Nobody could tell whether the agent was any good
A team shipped an agent in nine days and argued for two months about whether its answers were right. Write down what a good answer is first.
Databricks Repos for QA test automation workflows
Moving QA off ad-hoc notebooks and onto version-controlled test automation with Databricks Repos. Repo structure, modular tests, and how a team works in one.
Using PyTest for Data Quality Checks Directly within Databricks
Writing PyTest data quality checks inside Databricks notebooks. Bronze-to-silver validation, edge cases, fixtures and error handling, with examples.
Governance You Can Actually Ship
AI governance fails when it lives in documents instead of infrastructure. Compliance comes from systems that enforce the rules while the model is running.
DASF - Databricks AI Security Framework
DASF stands for Databricks AI Security Framework. Version 3.0 maps 97 AI risks to 73 controls. What it covers, and how to enforce those controls at runtime.
Unity Catalog for enterprise data governance
Who can see a row, who changed it, and is an agent held to a person's grants? A Unity Catalog Health Check finds where coverage stops halfway.
Databricks AI/BI dashboards and Genie
What Databricks AI/BI is: dashboards plus Genie, which answers plain-English questions with SQL. What changed in 2026, accuracy, and cost.
From data warehouses to data intelligence
How data platforms went from warehouses to lakehouses to data intelligence, and the four platform decisions that are hard to undo later.
Migrating from Legacy Data Warehouses to Databricks Data & AI Platform
Warehouse migrations slip on scope, not conversion. Two weeks on TechFabric Airlift inventories the estate, agrees exclusions and hands you a wave plan.
Fiber Copilot, a multi-tenant AI platform orchestrated by Temporal
How we designed Fiber Copilot to stand up chatbots and copilots on your own data, in your own cloud, with the parser, vector store and model you choose.
What Every CTO Needs To Know About Databricks
What Databricks is, what changed between 2024 and 2026, how the bill works and how to read it, and the governance and lock-in questions to settle first.
Hiring Databricks specialists: 12 expert insights
Twelve things to get right when hiring Databricks specialists in-house, from people who have built the teams and know what they screen for.