# From data warehouses to data intelligence

> How data platforms went from warehouses to lakehouses to data intelligence, and the four platform decisions that are hard to undo later.

Published: 2025-07-03 · Updated: 2026-10-25
Author: Preetham Reddy
Tags: Databricks, AI/ML
Canonical: https://www.techfabric.com/blog/from-data-warehouses-to-data-intelligence-why-your-next-platform-choice-will-define-the-next-decade

---

![](/blog-media/d1b7f2f1-Screenshot-202025-06-11-20081026.jpg)

A data platform chosen in 2026 will still be running in 2036, because the expensive part of a data platform is the years of tables, permissions, pipelines and reports that pile up on top of it.

Most of what goes on top can be moved later at a reasonable cost. Four things can't. They are the format your data is stored in, the catalog that holds its permissions, the way you pay for compute, and where your AI runs relative to the data.

*Updated by the TechFabric editors in October 2026.*

## How we got here

**Warehouses.** For about thirty years, analytics meant a data warehouse, from Teradata and Oracle to SQL Server and the first cloud warehouses. Data was cleaned, shaped into tables and loaded on a schedule. Warehouses were fast and well governed, and they were built for structured data and SQL. Logs, documents, images and sensor readings didn't fit, and storing a lot of anything was expensive.

**Data lakes.** The answer was to put everything in cheap file storage and work out the structure later. Apache Spark, which [started at UC Berkeley's AMPLab in 2009](https://spark.apache.org/history.html), made it practical to process those files at scale, and its creators founded Databricks in 2013. But lakes had no transactions and no reliable schema, and most companies ended up running a lake for data science and a warehouse for reporting, with copies flowing between them and two sets of permissions to keep in step.

**Lakehouses.** Delta Lake, which Databricks [open-sourced in April 2019](https://www.databricks.com/company/newsroom/press-releases/databricks-open-sources-delta-lake-for-data-lake-reliability) and handed to the Linux Foundation that October, added transactions and schema to files in a lake. In January 2021 Armbrust, Ghodsi, Xin and Zaharia set out the argument in a paper at CIDR, ["Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics"](https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf). One copy of the data, in an open format, serving both BI and machine learning.

**Data intelligence.** In November 2023 Databricks began calling its product the [Data Intelligence Platform](https://siliconangle.com/2023/11/16/databricks-debuts-ai-powered-data-intelligence-platform/). The idea is that the platform uses AI to understand your data, its meaning, usage and lineage, and that people can ask questions of it in plain English. In product terms, that means Genie Agents for natural-language questions over tables, Agent Bricks for building agents, and Unity Catalog underneath them so an agent sees only what the person asking is allowed to see.

The thread through all four stages is a single question: how many copies of the data, under how many permission models? Each stage reduced the number.

## The four decisions that are hard to undo

### 1. The storage format, and who owns the files

If your tables are stored in a format only one engine can read, leaving that engine means exporting everything. If they're open-format files in a storage account you own, another engine can read them in place.

This is much less of a differentiator than it was in 2023, which is good news for buyers. Databricks stores tables as Delta Lake and, with [UniForm](https://docs.databricks.com/aws/en/delta/uniform), writes Iceberg metadata alongside, so Iceberg engines can read the same files. Snowflake's [Iceberg tables](https://docs.snowflake.com/en/user-guide/tables-iceberg) keep Parquet in your own storage too. The question to ask a vendor is concrete. Can another engine read these tables, today, without an export? The answer depends on which table type you chose, not on the brand.

### 2. The catalog that holds your permissions

Format gets the attention, but the catalog is stickier. Grants, row filters, column masks, lineage, audit history and the names everyone's queries use all live there. Moving files is a copy job. Rebuilding three years of access policy is a project, and a risky one, because a mistake means somebody sees data they shouldn't.

On Databricks this is Unity Catalog, [open-sourced in June 2024](https://www.databricks.com/blog/whats-new-databricks-unity-catalog-data-ai-summit-2024), with tables exposed through the Iceberg REST Catalog API. Whatever you choose, choose it before the first pipeline, put every table in it, and don't let a second catalog grow up beside it for the AI team.

### 3. How you pay for compute

Some platforms bill for capacity you reserve. Others bill per second for what runs. Per-second serverless billing is better for spiky analytics and worse for anything left running, and the defaults decide which you get. A Databricks serverless SQL warehouse [stops after 10 minutes idle](https://docs.databricks.com/aws/en/compute/sql-warehouse/create), a classic one after 45.

The decision here is less about price per unit than about whether you can see spend. On Databricks the bill is a table, `system.billing.usage`, kept for 365 days and queryable in SQL by job, warehouse and user. A platform whose cost you can only see on next month's invoice will surprise you every quarter for ten years.

### 4. Where the AI runs, relative to the data

This is the decision that most separates 2026 from 2021. An AI system that works on an export of your data has a copy of it, with none of your permissions attached. One that runs inside the platform reads through the same catalog as everyone else.

```mermaid
%% caption: An AI system working on an export has a copy with none of your permissions, while one inside the platform reads through the same catalog.
flowchart TD
  A["Your data"] --> B["Export"]
  B --> C["AI system works on a copy"]
  A --> D["Same catalog"]
  D --> E["AI system runs inside the platform"]
  class E accent
  class C legacy
```

That's the practical meaning of "data intelligence" for a buyer. When a sales manager asks a Genie Agent about margin by region, the row filters that apply to their dashboards apply to the answer. When an agent built with Agent Bricks looks up a customer, Unity Catalog decides which columns it sees. The alternative, exporting tables to wherever the AI tool lives, recreates the warehouse-and-lake problem of 2015 with a language model in the middle.

## What's easy to change later

Not everything deserves this much care. The BI tool can change, since Power BI, Tableau and Databricks' own AI/BI dashboards all read the same tables. Orchestration can change. Individual pipelines get rewritten as sources change anyway. Spend your evaluation time on the four above and let the rest be a reasonable default.

## Why we chose Databricks

TechFabric is a Databricks partner, and Databricks is where we've chosen to go deep. When I wrote the first version of this post I listed what we look for in a platform, and the list hasn't changed. Open standards over proprietary lock-in. Scale that grows with the business. Governance and security designed in from the start. And a platform that lets every team member, not only engineers, work from the same trusted data.

Read against the four decisions, Databricks scores well on all of them. It has open formats in your own storage, one catalog that is itself open source, billing you can query, and AI that runs under the same permissions as your reports. Snowflake has closed much of the gap on the first. Microsoft Fabric is a reasonable choice for an organisation whose data is mostly Microsoft and whose reporting is all Power BI, and we've written about [how Fabric and Databricks fit together on Azure](/blog/fabric-or-databricks-is-the-wrong-question).

## A test you can run this quarter

Take one question your leadership asks every month that needs data from more than one system to answer. Revenue by customer with support tickets beside it is a good one. Then ask of your current setup, or of any platform you're evaluating:

1. How many copies of each source table exist on the way to the answer?
2. Where are the permissions defined, and how many places?
3. Can you say, in one query, what producing the answer cost last month?
4. If you pointed an AI assistant at it, would the assistant see more than the person asking?

If the answers are "several," "several," "no" and "probably," the platform choice is still ahead of you, and it's worth making deliberately. Our [Databricks consulting](/databricks/consulting) page describes how we help teams make it.

## Frequently asked questions

### We already run a warehouse and a lake. Do we have to pick a new platform to fix the copies problem?

Not necessarily, because the test works on the setup you have now. Run the revenue-by-customer question through it and count the copies and the places permissions are defined. If both counts come back as "several," the catalog is the first thing I'd look at, because it's the stickiest of the four and the one a second copy of the data makes worse.

### Does choosing open-format tables mean I can switch engines without any work?

No, it means the files don't have to be exported, which removes the largest part of the work. The catalog still holds your grants, row filters, column masks and lineage, and rebuilding that is the risky part of a move. So an open format makes leaving possible, and the catalog decides how hard it is.

### How do I tell whether serverless billing will cost me more than reserved capacity?

Look at how much of your compute sits idle. Per-second billing helps spiky analytics and hurts anything left running, so a warehouse that's busy for an hour and then idle favours serverless, while one that never stops doesn't. Because the bill is queryable in `system.billing.usage` by job, warehouse and user, you can check this against a year of your own usage instead of guessing.

### Which parts of our stack can wait until after we've chosen the platform?

The BI tool, orchestration and individual pipelines can all change later at a reasonable cost. That means a reporting tool you already like shouldn't decide the platform for you, since Power BI, Tableau and the Databricks dashboards all read the same tables. Put the evaluation time into format, catalog, compute billing and where the AI runs.