Skip to content

Exporting AI decision history from Lakebase through one event table

Raja Vemuri5 min read

Lakebase Change Data Feed exports a whole Postgres schema, every table in it now and later, so pointing it at an app's public schema would export its credential tables too. At TechFabric we built Decision Replay into Fabric, our AI work platform, and gave the feed a schema of its own with one event table that triggers fill.

Decision Replay records what happens to the work Fabric's AI proposes, such as a backlog change that is suggested, reviewed by a person, applied, rated or flagged as a duplicate. Triggers on five Fabric tables fill the event table, because a proposal's status alone is written in 25 places across 8 files. The feed streams it into Delta, where a scheduled job rebuilds six gold tables; across four runs, a refresh took 6 to 8 minutes and cost about 6 to 7 cents at list price. A dashboard embedded in Fabric shows each project its own numbers. Only the columns we chose leave Postgres, and the code is in the fabric-oss repository.

Why the feed gets a schema of its own

The Lakebase Change Data Feed documentation configures a feed per schema, and every current and future table in that schema is included. Each table needs REPLICA IDENTITY FULL, which makes Postgres log the whole row before and after a change. Adding, dropping or retyping a column makes the feed re-read the whole table and rewrite its history.

Pointed at Fabric's public schema, the feed would have taken all of it: the credential tables, the busy tables, and a fresh copy of a table each time a migration touched its columns. So we created lakehouse_export, a schema with one table, and pointed the feed there. A new table or column in public can't start flowing by accident, credential tables stay out for good, and the busy tables keep their default replica identity. The price is that we design the event's shape ourselves and fill it from triggers.

One event table filled by triggers

The event table, from the capture migration, is append-only and small:

CREATE TABLE lakehouse_export.decision_event (
    id text PRIMARY KEY DEFAULT gen_random_uuid()::text,
    occurred_at timestamptz NOT NULL DEFAULT clock_timestamp(),
    organization_id text NOT NULL,
    project_id text,
    subject_type text NOT NULL,
    subject_id text,
    event_type text NOT NULL,
    from_state text,
    to_state text,
    actor_kind text NOT NULL CHECK (actor_kind IN ('human', 'ai', 'system', 'unknown')),
    actor_id text,
    related_subject_id text,
    feature_key text,
    model text,
    prompt_version_id text,
    payload jsonb,
    schema_version integer NOT NULL DEFAULT 1
);
ALTER TABLE lakehouse_export.decision_event REPLICA IDENTITY FULL;

Each row says who or what acted, on which subject, and the state before and after. Triggers on five Fabric tables write the rows. They sit on backlog proposals, applied proposals, AI outcome events, duplicate links and user stories.

We chose triggers over app code because a proposal's status alone is written in 25 places across 8 files, and a trigger catches every one of those writers. The proposal triggers fire only when the status actually changes. Capture stays off until an organization's DECISION_REPLAY flag is on, and the triggers check that flag themselves. The trade is work inside the business transaction. Once one transaction captures more than about 64 rows, Postgres starts paying subtransaction overhead.

What may leave Postgres and what never blocks a write

Two kinds of failure needed opposite answers. A capture error shouldn't cost a user their write, so each capture function catches its own error, logs a warning and lets the business write commit. Erasure goes the other way. Deleting an organization or a project deletes its events, deleting a user removes their id from them, and if that purge fails, the delete doesn't happen.

A sanitizer runs before each insert and keeps only values we know. Event and subject types come from fixed lists, and a model name stays only if it's in Fabric's model catalog; anything else becomes unknown. So the analytics downstream carry no free text, and building them makes no AI calls.

From the history table to six gold tables

The feed writes each event to lb_decision_event_history in Unity Catalog, with the change type and log position on every row. A serverless job then runs three SQL steps in order:

01_current.sql   rebuild decision_event_current from the whole history
02_gold.sql      rebuild the six gold tables
03_grants.sql    grant SELECT on them to the dashboard's service principal

We rebuild the current state from the whole history on every run, so deletions and the feed's re-copies carry through without tracking offsets. The six gold tables cover change outcomes, the journey of each suggestion, daily decisions per project, what needs review, duplicates and ratings. An AI/BI dashboard embedded in Fabric reads them, filtered per project, the pattern our post on tenant isolation in embedded dashboards walks through.

End to end, five Fabric tables fire triggers that keep only known values, the triggers write lakehouse_export.decision_event, the Change Data Feed copies it to lb_decision_event_history in Delta, and the job builds the current state and six gold tables that the dashboard in Fabric reads per project.

Five Fabric tables

Triggers keep known values

lakehouse_export.decision_event

Change Data Feed

lb_decision_event_history in Delta

Current state

Six gold tables

Dashboard in Fabric, per project

Only the event table reaches the feed, and the dashboard reads gold tables built from it.

A schedule instead of a table-update trigger

The job runs on a fixed schedule, every 6 hours from midnight, rather than on each update to the history table. Triggered by updates, it would have run close to hourly and paid for a refresh each time. Across four runs, a refresh cost 0.13 to 0.16 DBU, about 6 to 7 cents at list price, and took 6 to 8 minutes.

We left three ideas out: monitoring from the feed, a compliance-archive claim, and syncing results back into Lakebase, the last because synced tables can't carry row-level security.

Before you turn on the feed, decide which changes should leave Postgres. Put them in one schema with one table, give that table full replica identity, and point the feed at that schema alone, so your app's migrations stay out of Delta.

Frequently asked questions

Does Lakebase Change Data Feed export every table in a schema?

Yes, every current and future table in the schema, once it has REPLICA IDENTITY FULL and at least one row. A schema has one feed, so you add a table by creating it in the schema with full identity rather than starting the feed again. Partitioned tables aren't supported.

Does every table need REPLICA IDENTITY FULL for Lakebase Change Data Feed?

Every table you want in the feed does. By default Postgres logs only the primary key when a row changes, and the feed needs the whole row to write an update as a before row and an after row. A table without full identity is left out of the feed.

Can I add row filters to a Lakebase Change Data Feed history table?

No. The feed stops writing to a history table once a row filter or column mask is applied to it, and the documentation also warns against turning on Delta change data feed there. Filter in what you build downstream; our dashboard's datasets filter per project on the gold tables.

What happens to the Delta history when a Lakebase table's columns change?

Adding, dropping or retyping a column makes the feed re-read the table from Postgres and rewrite its history table. Renaming a table keeps the feed running under the old destination name. With one export table whose shape we change on purpose, migrations to Fabric's own tables never cause that re-read.