TechFabric helps design, deploy and maintain AI agents for many companies. Our job is to understand the business domain and design AI workflows that address business problems. We're not just implementing current workflows and slapping AI on top of them. We try to understand the essence of the problem and see how AI agents can help address it and bring the efficiencies, cost savings or productivity that the business needs. In AutoFabric, our own auto loan product, that meant agents that run credit pulls, VIN decode and payoff quotes against DealerTrack, RouteOne and CDK, with an OPA policy engine checking every action against TILA, ECOA and FCRA. It's really important to ensure AI agents work as reliably in production as they did during development. We also continuously evaluate what these agents produce with MLflow on Databricks, whose LLM judges and code-based checks score each agent trace, and the same scorers can run automatically on a sample of production traffic. When a workflow runs for hours, that means planning for model calls that time out, workers that restart mid-run, approvals that wait overnight and new code that ships while runs are still going, all before an agent reaches production.
An agent demo runs for ninety seconds on one laptop, and in production the same agent runs for hours, calls a model dozens of times, writes to three systems and waits overnight for a person. At Location Services a single vendor call that nobody retried could lead to a wrongful repossession, so our engineers rebuilt the process as a Temporal workflow, and that is the pattern we build durable AI agents on, with Databricks holding their data, models and permissions.
The Location Services workflow carries one status update that has to reach a vendor, and before we rebuilt it, that update failed quietly because nothing remembered where the process had got to. An agent run makes dozens of calls like that one, to models, tools and vendors. Models drop connections and answer slowly or badly, tools fail halfway, and every one of those is a place where a run can stop without anyone being told. Temporal recovers from them through durable execution, rehydration and replay, and Databricks governs everything the agent touches along the way.
We covered the platform split, business data in Databricks and process state in Temporal, in Databricks and Temporal for durable data and AI systems. The agent run itself is where the trouble starts, somewhere after the first hour.
Why a working demo says little about hour three
A demo is one run, on a good day, watched by the person who wrote it. The model answers on the first try, every tool is up, nobody deploys mid-run, and if something goes wrong the presenter starts again. Starting again is the step production can't take, because by then the agent has already sent an email, written a record or asked someone to approve something.
Time is what changes the odds. A model call that fails one time in a few hundred is invisible in a demo and certain across a day of runs that each make dozens of calls. A worker that restarts once a week for a deploy will restart under some of the runs that last for hours. A person who approves in five minutes during a demo approves the next morning in real life. None of these is exotic, and an agent loop that keeps its progress in a Python process's memory loses all of it to any one of them.
Six failures a long-running agent will hit
Each of these has a Temporal mechanism that handles it and a Databricks piece behind it. In every case the workflow is ordinary code, and every call that touches the outside world, a model, a tool, a database, is an activity whose result Temporal records.
1. The model call times out or comes back unusable
Model endpoints rate-limit, time out and occasionally return text that doesn't parse. As an activity, each model call gets its own retry policy with backoff, so a rate limit means that one call waits and tries again while the rest of the run keeps its place. Output that fails validation can raise a retryable error and get another attempt. A policy violation should raise a non-retryable error, because asking again won't change the answer.
On Databricks the model sits behind Unity Gateway, so the rate limit the agent hits is a limit you set per endpoint and per user, and not only the provider's.
2. A retry repeats something that already happened
Activities run at least once. If a worker dies after the vendor accepted a write but before Temporal recorded the result, the activity runs again. We'd pass an idempotency key built from the workflow ID and the step name into every write, so the second attempt finds the first one and returns its result. Temporal retries the step, and the key keeps the vendor from seeing the same write twice.
3. The worker crashes or gets redeployed mid-run
When a worker dies, Temporal rehydrates the workflow on another one. The new worker picks the workflow up, replays its event history to rebuild the agent's state, and carries on from the next step. Steps that already finished aren't run again, and the run doesn't start over. At Location Services, Temporal's ability to replay workflows from any point in their execution history improved debugging.
4. The agent waits for a person for days
A human approval is a signal the workflow waits for, with a timer as its deadline. Neither holds a thread open, so a wait of three days survives every deploy in between. In AutoFabric, our auto loan product, an approval wait is its own workflow with a deadline that rechecks the approval every ten minutes, and when the deadline passes it expires the request instead of leaving it open.
5. New code ships while runs are still in flight
Replay depends on the workflow code making the same decisions it made the first time. Change the order of steps or add a new one, deploy it, and a run that started yesterday will replay against code that no longer matches its history. Temporal reports that as a non-determinism error and stops the run. Patching lets old runs follow the old path while new runs take the new one, and Worker Versioning pins each run to the build it started on. For agents, prompt and tool changes count as code changes when they alter which steps the workflow takes.
6. A late step fails after earlier steps have written
An agent that reserves a slot, writes a record and then fails to notify a vendor has left the world half-changed. Retrying the last step is right when the failure is temporary. When it's final, the workflow runs compensations in reverse order, releasing the slot and marking the record void, so the systems the agent touched agree with each other again. In Temporal that's ordinary error handling in the workflow code, and it's recorded in the history like everything else.
Put together, one run goes like this. The agent's model call goes through Unity Gateway and retries on a rate limit or timeout. Its tool calls write to Lakebase with idempotency keys. It waits for an approval signal with a deadline. A worker crashes, a new worker replays the history, and the run resumes at the next step. If notifying the vendor fails for good, compensation voids the earlier writes, and either way the full history is kept.
In Temporal's Python SDK, the model retry, the idempotency keys, the approval wait and the compensation fit in one workflow class.
import asyncio
from datetime import timedelta
from temporalio import workflow
from temporalio.common import RetryPolicy
from temporalio.exceptions import ActivityError
with workflow.unsafe.imports_passed_through():
from activities import plan_with_model, write_case, notify_vendor, void_case
MODEL_RETRY = RetryPolicy(
initial_interval=timedelta(seconds=2),
backoff_coefficient=2.0,
maximum_attempts=8,
non_retryable_error_types=["PolicyViolation"],
)
@workflow.defn
class CaseAgent:
def __init__(self) -> None:
self.approved: bool | None = None
@workflow.signal
def review(self, approved: bool) -> None:
self.approved = approved
@workflow.run
async def run(self, case_id: str) -> str:
key = workflow.info().workflow_id
plan = await workflow.execute_activity(
plan_with_model,
case_id,
start_to_close_timeout=timedelta(minutes=2),
retry_policy=MODEL_RETRY,
)
await workflow.execute_activity(
write_case,
args=[case_id, plan, f"{key}:write"],
start_to_close_timeout=timedelta(seconds=30),
)
try:
await workflow.wait_condition(
lambda: self.approved is not None, timeout=timedelta(days=3)
)
except asyncio.TimeoutError:
return "expired"
if not self.approved:
return "rejected"
try:
await workflow.execute_activity(
notify_vendor,
args=[case_id, f"{key}:notify"],
start_to_close_timeout=timedelta(seconds=30),
)
except ActivityError:
await workflow.execute_activity(
void_case,
args=[case_id, f"{key}:void"],
start_to_close_timeout=timedelta(seconds=30),
)
raise
return "done"
Replay reads the model's recorded answer
A model is non-deterministic and Temporal replays workflow code, which raises the obvious question of what happens to the model's answer on a replay. The model call is an activity, so its result is written to the event history once, and on replay the workflow reads that recorded result and moves on. A resumed agent rebuilds exactly the state it had, with the same plan and the same tool results.
That has two effects worth knowing. Recovery after a crash costs no extra model calls for the steps that already finished. And the history becomes a record of what the agent decided and why, which is the property AutoFabric relies on when an examiner asks months later why a deal went the way it did.
The history does grow with every step, and an agent that loops for hours can make it large. Continue-As-New closes the current run and starts a fresh one carrying only the state the agent needs, such as its plan and a summary of what it has done, so a long agent stays fast to replay. A long-running activity, such as waiting on a Lakeflow job, should heartbeat so Temporal notices quickly when the worker running it has gone.
What Databricks holds while the workflow runs
Temporal knows where the agent has got to. Databricks holds what it reads, what it calls and what it's allowed to do.
- Models. The agent's models run on Model Serving behind Unity Gateway, which applies rate limits, budgets and per-user caps and records who called what. An agent stuck in a loop stops at its budget, and the workflow sees a failed activity it can handle.
- Permissions. Unity Catalog treats models, tools, MCP servers and agents as securable objects, so each tool gets the one grant it needs. Our post on governing AI agents with Databricks and Temporal goes further into those limits.
- Case data. The records the agent reads and writes live in Lakebase, with Temporal keeping the state of the process apart from the business record.
- What the model saw. MLflow Tracing records the inputs, outputs and latency of every intermediate step of an agent, including its model calls and tool calls. We'd tag each trace with the Temporal workflow ID, so a step in the history leads straight to the trace that explains it.
Temporal doesn't replace Lakeflow Jobs here. When an agent needs a pipeline refreshed, the workflow triggers the job as an activity and waits for the result, and Lakeflow runs the pipeline inside the platform.
Testing the failure before it happens
At Athlinks, matching more than 400 million athlete records was a job that couldn't start over when it died halfway, so Temporal held the state, failed steps retried on their own and failures reached Slack as they happened. An agent needs the same guarantees, and we'd prove them by causing each of these failures on purpose in a staging environment before production causes them.
- Kill the worker between the approval and the action that follows it, and confirm the run resumes without asking twice or acting twice.
- Deploy new workflow code with runs in flight. Temporal's docs recommend replay testing, which runs a workflow's recorded history against your code, to check a change is replay-safe before you deploy it.
- Cut the model endpoint or set its gateway limit to zero, and check that the activity backs off, the run waits, and it recovers when the endpoint returns.
- Fail the last step for good, and check that the compensations leave every system consistent.
Start with the agent whose half-finished run costs the most, where a missed step has a penalty or a customer attached. Our AI workflow automation and durable execution work starts from that one process.
Frequently asked questions
Does Temporal re-run the LLM call when a workflow replays?
No. The model call runs as an activity, so its result is recorded in the workflow's event history the first time. On replay the workflow reads that recorded result, so a resumed agent gets the same answer it got before and the replay costs no extra model calls.
How do you deploy new agent code while workflows are still running?
With Temporal's versioning. Patching lets runs that started on the old code keep following the old path while new runs take the new one, and Worker Versioning pins each run to the build it started on. Run the SDK's replayer against recorded histories in CI to catch changes that would break runs in flight.
What's the difference between Temporal and Databricks Lakeflow Jobs for agents?
Lakeflow Jobs orchestrate data pipelines inside Databricks. Temporal orchestrates the agent's process across systems, including model calls, vendor APIs and waits for people, and it triggers Lakeflow jobs as activities when the agent needs data refreshed. Our Temporal vs Airflow comparison covers the same line for schedulers in general.
How do you stop a long-running agent from running up model costs?
Put the limit in two places. Unity Gateway applies budgets and per-user caps to every model call, so a looping agent stops at its cap. In the workflow, cap the number of model steps per run, so the agent ends the run or asks a person once it reaches the limit.