Skip to content

How we cap the cost of AI coding agents reviewing each other

Preetham Reddy7 min read

When one AI coding agent reviews another's work, the loop can keep finding something to fix long after the real problems are gone, and every round is paid for in frontier-model tokens. Our own reviewer read one pull request on Heddle, our SEO product, 19 times before we stopped it, and by the late passes it was only turning up rare edge cases.

At TechFabric one coding agent writes each change and a model from another vendor reviews it read-only, so the reviewer doesn't share the author's blind spots. Another pull request, on the repository behind techfabric.com, went 14 passes. Every pass was a frontier model reading the whole change again at high effort. We could tell the loop was slow. We couldn't tell what share of our agents' spend it took next to the building itself, and a team running several coding agents needs that number before it decides how much review to pay for. Databricks' new Unity Gateway Insights reports where coding agents' tokens go.

How one agent writes and another reviews

We use Codex, Claude Code, Gemini, Droid and Kimi to write code, and pick the agent for the job. Whichever agent wrote the change, a frontier model from a different vendor reviews it. An agent reviewing its own code shares its own blind spots.

Every review goes through one of two small scripts we wrote, claude-review and codex-review. Each pins the model and its effort, Claude Opus 5 at high effort or GPT-6.1 Sol at high reasoning effort, and starts the reviewer read-only, so it can read the repository but can't change a file in it. Each also fixes the prompt and limits the review to what changed since the branch left the main line, plus any uncommitted work. The third round of a review then runs with the same model, instructions and scope as the first, and the two can be compared. We wrote about a model-name trap we hit in that script.

# Read-only review of everything since the branch left master
codex-review origin/master "What the change should do, and is it ready to merge?"

A review that crashes, times out or comes back empty counts as no evidence. We never read it as a pass. The script retries a failed review once and gives each attempt 30 minutes. One reviewer, agy, run in its read-only plan mode, gates every pull request on techfabric-web.

Why the reviewers never fully agreed

Our first rule was to re-run the review until it came back with no findings. It felt like the safe rule. The implementing agent fixed what the reviewer flagged, the reviewer read the new version, and it flagged something else. Pull request #251 on techfabric-web went 14 passes like that, and Heddle's #27 went 19.

By the late passes the reviewer was raising rare edge cases, each one worth a ticket and none worth holding the merge. Two frontier models from different vendors never fully agreed on a change, and we'd been waiting for them to.

We decided the loop was taking too long and changed the bar.

The stop rule we use now

We merge once the reviewer reports no P1 finding and CI, our automated build and test run, is green. P1 is our label for the most serious finding: for us that's data loss, publishing without approval, an unchecked claim on the site, or a security exposure. Anything below that becomes a tracked issue, and the pull request merges. Later we tightened it again, to at most two review rounds before a merge.

Here's the loop as it runs now. The coding agent makes the change and the review script scopes the diff. The reviewer from another vendor reads it read-only and answers PASS or BLOCK, with each finding tied to a file and line. When a P1 is found, the change goes back to the coding agent. When no P1 is left, CI runs, and a green CI means we merge. Every call both agents make would also pass through Unity Gateway into its trace table, where the Insights report reads it.

P1 found

No P1

Coding agent makes the change

Review script scopes the diff

Reviewer from another vendor, read-only

PASS or BLOCK, with file and line

CI green, merge

Unity Gateway trace table

Insights report

Every change passes a reviewer from another vendor before merge, and Unity Gateway would trace both agents for Insights.

We accepted what the rule costs. Some P2 edge cases ship as tracked issues and get fixed after the merge, where the old rule would have held them back. We also stopped asking several reviewers to agree. Each pull request answers to one independent reviewer, run through the same script every time, so any two rounds compare.

What a review loop costs, and why we couldn't see it

Each review round is a frontier model at high effort reading the whole change, and each fix is another turn for the implementing agent. Over 14 or 19 rounds that adds up, spread across several vendors' agents. The loop's share was the part we couldn't see: how many of our agents' tokens went to review, next to building the change in the first place.

That share should set the round cap. If review is a small slice of the spend, two rounds are cheap and a third would be affordable. If it's a third of the spend, the cap should come down, or the simple changes should go to a cheaper reviewer. We changed the bar because the loop took too long, and set the two-round cap, both without knowing what review cost us.

What Unity Gateway Insights would show about the loop

Unity Gateway is Databricks' governed entry point for model calls, and Insights is a report on what flows through it. In the announcement, teams launch Claude Code, Codex or GitHub Copilot through the ug command line, which turns on each agent's OpenTelemetry tracing, a record of every call it makes. The traces land in a table in Unity Catalog, where Databricks controls who can read them. Insights uses Genie, Databricks' natural-language analyst, to read them and write the report, which sits under AI Gateway, Observability, Insights in the workspace and covers the last day, 7 days or 30 days. Each finding links to example sessions, so an engineer can open the agent run behind it.

We'd read the workstream view first. It sets each kind of work's share of sessions beside its share of tokens, and Databricks' example is a workstream with 10% of sessions and 35% of tokens. If code review shows up as its own workstream, that's the number we set our cap without.

Insights also ranks waste by cost: failing calls to MCP servers (the connectors agents use to reach other tools), simple tasks running on frontier models, bloated context, and repeated workflows that should become a shared skill. Databricks says seven small bugs in its own MCP servers came to an estimated $1.2M a year, and fixing them took about an hour. For us the routing finding matters most. We'd expect a review of a one-line config change to need less than Opus 5 at high effort, and Insights would show how often we pay for it anyway. Databricks also writes that offline replays showed no gain from Opus 4.8 to Opus 5 on its own work, so it stayed on 4.8. We pin Opus 5 for review and would want that choice checked against our own traces.

We haven't run Insights on our own agent traffic yet.

The announcement names Claude Code, Codex and GitHub Copilot, so before we'd trust a report on our loop we'd check how Gemini, Droid and Kimi traffic reaches the gateway. It doesn't say whether Insights is in preview or generally available, or what it costs.

Where we'd start with your own agents

Route your coding agents' traffic through Unity Gateway first, with the ug command line for the agents it names, and run an Insights report over 30 days. Read what share of tokens goes to code review against the share that goes to building, and set the round cap from that number. If your agents review each other with no cap at all, set one at two rounds now and let the report tell you whether to move it.

Frequently asked questions

What is adversarial code review with AI coding agents?

One coding agent writes a change and a model from a different vendor reviews it, read-only, looking for what the author missed. The implementing agent fixes what the reviewer flags, and the reviewer reads it again. It works because the two models have different blind spots, and it needs a stop rule or it runs on into edge cases.

How many review rounds should a coding agent loop run?

We cap ours at two rounds before a merge, and stop sooner once no P1 finding is left. Rounds only compare if each one uses the same model, effort, prompt and scope, so pin those before you start counting. The cap should come from what share of your agents' tokens review takes.

Which coding agents does Unity Gateway Insights support?

Databricks' announcement names Claude Code, Codex and GitHub Copilot, launched through the ug command line. Their traces go to a Delta table in Unity Catalog, and only the table owner can query it until admins grant access with Unity Catalog grants and row filters. Check with Databricks how any other agent's traffic reaches the gateway before relying on the report.

Should the reviewer model come from a different vendor than the coding agent?

We use a different vendor's model for every review, because an agent reviewing its own code shares its own blind spots. We also keep the reviewer read-only. A reviewer that can edit can quietly rewrite the work it was asked to judge, and then nobody has reviewed the result.