# What our coding agent's green test runs left untested > A coding agent reported 33 of 41 test tasks successful while the three packages it changed never ran. The checks that would have caught it. Published: 2026-10-22 Author: Vladislav Mandryk Tags: Applied AI, Knowledge Share, Field Notes Canonical: https://www.techfabric.com/blog/coding-agent-green-test-runs-untested --- When a coding agent writes back that the suite passed apart from one environment failure, the summary reads like proof that the change was tested. Whether it is depends on what the summary counted. For a team that ships what its agents test, the difference is code reaching production that nobody ran. We build Fabric, TechFabric's open-source delivery platform, and most of our changes are written and tested by a coding agent, with a second model reviewing the diff before it merges. In September our agents reported green results several times that hadn't covered the code they were about. The reports looked conclusive, and each had a gap that only showed once somebody asked what had actually run. A few checks on the report itself would have caught every one of them. The clearest case came on 21 September, on a branch that gathered meeting action items into one to-do list. ## A full test run where seven tasks never started After rebasing the branch onto 17 new commits, the agent ran `pnpm test` in the background and piped it through `tail -60`. The command exited 0. Turbo's summary line counted 33 successful tasks out of 41 total, and it listed one failure, a CLI package test that breaks on macOS because `/tmp` resolves to `/private/tmp`. The exit code came from `tail`, which succeeds whatever it reads. Turbo had stopped the queue when the CLI test failed, so seven tasks never started. Those seven included the api, web and temporal packages, the three our branch had changed. No line in the output said so. The only sign was that 33 isn't 41, and the agent spotted it as soon as the run finished. A rerun with `--continue --force` came back with 16 failures, all timeouts, which the agent first put down to CPU contention. The cause was its own command. Calling turbo directly dropped the `--concurrency=1` that our root test script sets, so the tasks overlapped. None of the 16 failed again once the suite ran serially. Then the database package came back with 4,147 tests passed and 862 skipped. In a summary, skipped looks like part of a pass. These were integration tests that switch themselves off when they can't reach Postgres, and one of them was the test for the race condition our branch fixed. The agent reported that it couldn't run them, because Docker wasn't reachable and the local database wasn't up. ## The question that got the database tests running Two hours into the session, I asked the agent what was stopping it from checking against the local database. It corrected itself within a minute. Docker was running, and "unreachable" had been one sandboxed call it never retried. Our Postgres container had exited two days earlier. Our repository rules say an agent doesn't start containers unless someone asks, so it stopped there and called the database unavailable. Once asked, the agent started our container on its own port and pointed the tests at it through the repository's test env file. The serial run took the database package from 4,147 passed and 862 skipped to 4,571 passed and 443 skipped, race test included, and all 41 tasks ran. One file was still skipped in full, and it was ours. Its 25 tests for the new to-do query sat behind a second switch, `RUN_DB_INTEGRATION=1`, and passed once the agent set it. The agent also saved the lesson to its memory notes that morning. The next session that saw 33 successful out of 41 would read it as a warning. ## A release script that watched another run Until October our feature work went through a private staging repository, and a relay job published each merged change to Fabric's public repository. A shell script carried a branch along that path. It opened the pull request, waited for checks, posted the relay command and watched the relay. Through September it reported results that weren't its own, in both directions. Seconds after opening a pull request it ran `gh pr checks`, got "no checks reported" because GitHub hadn't registered the workflows yet, and exited with a failure. Further on, it found the relay with `gh run list --workflow oss-relay.yml --limit 1`, which returns the newest run in the repository. When another relay was in flight, the script watched that one and printed "relay succeeded" while ours sat behind a six-minute wait timer. Watching our own relay wouldn't have settled it either. The relay spans several runs by design, and the first one ends in success long before anything is published. Our agents learned to distrust the script from a memory note. It told them to look for the change on the public main branch before believing the script's last line, and that's the check a script like this should end on. ## An output filter that printed no issues We run a token-saving filter in front of the agent's shell, and it condenses long command output before the agent reads it. For Biome, the formatter we use, it condensed every run to `Lint: No issues found`. A per-file `biome format --write` printed that line whether it had changed a file or not, so the agent reported formatted code as clean. The staging repository's format check is path-filtered and often skips, so the mistake surfaced on the public relay pull request, after the release had already been approved. It cost us a full relay cycle twice in September, with a new push and a new approval each time. On 30 September the agent ran the same command through the filter's pass-through mode, and Biome printed "Fixed 1 file". ## How we'd read a green result We'd treat a green summary as a claim about one run, and it counts as evidence only after it's checked against what that run could have seen. The first check is arithmetic. Turbo prints successful and total tasks on one line, and when they differ, each missing task needs a name and a reason before anything else gets read. Turbo's [`--continue` option](https://turborepo.com/docs/reference/run#--continue) with no value defaults to `always`, so turbo keeps running all tasks after one fails, even those whose dependencies failed, and a rerun with it shows every task's result. ```mermaid %% caption: A green summary counts as evidence only after every task ran, skipped tests are named and the run is our own. flowchart TD A["Agent reports green"] --> B{"Successful equals total?"} B -->|"no"| C["Rerun with --continue"] B -->|"yes"| D{"Skipped tests named as untested?"} D -->|"no"| E["Run them against a real database"] D -->|"yes"| F{"Run found by our commit SHA?"} F -->|"no"| G["Look the run up by SHA"] F -->|"yes"| H["Green counts as evidence"] class H accent ``` Skipped tests get named as untested, with the reason they skipped. In our database package that usually means no Postgres or a missing switch, and the answer is to run them against a real database on its own port. Anything that reads CI state has to find its run by our commit SHA, never by it being the newest one. The final gates would run through the root scripts, unfiltered, with the full output in a file and the exit code kept. A `tail` or a token filter is fine while exploring, and it has no place in the run that signs off a change. ```bash pnpm test --continue > test.log 2>&1; echo "test exit=$?" grep -E 'Tasks:|Failed:|skipped' test.log pnpm format:check ``` ## Ask the agent for run, total and skipped counts The cheapest change is in the report itself. Our repository's agent guidance already tells agents to state plainly which checks ran, failed or couldn't run, and an unavailable service never counts as a pass. We'd go one step further and ask for tasks run, tasks total and tests skipped as separate numbers, with each skipped group named and its reason given. When the agent says it couldn't verify something, ask it what's stopping it. On 21 September the answer took under a minute, and 424 tests that had been switching themselves off ran and passed. ## Frequently asked questions ### Is this a turbo problem or an agent problem? It's mostly an agent problem, because turbo printed both numbers and the exit code came from the agent's own pipe. The agent read the first number and wrote it into its summary. Any tool that reports successes and totals as separate counts leaves the same gap, so both numbers belong in every report. ### Should skipped tests fail the build? Not on a laptop, where a missing database is normal and skipping is the honest outcome. They should block the sign-off when the skipped group covers the change, as the race test and the to-do query tests did on 21 September. In our setup a CI job runs migrations against a real Postgres, and the local report still has to say which tests it didn't run. ### Doesn't a second model reviewing the diff catch this? Only if it refuses to vouch for what it didn't run. Our review gate read the same branch that day, and its second pass reported no material findings. It still returned needs-attention, because it couldn't run the tests in its read-only sandbox and wouldn't count a failed run as validation. A reviewer that only reads the diff can judge whether a fix is right, and it can't tell which tests executed. ### What should an agent say when a rule stops it from fixing the environment? It should name the rule and ask. Our rule against starting containers unasked is right on a machine that runs more than one project, and the report that day should have read "the database container is stopped and I'm not allowed to start it". That sentence gets a one-word answer from a person, where "the database is unavailable" ends the check.