Error Reports an Agent Can Act On

CI FOR AI-WRITTEN CODE — ERROR REPORTS A HUMAN CAN RULE ON IN MINUTES

0.4 s test loop · 248 tests in ~14 s · 13 real bugs re-injected as mutations · 2026-07

1 · Background & problem

In this project, AI agents write most of the code. The limiting resource is the human's review time and judgment.

Problem: standard CI reports failures as raw logs, and a passing test suite proves nothing if the tests are incapable of failing.

2 · Gaps & countermeasures

gap	countermeasure
raw logs require manual interpretation	failures arrive as a one-page visual report (A3)
slow feedback breaks the agent's workflow	a single test runs in 0.4 s; the full suite runs in parallel
a passing suite can be meaningless	mutation testing: 13 real bugs re-introduced; the suite must catch every one
costly operations fail only after the cost is paid	budget checks run first and refuse over-budget work up front

BEFORE · failures arrived as raw logs that required manual interpretation · there was no proof the tests could actually detect bugs

AFTER · every failure arrives as a one-page visual report · all 13 re-introduced bugs are provably detected · full suite: ~14 s · single test: 0.4 s

3 · Results — the evidence

CI shard timeline — 52 parallel shards, 248 tests, ~14 s wall time — **Full test suite, visualized:** each bar is one parallel shard of the 248-test suite. Total wall time is ~14 seconds, limited by the longest single test — a slow or failing test is immediately visible as a long bar.

Mutation catch rate and test-loop timing versus design targets — **Proof the tests work:** every historical bug is re-introduced on purpose (left grid), and the suite must catch each one. On the right, test execution time is measured against its speed target.

A complete one-page A3 report sheet — **What a finished report looks like:** a complete one-page A3 from this project — problem at the top left, evidence dominating the middle, decisions at the bottom. Test failures are delivered to the human in this same format.

mutation #12 · a race condition: s/await attack_phase(team)/attack_phase(team)/ — removing one keyword re-introduces a real shipped bug; the suite must detect it

4A · Resolved: the test suite is proven trustworthy

A single test runs fast enough (0.4 s) for an AI agent to test after every edit · all 13 historical bugs are demonstrably detected · failures are delivered as decisions, not logs

4B · Unresolved: the bug library grows with the project

Each new production bug is added to the mutation library once a test provably detects it · the same bug library also serves as a live LLM benchmark — see the model-benchmark article