Model Bench — A3

MODEL BENCHMARK — LLM PERFORMANCE ON REAL BUG REPAIR

10 models × 13 real bugs = 130 attempts · judged by the project's test suite · 2026-07-28

1 · Background & problem

This project uses language models as repair assistants for real defects. Public leaderboards rank models on curated puzzles — not on any particular codebase or bug class.

Problem: a leaderboard cannot tell you whether a model can repair your codebase's bugs, how fast, or at what cost — so model choice defaults to reputation.

2 · Gaps & countermeasures

gap	countermeasure
no ground truth for "model X can fix our bugs"	13 real historical bugs are re-introduced; the test suite alone judges each fix
multi-step agent loops make comparison unfair	one completion per attempt, identical prompts, 300 s limit
claims are unverifiable later	every attempt is archived: prompt, raw reply, seconds, dollars
results go stale as models change	the benchmark re-runs whenever the model set changes

BEFORE · model choice rested on public leaderboard reputation · repair ability on this codebase, speed, and cost were unknown

AFTER · 130 attempts measured and archived · model choice is backed by evidence · total cloud cost of the study: ~$9 · one model repaired all 13 bugs

3 · Results — the evidence

bug (each is a real shipped defect)	Llama 4 Scout	gpt-oss 20B	Gemma 4 26B	Qwen 3.6 35B	Devstral 24B	gpt-oss 120B	Claude Haiku 4.5	Claude Sonnet 4.6	Claude Opus 4.8	Claude Fable 5
1 rout threshold regression	–	✔	✔	✔	✔ˡ	✔	✔	✔	✔	✔
2 missing null guard (crash)	–	✔	⚠	–	⚠	✔	✔	✔	⚠	✔
3 dead squad never pruned	–	✘	✔	✔	✔	✔	✔	✔	✔	✔
4 phase reset leak	–	✔	✘	✘	✘	✔	✔	✔	✔	✔
5 wrong status glyph	–	✘	✔	✔	⚠	✘	✔	✔	✔	✔
6 charge range stuck	–	✘	⚠	–	–	✔	✔	✔	✔	✔
7 highlight ignores fog	–	✔	✔	✔	–	✔ˡ	✔	✔	✔	✔
8 command points flat	–	✔	✔	✔	✔	✔ˡ	✔	✔	✔	✔
9 battle result never saved	–	✔	✔	–	–	✔	✔	✔	✔	✔
10 morale clamp disabled	–	✘	✘	✘	✘	✘	✘	✘	✘	✔
11 leader-death rout unwired	–	✔	✔	–	–	✔ˡ	✔	✔	✔	✔
12 turn skips await (race)	–	✘	✘	✘	✘	✘	✘	✘	✔	✔
13 hp columns skip swap	–	✔	✔	✔	–	✔ˡ	✔	✔	✔	✔

Each cell is one repair attempt: ✔ repaired — all tests pass ✔ˡ repaired, with unnecessary extra edits ✘ patch did not fix the bug – no usable patch produced ⚠ model error or timeout

Bugs 10 and 12 (highlighted) are silent logic deletions — nothing crashes, the behavior simply stops. They defeated every model until the newest frontier tier: bug 12 fell to Claude Opus 4.8 and Claude Fable 5, bug 10 to Claude Fable 5 alone.

Bugs repaired (of 13) vs average speed — dot size reflects total cost

average seconds per attempt — fast is up

20 s — fast 87 s — slow

◉ best corner: accurate + fast

Claude Fable 5 $4.76

Claude Opus 4.8 · $2.35

Claude Sonnet 4.6 · $1.35

Claude Haiku 4.5 · $0.61

gpt-oss 20B free

gpt-oss 120B free

Qwen 3.6 35B free

Gemma 4 26B free

Devstral 24B

Llama 4 Scout — 0 usable patches

0 bugs repairedbugs repaired →13/13

filled orange dots run in the cloud · outlined dots run locally for free

4A · Resolved: model selection is now a measurement

Best free local model: gpt-oss 120B — 10 of 13 bugs at $0 · best paid value: Claude Haiku 4.5 — Sonnet-level accuracy at 45% of the price · only perfect score: Claude Fable 5 — all 13 bugs, including both silent ones, for $4.76 a run

4B · Unresolved: what the hardest bugs require

The silent-deletion bugs yield only to the newest frontier models — the open question is the cheapest added context (for example, an execution trace) that lets smaller models solve them · Llama 4 Scout produced no applicable patch in 13 attempts (malformed diffs) — patch discipline is part of the task · the bug library grows as the project ships new systems