CodeScene published a case study in which coding agents refactored 300,000 lines of C over three weeks for roughly $4,000 in tokens, verified by a frame-by-frame replay harness. The agents also built a playbook of codebase-specific recipes. Practitioners have questioned the scope, the metric, and how much of the result depends on the harness.
SynthePulse Insight · AI deep reading
Version 1 · 1 source
CodeScene published a case study claiming a coding agent refactored 300,000 lines of C code in three weeks, at a token cost of about $4,000, with the Code Health score rising from 5.6 to 10.0. The disagreement among practitioners is not whether the result happened, but what it actually proves.
CodeScene published a case study claiming a coding agent refactored a 300,000-line C codebase in three weeks, at a token cost of about $4,000. The work produced 2,903 commits, touched 726 files, modified 252,055 lines of code, and raised the codebase's Code Health score from 5.6 to 10.0. The codebase was Street Fighter III: 3rd Strike, taken from an open-source decompilation project.
CodeScene founder Adam Tornhill called it the first time in his thirty years of working on large systems that he had seen so-called "superhuman-level AI performance" at scale. But reactions from practitioners on LinkedIn were clearly split, with the debate centering on the scope of what the result can prove, not on whether the result happened.
Two mechanisms underpin this work. The first is the quality signal: the CodeHealth MCP Server gave the agent a deterministic score to optimize against and to judge whether a given transformation helped. The second is correctness: the replay-trace test harness compares rollback state hashes frame by frame, so that behavior can be checked after every change.
Some practitioners argue that most refactoring claims rely on a green test suite, and a test suite only shows that the tests survive; for a fighting game replay trace, the bar is much higher. This assessment is itself a practitioner view from the source, not independent verification.
More noteworthy than the score is what the agent accumulated along the way. Rather than applying a fixed catalog, it built up a refactoring playbook that ultimately contained 22 recipes and 82 supporting notes. The playbook includes common transformations such as Extract Function and Guard Clauses, as well as recipes specific to this codebase: Shared Index Range catches duplicate loops that differ only in their start and end ranges, Action Parameter handles duplicate control structures whose main difference is which function is called, and Uniform Step Table converts heterogeneous calls into table-driven dispatch.
Failed attempts were recorded too, including transformations that made Code Health worse. This is presented in the source as a process record, not as an outcome metric.
The team ultimately chose Claude Opus for most of the work, and said Claude Code with Opus was clearly better than Codex with Sol at catching and recording emerging patterns. When smaller models performed the tasks, files often hit a plateau, seemingly reaching a local optimum they could not break through.
These are the team's own claims and are source claims. Whether the plateaus encountered by smaller models represent a ceiling on model capability is not settled by the source.
Skeptics focused their fire on scope. Tech lead Konrad Otrębski proposed that the real test would be offering this kind of refactoring service to well-known open-source projects such as Grafana, with merging to master as the completion criterion. Daniel Webb, one of the engineers involved in the work, replied that the code had been merged into a fork's main branch through 54 pull requests.
This means the subject was open-source decompiled code, not revenue-generating production code; the code was merged into a fork's main branch, not the upstream main repository. The case study was published by CodeScene itself, and the results have not been independently verified by a third party.
Asko Nõmm raised a methodological question: because Claude Code and Codex are tuned for their respective models, it is unclear how much of the result measures the model and how much measures the test harness, and costs would vary the same way. He also noted that architectural quality was not measured, and that code looking healthy does not mean underlying problems will not surface later.
Denis Baltor questioned the recipes that were discovered, arguing that the DRY principle is about duplication of knowledge and intent, not duplication of identical lines of code, and that recipes differing only in start and end ranges or only in which function is called may conflate the two. Tracy Bannon considered describing the result as "perfect" rather bold, and objected to the framing itself.
Several unanswered questions came from the authors' side. Marc Bouvier asked whether non-functional performance had improved, given that frame rate, memory footprint, and input latency matter for a game; Webb said a performance expert would be brought in. Asked whether the test harness caught subtle frame-timing regressions, he said there may be no definitive answer, because it runs as a pre-commit hook and some failures were fixed before they were ever observed.
Putting these together, one can cautiously infer: this case study provides concrete evidence on the dimension of "can an agent rewrite code at scale under controlled signals," but it leaves gaps in production applicability, architectural quality, non-functional performance, and attribution breakdown. The disagreement exists because the two sides set different bars for "what is proven," not because they dispute the facts themselves.
The facts in this article come from InfoQ's reporting on the CodeScene case study. The case study was published by CodeScene itself, and the results have not been independently verified by a third party; the subject was open-source decompiled code, and the merge target was a fork's main branch. The team's claims about model superiority, plateaus, and the performance-expert arrangement are all source claims, not independently confirmed.
The hardest evidence this case study offers is that, constrained by two signals—deterministic scoring and frame-by-frame replay—an agent completed a large-scale rewrite of 300,000 lines of C code and accumulated 22 recipes and 82 supporting notes on its own. But what it does not answer is: how much of this result comes from the model and how much from the test harness, and whether it holds up for production code, architectural quality, and non-functional performance.
Primary report
Primary source