Epoch and METR have released MirrorCode, a benchmark to evaluate AI systems on programming tasks that take humans weeks to complete. Results show that advanced models like Claude Opus 4.7 can solve some tasks in hours at high cost, but the hardest tasks remain unsolved. The benchmark highlights rapid AI improvement, with models from a year ago scoring only 30%.
SynthePulse Insight · AI deep reading
Version 1 · 1 source
Epoch and METR jointly release the MirrorCode benchmark to evaluate AI systems' ability to complete long-cycle programming tasks. Results show top models can now solve tasks that would take humans weeks, but nearly a third of target programs remain unconquered.
MirrorCode, released by Epoch and METR, aims to measure AI systems' ability to complete tasks that take humans significant time. Initially announced in April 2026, it has now been expanded and formally published. The test requires AI to reimplement a target program from scratch using only command-line interface access, without touching source code or the internet. The authors note that this requires designing the entire program structure, not translating code piece by piece.
Target programs include Apple's configuration language pkl (61,000 lines of code), phylogenetic tree tool gotree (16,000 lines), and CSV column selection tool qsv_select (87,000 lines), among others.
Test results are striking: Claude Opus 4.7 completed one task in 14 hours at a $251 inference cost, while METR and Epoch estimate the task would take humans 2 to 17 weeks. Both Opus 4.7 and GPT-5.5 successfully reimplemented gotree in multiple programming languages, costing between $100 and $400. Opus 4.7 even reimplemented the larger pkl program.
AI models have progressed rapidly: a year ago, leading models scored only about 30% and were limited to simple programs like calendar tools. Current models achieved at least one perfect score on 17 of the 25 targets, and near-perfect scores above 99% on another 4.
Despite impressive results, MirrorCode retains sufficient difficulty. Of the 25 targets, 8 never reached a 100% solve rate, and 4 never reached 99%. AI performed worst on the Python linter and formatter ruff, and also struggled with the mathematics package giac_subset and the email authentication library mailauth.
This indicates that current AI still has limitations in complex, domain-specific software reimplementation; the benchmark is not overly simple.
Import AI editor Jack Clark comments that MirrorCode not only demonstrates improved AI coding ability but, more critically, AI's self-direction: reimplementing external software from scratch using only input-output access. This suggests that intelligent AI agents might bootstrap their own form of industrial civilization through black-box contact.
This perspective places MirrorCode's results in a broader context, hinting that AI's generalization capabilities may far exceed current applications.
This article primarily sources information from Import AI's coverage of the MirrorCode benchmark released by Epoch and METR. All data and conclusions come from that report, with no external knowledge introduced. Some explanatory content (e.g., the extended meaning of 'self-direction') is inferred by Import AI's editor and has been noted in the text.
The MirrorCode benchmark shows that AI coding ability has achieved a qualitative leap, capable of completing long-cycle tasks that take humans weeks, with astonishing speed of progress. However, about a third of complex programs remain unsolved perfectly, indicating current AI bottlenecks in specific domains. More importantly, the self-direction capability exhibited by AI may herald broader autonomous intelligence.
Primary report
Primary source