This article from Epoch AI explores the current limits of AI in software development, asking what the largest software project an AI can complete autonomously is. It likely discusses recent advancements and benchmarks in AI coding capabilities.
SynthePulse Insight · AI deep reading
Version 1 · 1 source
Epoch AI and METR have jointly launched the MirrorCode benchmark to test AI's ability to rebuild complete programs end-to-end without source code or internet access. Results show AI can handle some long-horizon tasks, but data contamination and generalization issues remain.
MirrorCode is a benchmark developed by Epoch AI and METR to test AI's capabilities on long-horizon coding tasks. Unlike common benchmarks (such as fixing bugs or implementing a single feature), MirrorCode requires AI to rebuild an entire program end-to-end in a sandbox without source code, internet access, or cheating avenues, and the output must exactly match the original program on end-to-end tests, including hidden tests.
The benchmark includes 25 target programs covering Unix tools, data serialization and querying, bioinformatics, interpreters, static analysis, cryptography, compression, and more. Task sizes range from 'small' to 'large', with the largest task costing $2,600 per run and requiring 19 days of continuous AI work.
Despite the extreme challenge, AI has solved some MirrorCode tasks. For example, Claude Opus 4.7 rebuilt gotree (a bioinformatics toolkit with about 16,000 lines of Go code and 40+ commands) in 14 hours at a cost of $251. It is estimated that a human engineer without AI assistance would need 2–17 weeks to complete this task.
However, note that the best AI implementation passed 2000 out of 2001 tests, failing only on one edge case, so it is not strictly 100% complete but is considered 'near-perfect'.
Since the target programs are open-source projects, AI may have seen the original code during pretraining, potentially inflating scores. However, AI successfully rebuilt several target programs that passed memory screening, while failing to rebuild programs that showed signs of memorization, suggesting results are not primarily driven by memory, though memory contributions cannot be fully ruled out.
The authors believe that the capabilities measured by MirrorCode may generalize to unseen codebases, but this conclusion remains inferential.
MirrorCode is not fully solved. The leaderboard uses 15 target programs (Medium and Large buckets), each implemented in two languages (usually Go and Ada), totaling 30 tasks, each with a budget of 10 billion tokens, run three times.
Epoch AI has open-sourced 22 target programs (132 task instances in total, covering six programming languages), retaining 3 targets as private test sets.
This report is based on the official Epoch AI page, a first-party source. All data (such as costs, times, and test pass counts) come from that page, but some conclusions (such as generalization capability) are author inferences, and data contamination issues are not fully ruled out.
MirrorCode shows that AI can independently complete some long-horizon coding tasks, but data contamination and generalization issues require careful consideration.
Primary report
Primary source