A research team reproduced 2,200 papers from ICML, a top machine learning conference, and shared insights from the process. The effort highlights the challenges and importance of reproducibility in AI research.
SynthePulse Insight · AI deep reading
Version 1 · 1 source
A reproducibility hackathon involving 1,221 community members, which verified one-third of ICML 2026 papers claim by claim, reveals the true landscape of AI research reproducibility: 51% of papers had at least one claim verified, 23% were disputed, and human reviewers may have missed critical errors.
ICML 2026 received 23,918 submissions and accepted 6,352, nearly double the previous year, showing exponential growth partly driven by AI agents accelerating experiments and writing. However, review capacity has not kept pace; reviewers are mostly volunteers who may lack time or expertise.
One reviewer of an accepted spotlight paper self-reported: "My low confidence score was because I didn't carefully check all the proofs." The paper was later found to contain errors during reproduction, highlighting the limitations of the traditional review process.
AI agents (such as Claude Code, Codex, Cursor, and Pi) can now read papers, write code, run experiments, and report results, making large-scale reproduction possible. This was the starting point of the hackathon.
From July 15 to August 2, 2026, Hugging Face organized the ICML 2026 Open Reproductions challenge. Organizers indexed 6,341 papers and extracted core scientific claims. Participants brought their own agents (e.g., Claude Code, Codex, Cursor, orx) to reproduce papers, with multiple independent reproductions of the same paper encouraged.
Each run generated a Trackio log containing code, artifacts, and optionally the full agent execution trace, ensuring the audit process itself is auditable. An automated log judge (based on the open-source model GLM-5.2) gave each claim a verdict of "verified," "falsified," "toy-scale," or "uncertain," explicitly treating the log's self-assessment as untrustworthy.
Participants received $20 in compute credits, and 2,962 cloud tasks were launched. When full reproduction was impossible (e.g., proprietary datasets), participants ran toy reproductions with synthetic data. Ultimately, 1,221 members joined, publishing 6,816 logs covering 2,226 papers (34%), with 35,908 claims adjudicated and 274 full agent trajectory datasets made public.
Aggregating claim-level verdicts by paper: 51% of papers (1,103) had at least one claim independently verified, with 266 fully reproduced (all claims verified) and 632 partially reproduced without falsification. In total, 3,978 claims were confirmed through real experiments.
23% of papers (496) had at least one claim falsified or disputed, including 49 fully falsified and 242 where independent teams reached contradictory conclusions on the same claim. This shows reproducibility is not binary but adversarial.
Of the remaining papers, 502 had only toy-scale evidence, and 280 could not be adjudicated due to missing artifacts (the most common reason).
Some papers performed well, such as "Flat Minima and Generalization," which was reproduced by 20 independent teams, with 12 verifying all claims; and "A Coin Flip for Safety," where 14 of 17 logs verified all claims, ironically given the paper's topic that LLM judges are unreliable.
35 participants formally claimed to have falsified something, and organizers adversarially re-verified all claims. Confirmed falsifications include: a learned paging paper claiming robustness of H_k + O(1), but measurements showed the additive term growing like 0.38 ln k, extending to k=1,024 confirmed a deviation of about nine sigma, with the true form being H_k + Θ(log k).
Another case is the "Attention's forward pass and Frank-Wolfe" theorem, which claimed token particles collapse to the origin when the origin lies inside the convex hull, but three independent teams found counterexamples, the earliest appearing at t=224 steps, explaining why finite-horizon checks would stop prematurely. The authors confirmed and are fixing it the same day.
In the paper "Self-Distillation Enables Continual Learning," the central equation and theory section analyze reverse KL divergence, but the released code's default settings (which according to the authors produce all results) compute forward KL. The log that found this issue also failed to reproduce the paper's claimed +4pp result.
These cases show that AI-agent-driven reproduction can uncover errors that human reviewers might miss, but the reproduction process itself is also challenging, such as the limitations of finite-horizon checks and theory-implementation mismatches.
The challenge demonstrates AI's potential in scientific verification, but also reminds us that reproducibility requires adversarial review and transparent logs.
This report is based on the official Hugging Face blog (published August 13, 2026), a first-party source. All numbers and events come from this source, with no external information introduced. Some conclusions (e.g., 'AI agents can find errors humans miss') are inferences based on cases, as noted in the text.
The ICML 2026 reproducibility challenge shows that AI agents can perform paper reproduction at scale, but the results are adversarial: 51% of papers had verification, 23% had disputes. This demonstrates both AI's potential in scientific verification and the shortcomings of the current review system, as well as the need for careful interpretation of reproduction results.
Primary report
Primary source