AI Takes a Bare Exam: Over a Dozen Institutions Release New Benchmark to Gauge Autonomous Scientific Research

ASI-Bench, developed by Tsinghua University Assistant Professor Chen Yongchao in collaboration with MIT, Harvard, CMU, USTC, Microsoft Research, and over ten other institutions, aims to measure how much real scientific work AI can autonomously complete without human methodological guidance. Unlike benchmarks with fixed task descriptions such as HLE and Terminal-Bench, ASI-Bench treats the degree of human guidance as a variable, designing four conditions (B1 to B4) for each task: B1 provides complete methods, formulas, and steps; B2 retains only method hints; B3 gives only research objectives and data; B4 adds irrelevant distracting information to B3.

To ensure task quality, the team selected 60 research projects covering 11 disciplines from over 1,300 candidate questions, after 5 rounds of more than 1,100 human reviews and over 2,000 revisions. During development, more than 1,500 sandbox runs were conducted to check reproducibility and information leakage, with a cumulative investment of over 31,000 human-hours.

Experimental results show that across 18 model-agent combinations, the average score is 50.91 at B1, drops to 29.10 at B2, further declines to 26.62 at B3, and is 26.99 at B4. The most significant change occurs from B1 to B2: merely removing specific implementation steps reduces the average score by 21.82 points, while removing method hints from B2 to B3 only decreases it by 2.48 points. This indicates that the biggest bottleneck for AI is not autonomously selecting methods, but rather the difficulty in filling in implementation details when operational steps are absent.

The best-performing configuration is Codex paired with GPT-5.6 Sol (ultra), scoring 71.78 at B1 and still 51.60 at B3, the only combination exceeding 50 at B3, though still about 20 points lower than B1. Additionally, increasing reasoning budget does improve autonomous research performance: GPT-5.6 Sol's B3 score rises from 40.86 to 51.60 when moving from xhigh to ultra.

Computational costs show a counterintuitive pattern: B1 has the lowest cost, averaging about 4.35 million tokens and 37.8 minutes per task; B2 rises to the highest, averaging 6.91 million tokens and 49.7 minutes, an increase of 59% and 32% over B1, respectively. This is because AI, constrained by methods but needing to fill in details itself, engages in more exploration. B3 and B4 consume about 5.4 million and 5.7 million tokens, respectively, which is lower than B2.

Agent frameworks significantly affect performance: MiMo V2.5 Pro scores 16.17 with MiMo Code, but rises to 23.25 when switched to Claude Code; Kimi K2.7 improves from 19.72 to 27.34. This shows that research capability is a joint result of model and system, not determined by the model alone.

ASI-Bench can serve as a diagnostic tool: a large drop from B1 to B2 indicates the system relies on complete procedures, a large drop from B2 to B3 indicates method selection is a weakness, and the difference between B3 and B4 reflects resistance to distraction. Currently, the benchmark is far from saturated, with B3 averaging only 26.62 and the highest at 51.60, so future improvements under low-guidance conditions can serve as progress references.

The research team acknowledges that real science is far more complex than 60 tasks, so ASI-Bench is designed as a continuously updated open benchmark, with plans to add new disciplines and tasks. However, a single evaluation cannot answer whether AGI or ASI has arrived; it merely turns 'can AI do research on its own' into a comparable metric.