Anthropic's Opus 5 blows past Fable 5 and GPT-5.6 Sol on the benchmark designed to measure real intelligence

Anthropic's Claude Opus 5 achieved a score of 30.2% on the ARC-AGI-3 benchmark, nearly quadrupling the previous record of 7.8% held by OpenAI GPT-5.6 Sol (Max). The ARC Prize team attributes this lead to "stronger logical reasoning capabilities, enabling more autonomous exploration, planning, and execution in unfamiliar environments."

During testing, Opus 5 exhibited behaviors never before observed in AI models: it converted tasks into algebraic symbols and independently derived reflection equations. Additionally, Opus 5 solved five previously unsolved environments, four of which matched or exceeded human performance.

ARC-AGI-3 is designed to measure an AI model's ability to solve novel tasks not seen during training, which humans can typically handle easily. The current version uses a game-like format where the model must infer the rules of an interactive environment, plan actions, and execute them step by step. This tests general reasoning ability rather than stored knowledge.

Notably, the official score only accounts for the language model's performance without additional software (i.e., the "harness"). The ARC Prize believes that future AGI systems should not require external help to solve new tasks. Opus 5's score could be higher if used within Claude Code.

On Guanghan Ning's private benchmark Witness, Opus 5 scored only 43.4, statistically tied with Kimi K3 and Fable 5, showing a much smaller improvement over Opus 4.8 than on ARC-AGI-3. Ning notes that this pattern is consistent with training on specific types of data, but Witness cannot determine which data Anthropic used.

Greg Kamradt argues that a game based on familiar mechanisms does not test adaptability to novelty, and a weak result does not negate overall model improvement. Witness itself is designed around ARC-AGI-3-style puzzles, so better performance may reflect genuine transfer rather than memorization.

Ning later clarified that Opus 5 did generalize on Witness, but to a much lesser extent than on ARC-AGI-3. He likened the process to the evolution of coding benchmarks: as the primary target for interactive reasoning, ARC-AGI-3 naturally attracts the most training effort. Covering more edge cases may help models generalize to a broader range of abstract reasoning tasks.

Anthropic has not explained the performance improvement, but targeted data annotation and reinforcement learning are plausible factors. Unlike earlier models, Opus 5 was developed after ARC-AGI-3 and its format were made public, which may have allowed Anthropic to train on the benchmark's skills and puzzle format, though this does not imply training on exact tasks. Annotators could label reasoning trajectories, useful actions, failed attempts, and recovery steps for similar puzzles, while reinforcement learning could reward exploration, planning, rule discovery, and self-correction.