Back to feed
News Story
APriority75
OpenAI (X)
3 sources

GPT-5.6 Sol Solved Open Math Problems, But Struggled on ARC-AGI-3 Benchmark

GPT-5.6 Sol, an AI model, has been used to solve open problems in mathematics, but initially struggled with the ARC-AGI-3 benchmark of 2D puzzle games. Investigation revealed that the standard harness discarded the model's reasoning after each move and dropped earlier actions as context filled up. By enabling retained reasoning and context compaction via the Responses API, the model's score rose 188% while using 6x fewer output tokens, highlighting that benchmark scores reflect not just the model but also the harness and settings.

SynthePulse Insight · AI deep reading

GPT-5.6 Sol's ARC-AGI-3 Score Triples: Model Improvement or Test Design Issue?

Version 1 · 1 source

OpenAI found that by simply enabling two API settings, GPT-5.6 Sol's score on the ARC-AGI-3 benchmark increased by 188%, while output tokens decreased by 6x. This discovery reveals that benchmark scores do not purely reflect model capability, but are the result of the interplay between the model, test framework, and settings.

  • GPT-5.6 Sol had previously been used to solve open math problems, but struggled on ARC-AGI-3, a 2D puzzle game benchmark.
  • The standard test framework discards the model's reasoning after each move and discards early actions when context fills up, forcing the model to constantly restart.
  • By enabling two API settings—'retain reasoning' and 'context compression'—the score increased by 188% and output tokens decreased by 6x.
  • OpenAI emphasizes that evaluations rarely measure models in isolation, but rather measure a combination of implicit choices such as API settings, framework design, and prompts.
Open section navigationBackground: The Math Genius's Puzzle Predicament

Background: The Math Genius's Puzzle Predicament

OpenAI posted on X that GPT-5.6 Sol has been used to solve open math problems, but struggled on the ARC-AGI-3 benchmark. ARC-AGI-3 tests a model's ability to learn unfamiliar 2D games without instructions. This contrast raises questions about the model's true cognitive capabilities.

Problem Diagnosis: Hidden Limitations of the Test Framework

OpenAI's investigation found that the standard test framework discards GPT-5.6 Sol's reasoning after each move and discards early actions when context fills up. This means the model must constantly restart, unable to leverage learned experience. This is not a lack of model capability, but a limitation imposed by the test framework design.

Solution: Two Settings Yield a Threefold Increase

OpenAI implemented a new test framework using the Responses API and enabled two settings: retain reasoning and context compression. On the public dataset, GPT-5.6 Sol's score increased by 188%, while output tokens decreased by 6x. This result highlights the significant impact of API settings on model performance.

Implications: The Complexity of Evaluation

OpenAI points out that benchmark scores reflect not only the model but also the framework and settings used to run it. For long-running agents, retaining reasoning and compressing context allows the model to build on what it has learned. OpenAI hopes these experiments remind developers that evaluations rarely measure models in isolation, but rather measure a combination of implicit choices such as API settings, framework design, and prompts.

Credibility boundary

All information in this article comes from OpenAI's official X account's series of tweets, which is a first-party source. However, the tweet content itself is OpenAI's self-reporting and lacks independent third-party verification. The mention of 'solving open math problems' does not provide specific details, and its scope and significance cannot be confirmed.

Insight takeaway

GPT-5.6 Sol's tripled score on ARC-AGI-3 is not due to a sudden improvement in the model itself, but rather the release of its existing capabilities through optimizing the test framework. This case reminds us that benchmark scores should be interpreted cautiously; they reflect the combined effect of the model, framework, and settings, not a pure measure of model capability.

Sources for this version

  1. GPT-5.6 Sol has been used to solve open problems in mathematics. So why was it s…OpenAI (X)

Primary report

OpenAI (X)

Primary source

Same-event coverage

Also covered by 2 sources

机器之心T2OpenAI Reveals Secret: Two Settings Triple GPT-5.6 Sol's ARC-AGI-3 ScoreTHE DECODERT2OpenAI Claims GPT-5.6 Sol Beats Opus 5 on ARC-AGI-3