OpenAI found that by retaining reasoning processes and enabling context compaction, GPT-5.6 Sol's score on the ARC-AGI-3 benchmark jumped from 13.3% to 38.3%, roughly a threefold increase. This suggests that model performance on long-horizon tasks depends not only on the model itself but also on its runtime environment.
SynthePulse Insight · AI deep reading
Version 1 · 1 source
OpenAI found that by simply enabling two API settings, GPT-5.6 Sol's score on the ARC-AGI-3 benchmark increased by 188%, while output tokens decreased by 6x. This discovery reveals that benchmark scores do not purely reflect model capability, but are the result of the interplay between the model, test framework, and settings.
OpenAI posted on X that GPT-5.6 Sol has been used to solve open math problems, but struggled on the ARC-AGI-3 benchmark. ARC-AGI-3 tests a model's ability to learn unfamiliar 2D games without instructions. This contrast raises questions about the model's true cognitive capabilities.
OpenAI's investigation found that the standard test framework discards GPT-5.6 Sol's reasoning after each move and discards early actions when context fills up. This means the model must constantly restart, unable to leverage learned experience. This is not a lack of model capability, but a limitation imposed by the test framework design.
OpenAI implemented a new test framework using the Responses API and enabled two settings: retain reasoning and context compression. On the public dataset, GPT-5.6 Sol's score increased by 188%, while output tokens decreased by 6x. This result highlights the significant impact of API settings on model performance.
OpenAI points out that benchmark scores reflect not only the model but also the framework and settings used to run it. For long-running agents, retaining reasoning and compressing context allows the model to build on what it has learned. OpenAI hopes these experiments remind developers that evaluations rarely measure models in isolation, but rather measure a combination of implicit choices such as API settings, framework design, and prompts.
All information in this article comes from OpenAI's official X account's series of tweets, which is a first-party source. However, the tweet content itself is OpenAI's self-reporting and lacks independent third-party verification. The mention of 'solving open math problems' does not provide specific details, and its scope and significance cannot be confirmed.
GPT-5.6 Sol's tripled score on ARC-AGI-3 is not due to a sudden improvement in the model itself, but rather the release of its existing capabilities through optimizing the test framework. This case reminds us that benchmark scores should be interpreted cautiously; they reflect the combined effect of the model, framework, and settings, not a pure measure of model capability.
Primary report
Primary source
Also covered by 2 sources