Kaiming He's team releases VISTA visual memory framework, boosting ARC-AGI-3 scores

Kaiming He's team (MIT and Google DeepMind) published the paper "VISTA: A Visual Framework for Interactive World Reasoning," proposing a visual interaction and memory framework for multimodal agents. It did not train a new base model, nor did it pre-write rules for each game; it merely added a layer of framework outside existing multimodal models.

The result: after integrating VISTA, Claude Opus 5's score on ARC-AGI-3 rose from 40.68 to 100, and GPT-5.6 Sol rose from 13.33 to 99; both models ultimately completed all 25 public game levels.

But this perfect score is limited to the public game set. The research team points out that these public games predate the release of Claude Opus 5 and GPT-5.6 Sol, so the possibility that the relevant problems entered the training data cannot be fully ruled out; stricter generalization capability still needs further testing on a not-yet-public private test set. In addition, before VISTA, perfect or near-perfect solutions had already appeared on the ARC-AGI-3 public set, and many of them relied on program synthesis to build runnable world models.

It should be noted that the above evaluation results come from the research team's own paper and project homepage; they are first-party releases and have not yet been reproduced by a third party.

ARC-AGI-3 is an AGI benchmark launched this March and led by the ARC Prize Foundation. It continues the line of thinking of the ARC series proposed by François Chollet, examining a model's ability to adapt and reason when facing unfamiliar tasks.

Unlike the earlier ARC, which mainly consisted of static graphical problems, ARC-AGI-3 places the test in a dynamic interactive environment: after participants enter a never-before-seen pixel mini-game, there are no operating instructions and they are not told the goal; they can only figure out the rules through constant trial and observation of environmental changes.

It does not only look at whether the game is cleared; it also references human performance on first play and measures the number of environment operations the agent needs to complete the task. In other words, what it wants to test is whether, when ready-made knowledge cannot directly provide an answer, the system can quickly understand an unfamiliar environment through limited observation and trial and error.

An ARC-AGI-3 game is essentially a 64×64 colored grid. The minimal interface previously provided to language models converted colors cell by cell into numbers, ultimately turning them into a long string of text; a two-dimensional image in which position, boundaries, and shapes could originally be seen at a glance was thus broken down into 4,096 numbers.

VISTA reverses this process: instead of converting colors cell by cell into numeric text, it directly hands the game screen to the multimodal model as a PNG image, letting the model judge in visual space where objects are and what relationships they have with one another, and then decide the next operation.

The effect is obvious in the ablation experiments: merely changing the input from numeric text to directly viewing images raised GPT-5.6 Sol's score from 13.33 to 47.32.

Directly viewing images also saves context. The paper estimates that fully converting a 64×64 numeric grid into text requires about 4,000 tokens, while the same screen input as an image requires only about 300 image tokens. This estimate comes from the research team's own calculation and has not been independently verified.

The research team found that simply expanding the context window does not continue to improve scores: VISTA uses about 200,000 tokens of context by default, and after expanding it to about 780,000 tokens, token consumption increased substantially but scores did not improve. This comparison also comes from the team's own calculation.

VISTA's second core design is lossless visual memory: every frame returned by the environment is completely saved in chronological order, including intermediate animation frames triggered by an action. The model does not have to keep stuffing these images into the current context at all times; instead, it actively retrieves them when needed.

Specifically, the model can use inspect to actively call up screens from dozens or even hundreds of steps earlier, compare multiple historical images at the same time, or zoom in on a local area; if it needs to confirm subtle differences, it can also use read_pixels to directly read pixel information at a specific location.

In addition, VISTA prepares two text notes for the model: GUIDE.md records game rules that are already relatively certain, and WORKING.md stores the current level state, hypotheses being verified, and the next-step plan. When a round of context is about to run out, the model can first write key progress into the notes, then start a new context to continue the game, while previously saved historical screens and notes can still continue to be called upon.

The paper's ablation experiments give a relatively clear ladder: after adding direct visual input and notes, GPT-5.6 Sol's score rose to about 70; after adding lossless visual memory and active look-back, it jumped to above 94; finally, after adding precise pixel reading, it reached 99.

Putting this set of numbers together with "merely changing to directly viewing images yields 47.32," one can infer (this is an inference): what truly creates the gap is not visual input itself, but the ability to "look back"—the jump from about 70 to above 94 occurred after the memory and look-back mechanisms were added.

This also points to a different agent design approach: rather than continuously expanding the context window and letting the model always "keep all experiences in its head," it is better to completely preserve the raw experiences externally and retrieve the corresponding information when it is truly needed. VISTA itself does not train a new base model, nor does it pre-write rules for each game; what it changes is only the way the model observes, saves, and re-invokes information.

The paper also transfers the same framework to more environments: after re-rendering ARC games as scenes with three-dimensional perspective relationships, VISTA achieved 84.12; on GameWorld, which includes 34 browser games, AI GameStore, and the static visual reasoning benchmark BabyVision, its performance also generally surpassed versions that used the same base model but the default interface.

In the BabyVision test, merely giving the model the ability to actively zoom in on local details raised accuracy from 41.0% to 63.2%.

These results are only described by the paper as generally surpassing the default-interface versions of the same base model, and lack third-party reproduction; the effectiveness of lossless visual memory and active look-back mechanisms over longer cycles and in more complex real environments also remains unclear.

Perhaps what is more noteworthy about VISTA is not only this perfect scorecard, but an increasingly important question it exposes: are we measuring the capability of the model itself, or the capability of the entire system?

After entering the agent stage, perception interfaces, memory mechanisms, tool calls, and context management outside the model all directly affect final performance. VISTA did not change model parameters, yet greatly improved scores, making this question even more prominent.

In the real world, this will be even more obvious: whether it is a browser, a computer, or a future robot, the environment will not actively organize information into a form suitable for the model to understand; the agent must continuously observe, remember key states, and re-retrieve past information when needed.

Therefore, VISTA proposes a path different from continuing to expand model scale. But its boundaries are equally clear: the perfect score occurred only on 25 public games, whether the public games have already entered the training data of the two models cannot be ruled out, and generalization capability still awaits testing on a private test set.