VA-Bench Released: Large Models Show Execution Gap in Spatial Intelligence

VA-Bench's starting point is to stop spatial intelligence from remaining at "answering an already-defined spatial question." The research team placed it inside a complete "observe→reason→act→correct" loop, requiring models to actively judge whether current information is sufficient and decide where to look next. The key difference from traditional spatial evaluations is that what to look at and what to answer are no longer decided in advance by the evaluator on the model's behalf.

The constraints are matched to this. Before testing, models only watch successful demonstration videos to understand how the task is done, but receive no object coordinates, expert trajectories, or precise action parameters; during testing they can actively switch viewpoints, at the cost of consuming interaction steps, so they must trade off between continuing to observe and starting to act. The final spatial judgment must also land on actions: models must directly output specific movement distances and rotation angles, and decide when to open or close the gripper.

In scale, VA-Bench contains 14 categories of robot manipulation tasks, of which 11 are single-arm and 3 are dual-arm, with 20 physically validated scenarios per category, totaling 280 base scenarios. It also includes geometric transfer and long-horizon composition tests, asking respectively whether models can re-judge after a layout change, and whether several independent manipulations can be stably chained into a longer task.

VA-Bench tested 12 major multimodal models. The best-performing groups reached 100% on target recognition and localization (TL) and 99.6%-100% on manipulation semantic understanding (MS). Looking only at these two items, the conclusion would be that models already "understand" space.

But complete task success rate was only about half: Qwen3.8-max, Opus-5, and GPT-5.6-sol scored 53.93%, 52.86%, and 51.55% respectively. The same batch of models had subtask completion rates of 65.9%-68.6%, indicating that failures are often not a complete lack of knowing how to proceed, but rather having completed many steps yet failing to finish the entire task reliably.

From these numbers one can infer (this is an inference, not a source conclusion): the main bottleneck of current models is not in target recognition and semantic understanding, but in execution stages such as fine-grained spatial execution, online correction, and dual-arm coordination. The source mentions that a model may accurately find a bottle but close the gripper prematurely before it has lowered to the appropriate height; or it may understand that "the object needs to be placed into the container" but cannot reliably judge the precise height, depth, and contact position between object and container.