LIBERO-MAX benchmark released: 224,000 simulations test robot dynamic reliability

Imagine a robot putting a cup on the table into a tray. The robotic arm is already reaching toward the cup when a person nearby casually moves the cup a few centimeters away. For a human, it is just a matter of adjusting the hand position; for the robot, what it saw before was the old position, and the action it has already generated also points to the old position. If it keeps executing, the hand may reach toward an empty patch of table.

This small episode points to a problem robots must face when entering real life: the environment changes while the robot is acting, and the task still needs to be completed. Researchers from Stanford, CMU, MIT, UIUC, UT Austin and other institutions propose LIBERO-MAX, turning environment changes during execution into repeatable, comparable tests.

Many robustness evaluations already change conditions before the task begins, such as moving a cup to a different position, adjusting lighting, or changing the viewpoint, so the robot sees the adjusted scene from its very first look. But "the cup was on the other side from the start" and "the hand was halfway there when the cup was moved away" place different demands on the robot: in the latter case, the robot has already acted based on the old image, and may still have a string of unexecuted actions, so it needs to keep coping with the new situation on top of an existing execution process.

If the robot ultimately fails, how do you tell whether it was caused by the cup being moved away, or whether it simply could not grasp well to begin with? LIBERO-MAX arranges two executions for the same task. The two share the initial state, instruction, model random seed, and all actions before the change occurs. At the same critical moment, they are then made to undergo different continuations: Base adds no extra mid-execution change and the robot keeps executing; Dynamic adds one environment change, such as moving the target object, and the robot keeps trying to complete the task.

If the robot succeeds in Base and fails in Dynamic, one can identify that this change disrupted a task that could originally have been completed. Even if both fail, the result is retained, helping distinguish insufficient original capability from the impact of a mid-execution change.

This design is built on tasks from LIBERO-Plus and LIBERO-PRO, and constructs 8,000 paired cases and 8 types of mid-execution changes: target movement, container movement, camera viewpoint change, sensor noise, lighting switch, visual theme change, distractor addition, and obstacle insertion. The eight event types cover object position, visual observation, appearance and clutter, and path constraints.

The research team evaluated 14 current robot policies, covering VLA, hybrid architectures, and world action models (WAM), all using their respective released inference configurations. Each policy completed 16,000 executions, and the main evaluation accumulated 224,000 simulations.

The results show that all evaluated policies dropped in success rate, by 11.0-25.7 percentage points. Even the best performers under dynamic conditions, MolmoAct2 and π₀.₅, reached only 66.9% and 65.7% success respectively, losing many tasks they could originally have completed. For one of the evaluated policies, more than half of its original successful cases turned into failures after the change.

Paired evaluation therefore provides two kinds of information: how many tasks can still be completed after the change, and which originally completable tasks were interrupted by the change. The rankings of different model types also interleave, with no single model family consistently dominant. The currently evaluated world action models likewise have to face mid-execution changes, and whether world prediction capability can translate into reliable action still needs to be tested through actual tasks.

The losses brought by the eight change types are not the same. Target or container movement, as well as camera and sensor changes, caused more pronounced shared difficulty; lighting and visual theme changes usually had smaller impact.

This connects intuitively with the cup story: if the cup changes position, the original reach target needs adjustment; if the tray moves, the placement position needs updating; if the camera viewpoint changes, the robot again has to understand its relationship with objects from a new image.

This breakdown lets researchers further locate the problem: should visual recognition, target updating, or action planning be improved? After model improvements, the same type of event can also be used to check the effect.

A natural idea is: if the environment may change, then let the robot look at images more frequently and regenerate actions. Robot policies often predict a string of actions at once, then execute several steps before obtaining new observations, which can reduce model calls, but when a change occurs, actions generated from the old image may still be executing.

On the same set of 800 paired cases, the research team tested 4 policies and 18 effective action execution intervals. Adjusting the interval did indeed change performance, but under all tested settings, the success rate after the change was still lower than the corresponding Base. More frequent feedback did not automatically eliminate this gap.

The camera control experiment also found that the same viewpoint change produces different results when it appears in the first frame versus halfway through execution. Under mid-execution change conditions, all three tested policies succeeded more easily than when using the changed viewpoint from the starting point. This shows that evaluation also needs to pay attention to where the robot has already gotten to and which actions it has previously executed; facing the same new image, different execution histories also affect subsequent results.

LIBERO-MAX can also be used to test targeted improvements. The research team selected 300 fixed sensor noise cases and added an image quality detection module before X-VLA: when the current image quality is found to be poor, it is first repaired and denoised, then handed to the policy for processing. The policy needs no retraining, and the detection module is not notified in advance that "a change has occurred."

After quality-triggered repair, the success rate after the change rose from 40.7% to 47.7%, an increase of 7.0 percentage points, while the Base success rate remained basically unchanged. Always repairing also brought a 6.7 percentage point improvement.

This small-scale experiment shows that compensation measures targeting specific changes can recover some failures. Paired evaluation can then check both: how many perturbed tasks it improved, and whether it affected tasks that could originally be completed. However, whether this effect generalizes to other change types remains unclear.

LIBERO-MAX provides evaluation code, a fixed case list, and paired diagnostics. It also provides a Lite version containing 800 cases, reducing the number of executions to one tenth of the full version, making it convenient for researchers to do quick validation first and then carry out full evaluation.

After robots enter homes, offices, or factories, environment changes will keep occurring alongside task execution. LIBERO-MAX hopes to provide a reproducible test platform for change detection, action adjustment, replanning, and autonomous recovery, so that model improvements can face the same set of dynamic environment tests.

Returning to the cup that was moved away at the beginning: a reliable robot needs to still put it into the tray after the situation changes.