Token usage cut by 74.9%: Real2Gym turns simulation experience into reusable real-robot skills

Multimodal models can already call modeling tools to generate detailed 3D scenes, but robot manipulation is especially sensitive to geometric errors: a slightly narrow cup rim, a shifted object, or a container interior sealed off by a collision model can all cause actions that seem reasonable in the visuals to fail during physical execution. Real2Gym's starting point is exactly this gap—looking realistic isn't enough; the scene must actually be operable.

To this end, Real2Gym treats "whether an action can be completed in a physics engine" as part of scene acceptance. The system must not only preserve the objects, layout, and viewpoints from the video, but also enable the robot to actually complete grasping, placing, and contact. This acceptance criterion runs through the entire construction pipeline from video to environment.

The first step is to constrain the scene with geometric cues. For multi-view videos, Real2Gym uses geometric cues provided by Pi3X, combines SAM2 segmentation to associate objects across different views, constrains their scale and position, and then builds an editable Blender scene and corresponding MuJoCo environment. For single-view input, a corresponding geometric initialization scheme is used.

The second step centers on interaction error correction. The system focuses on moments when interactions change: when an object is grasped, when it contacts a support surface, when the gripper releases. Around these keyframes, the Agent self-checks viewpoint, layout, contact, and appearance, and makes local corrections using adjacent frames. Taking cup pouring as an example, even if the tabletop and object are already aligned, a dimensional error near the cup rim can still cause clipping; keyframe checking concentrates corrections on the regions that truly affect manipulation, avoiding judging reconstruction quality from a single overall rendered image.

The corrected actions are executed in MuJoCo to check task completion as well as support, contact, and stability. The execution trajectory is then exported back to Blender and compared with the original video to verify visual correspondence and physical feasibility. Validated scenes can also vary object positions, table height, materials, background, lighting, or distractors; changes that affect the motion space require adapting actions and re-validating, thereby providing the Agent with more practice-ready task environments.

With a practice ground in place, the Agent still needs to explore efficiently. Real2Gym elevates the goal of each decision round from generating a single action to generating code that executes one manipulation stage. For example, one response can arrange approaching the cup, closing the gripper, lifting, and checking whether the grasp is stable, organizing multiple actions and verification conditions together to reduce the number of repeated model requests.

When translating task intent into executable actions, SAM3 provides object localization cues and GraspNet provides grasp candidates. After execution, the Agent organizes successful steps, failure causes, and recovery methods based on feedback to form reusable skills. The experience describes how to act relative to objects, when to check, and how to correct after failure; when reused, it must be re-localized and adjusted according to current observations.

Self-improvement here is achieved by updating the skill library, while the underlying model weights remain unchanged. Each new task starts from an independent context and receives selected skills, so the effect of the experience itself can be tested, rather than attributing improvement to changes in model weights.

The team selected 24 scenes from DROID and EgoDex for evaluation, covering robot demonstrations and human manipulation videos. In reconstruction evaluation, Real2Gym outperformed the compared GPT baselines on content alignment, viewpoint alignment, action fidelity, and simulation success scores; qualitative comparisons show it better preserves cabinet structure, relative object sizes and positions, and the original camera viewpoint.

In the Agent evaluation within these environments, after reusing first-round experience, the second execution achieved an average success rate of 87.5%, compared with 70.8% for the GPT-6 Astra Direct Mode baseline; policy execution Token usage decreased by about 74.9%, and the average number of model responses dropped from about 31 to about 14. The paper also presents a case in the bowl-stacking task where Real2Gym succeeded and two GPT baselines failed, as well as more direct cabinet manipulation and adapter placement after experience reuse.

It should be noted that the above overhead statistics target policy execution and do not include scene construction and separate skill extraction. These numbers all come from the team's own account; the source does not explain the statistical methodology or number of repetitions, and third-party independent verification is lacking.

On the Franka robotic arm, the team tested four tasks: placing blocks into a tray, hanging a cup, putting a cup into a microwave and closing the door, and placing a plate onto a rack. In 3 attempts each for the first three tasks, Real2Gym succeeded in all; the GPT-6 Astra baseline succeeded 1, 3, and 1 times respectively.

Placing the plate onto the rack proved difficult for both methods: grasp depth, plate rim orientation, and placement precision in a narrow space must all be satisfied simultaneously. In the initial execution, neither method succeeded. The team then recorded human demonstrations and reconstructed an interactive digital twin. The Agent failed its first two attempts in simulation, obtained a usable skill on the third iteration, and then combined it with real-time observation on the real robot to complete the task that had originally failed.

This case demonstrates the complete process of starting from real demonstrations, improving skills through simulation exploration, and returning to the real world for validation. But it also suggests that success in this task depended on manually recorded demonstrations and multiple iterations, rather than being completed automatically in one pass.

Currently, point cloud noise still affects reconstruction, and stage-level control has room for improvement in fine manipulation; real-robot validation is limited to Franka and a parallel gripper. Extending to dexterous hands and humanoid robots still requires further experiments.

Although the paper and code are publicly available, the source does not provide reproducibility verification or community reproduction results; whether skills accumulated from simulation can generalize to untested robot embodiments and tasks also remains unclear. In addition, the source is QbitAI's paraphrase of the paper and does not provide the baseline configurations and experimental details from the original paper, so conclusions such as reconstruction scores outperforming GPT baselines, the 87.5% success rate, and the 74.9% Token reduction can currently only be regarded as the team's self-reported source_claim, not independently confirmed facts.