Daxiao Robotics and partners release HSImul3R, a simulation-ready human-scene interaction reconstruction framework accepted to ECCV 2026

3D reconstruction, human motion estimation, and human-scene interaction modeling have continued to advance in recent years, but traditional methods mostly take geometric accuracy, pose recovery, and visual alignment as their main criteria, answering the question of "how much the reconstructed human and scene look like the real thing." The problem is that what a robot ultimately faces is a physical world governed jointly by gravity, collision, friction, and contact: a visually highly credible scene may immediately fail once it enters a simulator, a chair may fail to stand because of missing structure, and the human and objects may also penetrate each other.

HSImul3R's entry point is precisely this "perception-simulation gap." According to this source, the team defines it as the first framework for stable, simulation-ready human-scene interaction reconstruction from uncalibrated sparse-view input, and it further supports monocular video input; the work has been officially accepted to ECCV 2026. It no longer only requires the human and scene to align in the frame, but incorporates gravity stability, real contact, and interaction stability into the core evaluation criteria.

What needs to be distinguished is that the above "first" and "Simulation-Ready" are the team's self-described positioning, and the source does not provide independent third-party verification; the ECCV 2026 acceptance and the existence of the paper, project, and code links are factual information stated by the source.

HSImul3R does not first complete reconstruction and then use a simulator to verify it; instead, it proposes a physics-in-the-loop bidirectional optimization mechanism that lets simulation feedback directly participate in reconstruction: the forward process optimizes the human, and the reverse process corrects the scene.

The forward scene-targeted reinforcement learning adds scene interaction constraints on top of motion tracking, using the spatial relationship between key human contact points and object surfaces so that the optimized motion is both faithful to the original observation and able to maintain correct, stable contact with the current scene. The counterexample given by the source is that after ordinary motion optimization, a "sitting on a chair" motion may cause the human to deviate from the chair in order to maintain balance—the person does not fall, but the interaction of "sitting on the chair" has disappeared; after removing this module, unexpected object displacement appears in simulation and stable interaction is difficult to achieve.

The reverse Direct Simulation Reward Optimization (DSRO) targets another kind of failure: simulation failure does not necessarily mean "the person moved wrongly"; it may also be that the scene itself was not reconstructed correctly. In scenes with severe human occlusion and thin structures such as table legs or chair legs, image-based 3D model generation is prone to missing structure or geometric distortion. DSRO directly uses the interaction results of physics simulation as a supervision signal to reversely optimize the 3D object generation model, raising the evaluation criterion from "can the object stand on its own" to "can it remain stable after a human interacts with it." The source divides simulation feedback into four states, from "the object is unstable under gravity" to "the human and object achieve stable interaction"; only when the object is stable on its own, remains stable after interaction, and the human and object indeed form effective contact is the reconstruction considered truly valid.

To validate this evaluation system, the team built HSIBench for human-scene interaction. According to this source, the dataset contains 19 scene objects, over 50 human motion sequences, and 300 independent interaction instances, completed by 3 participants, with each case synchronously captured from 16 viewpoints. Unlike traditional benchmarks that focus more on geometric error, HSIBench takes human-scene interaction stability (Stability-HSI) as one of its core metrics, not only judging whether an object can be stable under gravity, but also requiring the entire interaction to remain stable after entering simulation and to preserve meaningful human-object contact.

Experimental results show that in the Easy, Medium, and Hard tiers, HSImul3R's human-scene interaction stability rates reach 53.68%, 30.56%, and 13.92%, respectively, while HSfM reaches 10.52%, 4.50%, and 2.66%, respectively; the human-scene 3D penetration rate also dropped from HSfM's 69.51% to 22.90%.

These numbers need to be interpreted cautiously. They come from this source's retelling, with no independent reproduction or third-party evaluation provided; and even by the source's own account, the Hard-tier stability rate of 13.92% shows that in high-difficulty scenes most reconstructions still cannot interact stably in simulation. What can be confirmed is that the relative gap is clear; what cannot be confirmed is that the absolute capability is already close to usable.

If it stopped at stable physics simulation, this work would still only have completed the transfer from the real world to simulation (Real-to-Sim). According to the source, the team further retargeted the physically optimized human motions to a Unitree G1, then used these motions as priors to train a whole-body control policy in simulation, and finally deployed it directly to a real G1, enabling the robot to reproduce the corresponding human-scene interaction in the real environment.

Thus, the behavior in a human video is described as completing a full transformation: recovering the 3D human-scene relationship from images or video, correcting the parts that are "visually plausible but physically impossible" through physics simulation, then converting human motion into a motion prior that a robot can learn, and finally having the real robot re-execute it. On this basis, the source proposes a more imaginative path: the internet already contains a huge scale of human behavior videos, and if the human-scene interactions in them can be continuously reconstructed, physically validated, and transferred to different robot embodiments, these videos could become an important source for robots to learn real physical skills.

But this pipeline is currently still a directional claim rather than a verified conclusion. The source itself also admits that this route is still at an early stage, and that complex interactions, multi-object scenes, and dynamic environments still present numerous challenges; the source does not disclose the success rate, task scope, or failure cases of the Sim-to-Real deployment, so "converting human videos into robot skill assets" should be regarded as the team's vision, not a capability already delivered.

From the verifiable information, HSImul3R's contributions are concentrated in three points: incorporating physical executability into reconstruction evaluation criteria; using a physics-in-the-loop approach to let simulation feedback participate in optimization rather than only serving as a final check; and using HSIBench to turn "can it interact" into a quantifiable metric. These three points are logically self-consistent in the source's narrative and consistent in direction with the comparison numbers given.

The uncertainties are equally clear. First, all performance numbers come from a single source's retelling, with no third-party reproduction; second, the source does not state whether HSfM's configuration is under the same conditions as HSImul3R, so the fairness of the cross-method comparison cannot be judged from the available material; third, HSIBench is limited in scale (19 objects, 300 interaction instances, 3 participants), and its coverage of real-world diversity is unknown; fourth, the robot transfer part lacks quantitative results.

Therefore, a more prudent judgment is: this work proposes a valuable problem framework and a set of testable metrics, and shows a relative advantage on its self-built benchmark; but whether it truly bridges the gap between perception and simulation still requires independent verification and testing on larger-scale scenes.