Shanghai AI Lab and partners propose InfiniHand for world-coordinate hand motion reconstruction

In first-person video, the camera itself is moving. A hand moving to the right in the frame may simply be the wearer turning their head; object occlusion and two-handed interaction can also make key finger details disappear from view. This means that looking only at a local image of the hand cannot determine where the hand is in space—the same palm size may correspond to different distances, and local appearance alone can hardly explain camera rotation.

Traditional world-coordinate reconstruction usually splits the task across multiple systems: a detector finds the hand, a hand network estimates pose, a SLAM system tracks the camera, and then the results are aligned. Each step can go wrong, and detection jitter, trajectory drift and scale bias propagate to later stages; when processing large-scale video, the computation and coordination cost of multiple models is also hard to ignore.

InfiniHand's entry point is to share the geometric context these steps need: a pretrained LingBot-Map backbone extracts geometric features from the full frame, a localization module predicts the left and right hand regions from them, and a hand reconstruction module then fuses local appearance to fill in finger and joint details. Global features help determine spatial relationships, while local features help recover fine shape. This design idea is a paraphrase of the paper's method, and its actual advantage over multi-system approaches still rests on the paper's self-reported results.

How much video the model has seen matters, but wrong annotations also enter training along with scale. Different datasets do not agree on left/right hand order, coordinate definitions or annotation types; some annotations, when projected back into the image, do not even match the actual hand position and pose.

The team gathered about 5,000 hours of video from 9 public datasets including ARCTIC, HOT3D and EgoDex, unifying images, hand parameters, joints, masks and bounding boxes. Existing 3D keypoints were used to fill in missing MANO parameters, and sequences with camera trajectories additionally retained world-coordinate supervision.

During cleaning, the system uses Sapiens 2 to predict 2D hand keypoints in the image and then compares them with the projection of the original annotations; sequences that remain inconsistent are filtered out, and uncertain samples are left for manual inspection. This step removed about 30% to 40% of the candidate training data. The cost is clear: whether the filtered-out portion contains edge cases, and whether this affects the model's generalization to rare motions, the paper does not conclude.

Training on this basis has two steps: the first stage uses valid camera-space annotations to first learn hand localization and fine reconstruction; the second stage introduces camera trajectories and temporal supervision to jointly learn hand and camera motion. In this way, data under different annotation conditions can all participate in training, and fine gesture learning and global trajectory estimation each have their own focus.

Long-video processing faces a practical problem: how to use past information while controlling computation. InfiniHand continuously receives video in windows, retaining an initial reference frame, recent detail features and a compressed memory of earlier moments, so the model carries forward geometric context without reprocessing all historical frames each time.

At inference, two task paths run in parallel: one extracts backbone features for the next window, while the other decodes the hand and camera results of the previous window, reducing serial waiting. The paper reports a prediction throughput of 11.19 FPS, about 2.04 times that of HaWoR. For batch video annotation, this means more frames can be processed in the same amount of time.

It should be distinguished that this number refers to prediction throughput. For residual drift over long sequences, the system also performs sparse bundle adjustment in the background, correcting the camera trajectory through geometric relationships between keyframes; a binary keyframe pool retains both recent observations and some earlier frames, avoiding losing long-range connections by focusing only on the present. This optimization step still carries extra computational overhead, so 11.19 FPS does not equal the actual speed of the end-to-end pipeline.

Quantitative testing covers 34 ARCTIC, 10 HOT3D, 166 HOI4D and 100 EgoDex scenes. Measured by the PA-p metric, which considers both pose error and missed-detection penalty, the paper reports that InfiniHand achieves the lowest error among the compared methods on all four datasets.

The specific numbers are: compared with ViDiHand, the error on ARCTIC drops from 9.82 mm to 7.72 mm, a reduction of 21.4%; on EgoDex it drops from 17.22 mm to 7.29 mm, a reduction of 57.7%. After placing the hand into the world coordinate system, InfiniHand achieves the lowest global joint position error W-MPJPE among the compared methods on ARCTIC, HOT3D and EgoDex, with EgoDex dropping from 78.41 mm to 28.77 mm compared with Dyn-HaMR, a reduction of 63.3%.

The boundaries of these numbers are equally clear: they all come from the paper's self-report, with no third-party independent reproduction or verification yet; the relationship between 11.19 FPS and 2.04 times HaWoR is also based on the paper's self-reported comparison, lacking independent benchmarking. In addition, in the Ego4D natural-scene example shown in the paper, InfiniHand used external hand localization, which differs from the full system setting, so that example cannot be directly equated with the full system's generalization performance.

The model output uses the MANO parametric hand model, describing joint pose, hand shape and orientation with a set of parameters; auxiliary 2D keypoint prediction also constrains the projection of the 3D result in the image, making the recovered hand match its visible position. The model then combines the camera trajectory to transform each frame's hand result into the same world coordinate system, yielding motion trajectories that can be compared across frames.

This means the whole path depends on the MANO parametric representation and on the availability of camera trajectory supervision in the training data. The paper mentions that only sequences with camera trajectories additionally retain world-coordinate supervision, which implies that data conditions themselves limit which sequences can participate in the second-stage training.

The paper's endpoint is to turn human manipulation video into usable 3D demonstrations: after quality screening and robot morphology adaptation, these 3D motions are expected to be used for dexterous-hand data augmentation, motion transfer and imitation learning, expanding the sources of human demonstrations available for embodied training. This is an application outlook proposed by the paper, with no deployment validation yet.

This path still needs improvement: the system requires an additional model to recover physical scale, large motions and intermittent occlusion remain difficult, and supervision generated by monocular reconstruction may also introduce scale drift. Higher-quality, more diverse training data is seen by the paper as the direction for extending reliable annotation to more complex daily activities.

The paper was released as a preprint on arXiv on September 28, 2026, and the project homepage and code repository were made public. Author affiliations include Shanghai AI Laboratory, Shanghai Jiao Tong University, University of Science and Technology of China, University of Liverpool, Nanyang Technological University, the University of Hong Kong and Zhejiang University. Whether it has undergone peer review is not clear.