AdaRoboVLG: Task-Adaptive Vision-Language Grasping Framework for Different Robot Hands

Vision-language grasping (VLG) must simultaneously handle four types of challenges—spatial, cognitive, temporal, and embodiment: locating the target and avoiding collisions, understanding object function, tracking continuously changing states, and adapting to different robotic hands. The source points out that many end-to-end VLG methods directly map visual observations and language instructions to grasp actions, and their adaptability is often limited by training data and robotic hand configurations; facing new task situations and embodiments typically requires additional data or retraining.

AdaRoboVLG's approach is to decouple "how the task requires grasping" from "how the robotic hand grasps stably," using a unified grasping interface to connect task understanding with physical grasp synthesis: the upper level converts task situations into grasp constraints, and the lower level generates stable poses for different robotic hands accordingly. This interface consists of the target object geometry, the contact grasp representation (CGR), and robotic-hand-compatible grasp types; CGR describes contact position, probability, approach and closing directions, and grasp width, while grasp types come from human grasp classification and are filtered based on robotic hand kinematic capabilities.

According to the source description, the base policy first generates candidates for a specific robotic hand based on CGR and grasp types, then a shared model scores stability using HOI representations; integrating a new robotic hand only requires configuring the corresponding kinematic mapping and compatible grasp types. This is the technical anchor of the framework's "scalable" claim, but the source does not provide the actual engineering effort or time required to integrate a new robotic hand.

The spatial prior answers "can the robotic hand reach in." It uses multi-view RGB-D observations, lifts image features extracted by DINOv3 into 3D to form a scene-level semantic representation, then combines with point clouds to predict scene-level CGR via Contact-GraspNet, providing mappable contact candidates for different robotic hands. The Contact-GraspNet in this module is trained on GraspClutter6D. On DexGraspNet 2.0's three clutter levels and three robotic hands, AdaRoboVLG achieves an average success rate of 88.4%, obtaining the best results in the table on the Random and Loose splits, but the source explicitly states that on the Dense split, comparison methods still perform higher.

The cognitive prior answers "how should it grasp." The open-set functional reasoning evaluation contains 125 image-instruction pairs, 57 manipulation tasks, and 80 object categories; the source states that the base LLM's functional part and grasp type accuracies are 82% and 63% respectively, rising to 94% and 87% after adding RAG and CoT, with CoT improving functional part judgment more noticeably and RAG helping grasp type selection more. In language-guided target grasping tasks, AdaRoboVLG achieves three-hand average success rates of 81.3% and 86.0% on GraspClutter6D and GraspNet-1Billion respectively, and the source claims it achieves the highest success rates on both benchmark sets across three robotic hands.

The temporal prior answers "how to keep up." It combines SAM3 mask tracking with DINOv3 cross-frame feature consistency to estimate the target's rigid body motion, and updates the target geometry and CGR online; this module itself requires no additional training. The source states that on GraspNet-1Billion's Seen, Similar, and Novel splits, AdaRoboVLG leads on all three metrics—multi-grasp tracking accuracy, translation error, and rotation error; however, the source also notes that symmetric objects may still suffer rotational drift, and severe occlusion may cause tracking loss.

The training scale of the base policy is one of the supporting pillars of this framework: the research team used three robotic hands—DH3, Allegro, and Inspire—in Isaac Sim, collecting 4.4 million labeled grasp trials. The source states that the model trained jointly on three hands achieves grasp quality prediction accuracies of 82.95%, 88.02%, and 91.07% on DH3, Allegro, and Inspire respectively, and that compared with encoding hands and objects independently, HOI converges faster with lower training loss. These numbers come from the paper's self-reported simulation experiments and are research conclusions relayed by the source.

Real-robot experiments covered 102 everyday objects, including food, household items, tools, toys, electronics, and textiles; each trial required finding the correct target, satisfying the functional intent, avoiding collisions, and lifting the object 10 cm and holding it for 5 seconds without dropping. On the XMAN-R1 and TianJi Marvin platforms equipped with the ROHand dexterous hand, each object underwent 5 randomized trials, totaling 510 real grasps, with an overall success rate of 83.3%, and the system underwent no real-world fine-tuning. The source also states that the system was deployed to a parallel gripper to demonstrate cross-end-effector adaptability.

For dynamic scenarios, the experiments used an AgileX Piper equipped with ROHand, selected 16 representative objects, and demonstrated functional grasping on a conveyor belt moving at 2-5 cm/s, with the grasp pose updated online at 5 Hz. Subsequently, on the same robotic arm with a parallel gripper, robustness under human interference was tested; facing four types of interference—recovery after being moved out of view, rapid movement, continuous occlusion, and simultaneous translation and rotation—the source states the total grasp success rate was 89.7%. These are all team self-test results relayed by the source, with no third-party reproduction observed.

The source grounds "scalable" in two demonstrations: after adding a task allocation module, AdaRoboVLG can perform dual-arm sorting; after integrating a depth completion module, it can also handle transparent objects. The two extensions respectively change task organization and perceptual input, and the source states that neither requires retraining the base grasping policy. Thus, new capabilities need not all be written into the same policy, but can be integrated through modules and carried by the existing grasping foundation.

However, the source also lists the boundaries of modularity: if the upper level misidentifies the functional part, the lower level may generate a stable but functionally inappropriate grasp; inconsistent multi-view localization, depth reconstruction noise, and the absence of tactile feedback also limit reliability. Tactile closed-loop control, as well as pre-manipulation such as pushing and poking under severe occlusion, remain future directions. These are limitations stated by the paper itself, not externally verified conclusions.

From a broader systems perspective, the source's argument is that the scaling of robotic capabilities depends not only on model scale but more on the coordination among perception, reasoning, skills, and embodiment. AdaRoboVLG connects foundation model priors, reusable grasping skills, and different robotic hands, demonstrating the idea of driving capability expansion through capability composition. It should be distinguished that: the framework design and experimental numbers come from the paper and its project homepage, and this article is based on a secondary report, in which the success rates, accuracies, and leading-performance statements are all relayed by the source, with no independent third-party reproduction or benchmark cross-validation yet.