From Computer-Use to Robot-Use: Show-Harness Lets VLMs Directly Control Real Robots

The starting point of this work is a question: if the foundation model is given a physical interface it can understand, compose, and continuously correct, can its existing intelligence enter the physical world more directly? The source groups current approaches into two categories—traditional VLA has the model directly regress continuous control signals from images and task instructions, requiring repeated adaptation for different robots and data distributions; the other has the large model only handle planning, with execution by a separate controller, creating distance between semantic decisions and actual actions.

Show-Harness takes a third path: inserting a semantic action interface layer between the model and robot control. For the model, actions are readable, reason-able semantic units; for the robot, actions are converted by each interpreter into local control commands bounded by safety limits. The source emphasizes that the model-side interface does not change with robot morphology; when switching embodied platforms, what is replaced is the underlying action interpreter, not a requirement for the VLM to relearn an action language oriented toward joints and coordinate frames.

The reference point for this idea is Computer-Use: on a computer, an Agent does not need to output low-level driver signals but faces a clear, composable action interface like click, type, scroll, from which it can verify results through observation, forming an observe → reason → act loop. The source's analogy is that the real difficulty in robot control is not just "letting the model decide an action" but designing an interface layer that can carry visual and spatial understanding while being fine-grained enough to accomplish approach, alignment, grasping, placement, and correction.

The action space is a compact set of semantic action units: forward/back, left/right, up/down movement relative to the current reference view, rotation about a specific axis, grasp, release, or finish task. The source gives two key points: semantically understandable—the model need not directly output joint or end-effector pose values but decides around "what direction is the target in," "should the next step approach or move down," "is alignment already achieved"; physically executable—each semantic unit corresponds to a small, bounded physical change, mapped by each platform's interpreter to its own motion control targets.

Execution is not a one-shot output of a long plan followed by blind execution. Harness organizes multi-view images, robot proprioceptive state, short-term action history, and task description into a loop: the model observes the current state, decomposes or updates subtasks, makes a semantic action, and after execution continues correcting based on new visual and state feedback. The source calls this a perceive-reason-act loop, with perception, reasoning, and execution modules organized around a shared semantic interface.

GUMI (GUI-based Manipulation Interface) presents the same set of semantic action units as browser graphical controls and keyboard shortcuts, unifying three control modes: a human operator uses the keyboard to remotely "play" the robot without dedicated teleoperation hardware; a Computer-Use Agent operates the same GUI as if operating a webpage; a VLM robot agent directly predicts the same set of semantic actions without going through GUI clicks. The observation before each operation and the chosen action are recorded as (observation, action) data pairs, and the action interpreter simultaneously retains low-level commands and trajectories, so a single demonstration dataset can be used both to train an action policy based on Show-Harness and to be converted into data for a continuous-control policy.