China's Orca world model matches specialized robotics systems without ever seeing a single action label

The current mainstream paradigm in AI is to predict the next token (language models), the next video frame (video generators), or the next robot action (control policies). Orca takes a different path: it predicts the next state of the world, represented in an abstract latent space. In the technical report, BAAI argues that intelligence should not be defined by specialized prediction models but should be based on a general understanding of how the world changes, then adapted to different tasks.

Orca's base model uses the pre-trained vision-language model Qwen3.5, with core parameters frozen after training. For each output type, researchers attach a lightweight module: text output uses Qwen3.5's native language head; image output passes internal states to Stable Diffusion 3.5 via a small adapter; robot actions are generated by a trained-from-scratch 'action expert' module. This modular design aims to demonstrate that a well-trained internal world state can serve as a shared foundation for vastly different tasks.

Orca's training integrates two learning modes. 'Unconscious learning' uses unlabeled raw video: the model sees an image and predicts the representation of the next image in the abstract space, learning motion patterns, occlusions, and typical scene dynamics. The training data includes 125,000 hours of video encompassing first-person daily interactions, third-person object manipulations, robot recordings without action labels, and natural scenes; only one-tenth of this data is used in the current version.

'Conscious learning' introduces language instructions. Videos are segmented into clips, each annotated with state change descriptions, enabling the model to learn how specific actions cause state transitions. Additionally, the model is trained on standard video question-answering tasks to maintain responsiveness to natural language queries. Both learning paths feed into the same internal world state representation, complementing each other.

Orca is trained in two sizes: 0.8B and 4B parameters. On text benchmarks, Orca-4B achieves an average score of 51.8% on MVBench, TemporalBench, 3DSRBench, and SWITCH, surpassing small vision-language models like Qwen3.5-4B, Gemma 4-4B, and DeepSeek-VL2-3B, as well as larger world models Emu3 (8B) and Emu3.5 (34B). However, in individual tests, Qwen3.5-4B still leads on MVBench.

For image prediction, BAAI constructed its own benchmark PRICE-V0.1 using real robot and first-person scenarios, requiring the model to generate resulting images based on instructions (e.g., 'Close the microwave'). Orca-4B averages 59.8%, surpassing FLUX.2 small (56.1%), FLUX.1-context (40.9%), and OmniGen2 (39.6%). Orca outperforms pure image models in preserving robot shape, object contact points, and instruction consistency, while the latter often introduce irrelevant objects or hallucinated hands.

Robot control experiments use a wheeled dual-arm humanoid robot performing five manipulation tasks: placing books on shelves, stacking bowls, scooping sugar, etc. Orca matches the performance of the specialized system π0.5, which is directly trained on robot action data. Orca's base model never saw action labels during pre-training; only later, for each task, it trained the action expert module using 200 real-world recordings (camera images and corresponding actions). In comparison, V-JEPA 2.1 and Qwen3.5 with the same control module lag significantly. Interestingly, Orca demonstrates error recovery: after a failed grasp, it retries, while π0.5 falls into repeated failures under the same circumstances.

The robotics field has long faced the challenge of insufficient annotated action data. Orca's core contribution is learning general state representations from large-scale unlabeled video via a world model, significantly reducing reliance on expensive action labels. Pre-training uses no action data at all, and only a small number of demonstrations (200 per task) are used during subsequent adaptation, achieving performance comparable to specialized systems. This provides a viable path for training robot foundation models using massive internet videos.

Training efficiency is also noteworthy. Using the in-house FlagScale library with memory and communication optimizations, BAAI achieves 2.91 training samples per GPU per second on H100, 4.4 times faster than StarVLA, a common codebase in robotics.

Orca currently only learns from images and text, missing multimodal signals such as sound, force, and touch. Visual prediction relies on the latent space of a pre-trained image encoder rather than learning world representations from scratch. The model parameter sizes (0.8B and 4B) remain too small for complete world modeling. Event descriptions cover only short time windows of a few minutes. BAAI states that the ultimate goal is to train a native world model from scratch that integrates multiple signal types.

Furthermore, there is no consensus on the definition of 'world model' in the academic community. The OpenWorldLib proposed by Peking University provides a unified framework that explicitly excludes pure text-to-video models like Sora. Benchmark tests from Tsinghua University show that even Sora 2 and Veo 3.1 struggle with physically plausible scene progression. Whether Orca's path represents the correct direction remains to be further validated.