Oxford, Monash, and Bristol teams propose interactive video world model Oneira

Video world models can already keep generating new scenes as the camera moves, but the research team argues that such models have two recurring failures in interaction: when a specific object in the scene needs to be changed, the model often acts on the wrong object; and even when the interaction target is correct, after the camera moves away and returns, the object's change often disappears.

The team attributes this limitation to existing models not recording the world's state separately—which objects are in the scene and which objects have changed exist only in the generated frames and context. As a result, the model struggles to distinguish individual object instances in the scene and can only infer the object to interact with from text descriptions, while the interaction result also exists only in the frames and is easily lost as the video progresses.

By contrast, game engines and traditional simulators can precisely record each object's state, but the scene and objects must be pre-built and cannot expand with generated content. This forms the dilemma Oneira must avoid on both sides at once: state must be trackable, yet the scene must be generatable.

Oneira consists of four parts: a Coding Agent, a World State Table, an Engine, and a Video Generation Model. The Coding Agent understands the goal and plans the interaction; the Engine updates the World State Table according to the plan and renders it into a condition video; the Video Generation Model then generates realistic frames based on the condition video.

The World State Table records only interaction-relevant information, including walkable ground areas, plus each object's semantic label, 3D BBox, and interaction state; object appearance is not written into the table and is left to the Video Generation Model. This is the key constraint of separating "state" from "presentation."

The Coding Agent calls off-the-shelf 3D perception models to build the World State Table from the input image. For objects that newly appear as the camera moves, the Coding Agent calls a 3D perception tool on the generated frames to obtain their 3D BBoxes, then adds them to the table. Thus the World State Table can keep expanding with generated content, and objects newly appearing in generated frames can also be directly interacted with.

Interaction with one object may simultaneously change the states of other objects. The example given by the source is the goal "pick up the apple in the lower left": the Coding Agent combines the World State Table and the current frame to determine which apple to pick up and in which chunk to execute it—each roughly 10-second video is divided into 8 chunks.

It also infers that the other two apples will roll away as a result, and writes these two apples together with the target into the plan. The Engine then updates the World State Table according to the planned action, marking all three apples as having left their original positions while the other objects remain unchanged.

Notably, the division of labor has a boundary: each interaction only writes the result into the table, while how the apple moves and where it rolls are left to the Video Generation Model to generate based on motion priors. After the other two apples come to rest, the Coding Agent re-registers their positions from the generated frames and also adds the newly appeared pear to the table.

In the condition video, each object is represented by a colored box. A change in box color indicates a change in the object's state, while the disappearance of a box indicates the object has left its original position. The grid and colored background in the condition video hint at camera motion, the 3D BBox indicates the object to interact with and when the interaction occurs, and the text description explains the type of interaction.

Long-range consistency is maintained by two things: after each video segment is generated, its last frame and the updated World State Table serve as the starting point for the next segment; and when the camera returns to a previously visited position, the model also references the frame from that time as a memory frame. Previously changed object states remain in the table throughout, and when the camera returns, the frames still preserve the changed state.

The team also proposes a generalization argument: the 3D BBox uses only color to represent state and is independent of object category, so the model can also interact with object categories unseen during training. This claim comes from the team's own account, and the source provides no corresponding independent verification.

The research team built the evaluation set InteractionBench, containing 100 interaction test cases and 50 memory test cases, and conducted qualitative and quantitative comparisons with MiniMax-H3 Ref2VA, LingBot-World-V2, YUME 1.5, and AlayaWorld.

The team's self-reported results are: on the interaction test cases, Oneira achieves the best results on all three metrics—whether the interaction occurs, whether the timing is accurate, and whether the object is correct—beating the strongest baseline by 29, 19, and 29 percentage points respectively; in scenes containing multiple identical objects, baseline models often act on the wrong object or change multiple objects at once, whereas Oneira changes only the specified object.

On the memory test cases, after the camera leaves and returns to the same position, Oneira's PSNR and SSIM improve by 93% and 83% respectively, and the state retention rate improves by 18%. On VBench video quality metrics, Oneira remains at the best or near-best level, but the source does not give specific values.

It is necessary to clearly distinguish levels of fact: Oneira's composition, the World State Table's recording scope, the chunk structure, the box representation in the condition video, and InteractionBench's test case counts and comparison baselines are descriptions directly confirmable from the source; whereas the 29/19/29 percentage points, PSNR 93%, SSIM 83%, state retention rate 18%, and the VBench performance are all self-reported by the team and lack third-party independent reproduction.

The source also does not answer several questions that affect extrapolating the conclusions: the specific model and accuracy of the 3D perception tool are not stated, yet the World State Table's accuracy directly depends on it; engineering metrics such as training data scale, compute cost, and inference speed are not disclosed; Oneira's failure cases and boundary conditions in real long videos or complex multi-object scenes are not explained; and whether the evaluation set InteractionBench is public is also not stated.

Therefore, the more prudent judgment is: Oneira validates the feasibility of the technical route of "recording world state separately through a World State Table and making the generated world interactive," but its robustness and reproducibility under open conditions still await independent verification.