MirroS Releases AgentGarten: Code-Built Worlds, Diffusion-Rendered Reality, Agents Evolve Through Play

To make AI understand the physical world, watching massive amounts of video is not enough—just as learning to swim cannot be done by only watching recordings. Agents need to step in themselves: take actions, observe results, and then decide the next step. This requires an environment with deterministic physical causality: a pushed box must stay where it is, a blocked passage must not be passable, and subsequent actions are continuously constrained by these physical changes.

AgentGarten, released by the MirroS team, is aimed precisely at this need. It connects an executable code environment with a real-time neural renderer: code defines the physics rules, and the model generates visual feedback, allowing agents to continuously interact with the world at over 30 fps throughput and stably supporting the closed loop at 480p resolution.

The core design of AgentGarten is to let two technical paths each do their own job. A standard interaction loop works like this: the agent gives an action command; the code environment immediately computes physical collisions and state updates, and exports a lightweight geometric sketch (spatial depth or surface normals) from a first-person camera viewpoint; the neural renderer reads this sketch, combines it with prior visual memory, and renders a realistic next frame to hand back to the agent.

This division of labor brings two direct results. First, the physics rules are deterministic and controllable: scene layout, contact determination, and win/loss are all adjudicated by code, producing no hallucinated physics. Second, building new worlds becomes extremely fast: as long as simple 3D depth and normal contours can be output, any minimalist code scene can use this neural renderer to obtain realistic lighting and materials. The cost of expanding virtual environments shifts from art-labor-intensive to code-procedural expansion.

A classic reference is OpenAI's 2019 hide-and-seek experiment: hiders and seekers competed in an arena with barriers and ramps, and through large-scale self-play they successively learned to build shelters and to vault walls using ramps. According to the source, shelter building took roughly 25 million episodes, and ramp-assisted wall vaulting took roughly 100 million episodes.

The MirroS team redid this experiment in AgentGarten, using a one-on-one setup: the hider first arranges the scene to block the entrance, then the seeker enters to search. Agents control movement and grabbing by writing Python code, making decisions solely from first-person frames, with no knowledge of spatial coordinates or the opponent's position. Both sides start from a completely blank manual, each maintaining a strategy library composed of single-skill entries; each round consists of ten matches, after which they review and consolidate, and the next round randomly selects some skills from the library to apply.

The result: hiders learn by round 4 to move barriers and build shelters, and seekers master vaulting walls via ramps by round 10. After a failed jump, the seeker will also proactively push the ramp closer, adjust its position, and try again.

What drives strategy evolution is the experiment manual the agents write themselves. The drill is divided into four progressive stages: receiving the task, blind-box trial and error, writing the manual, and sealing it for inheritance. When receiving the task, they get a task file that clarifies the goal and rule red lines, but there is no standard answer; in blind-box trial and error, they act autonomously under strict step and time limits, with only camera frames, no god's-eye coordinates, no minimap, and no score visible until the end.

After a round ends, the agent self-reviews, recording what it tried, what phenomena it saw, which judgments are doubtful, and what new hypotheses to verify next time. The manual is archived and frozen, directly handed to the next round's agent as a prior legacy. Experiences that withstand testing are carried forward by successors, while failed lessons prompt them to try new strategy branches.

Because the code world is essentially a programmable software environment, the same explore-review-consolidate loop can be replicated in more scenarios. The team ran four rounds of drills in each of four other very different worlds.

Companion dog: the agent must maintain the dog's favorability within 60 seconds by petting and playing ball at the right time, with the score rising from 13 in the first round to 19 in round 4. Narrow bridge crossing: two cars negotiate mutual yielding based on the first-person driving view, and the total time for both cars to pass is compressed from 71 seconds to 41 seconds. Cooperative sheep herding: two sheepdogs, relying only on their own lines of sight, cooperate to herd four sheep into the pen and maintain it for 5 seconds, from timing out in round 1 with only three sheep penned to stably penning all of them in the latter three rounds. Quarry loader: in round 1 it barely pushes rocks aside, but by round 4 it can complete the entire process 31 seconds early within the 360-second time limit.

These tasks, spanning an enormous range, demonstrate the same thing: this experience-consolidation loop also works when transferred to completely different worlds.

To let agents act while watching, the neural renderer must overcome three major challenges: keeping up with sudden actions, staying stable over extremely long interactions, and running fast enough. Starting from a base omni-modal model, the team proposed the Adversarial Forcing training method and combined it with inference optimization to open up the visual channel.

First, change from whole-segment offline generation to streaming block-by-block rendering: the agent performs an action, the model immediately produces the corresponding short block of frames, and the agent sees the result before making the next decision. Second, long-horizon interaction must not drift: the longer continuous generation goes on, the more small errors tend to accumulate into appearance drift; the team designed a precise replay mechanism—a first pass with gradient-free forward unrolling, and a second pass re-executed with gradients at exactly the same block-by-block rhythm, to stabilize temporal consistency in long-sequence interaction. Third, introduce a real-video discriminator to construct an adversarial signal, using real physical image quality as an anchor to constrain visual texture, and making generated results conform to the geometric contours given by code.