HKUST, CityU, Fudan, CMU, NYU, NTU Release First Survey on Memory for Autoregressive Video Generation

Over the past year, video generation has been moving toward long-horizon generation, streaming generation and interactive world models. The longer the generation, the harder it is to avoid a fundamental question: how exactly should the model "remember" the past? Whether a character who leaves the frame and reappears is still the same person, whether the spatial layout of a scene can still be maintained after the camera circles back to it, whether an object that was moved will return to its original position dozens of generation steps later—what these questions share is that the information the answer depends on is long gone from the current window.

Researchers from Hong Kong University of Science and Technology, City University of Hong Kong, Fudan University, CMU, NYU, NTU and other institutions jointly completed and released the 110-page survey "The Past Frames the Future: Memory for Autoregressive Video Generation," attempting to re-understand related research scattered across long video generation, streaming generation, interactive world model and other directions within a unified Memory framework. The paper link is https://arxiv.org/pdf/2609.28466, and the companion GitHub repository is https://github.com/HaroldChen19/Awesome-AR-Video-Memory.

The evidence boundary must be stated first: this article is based on a WeChat repost report, and the survey's classification framework and conclusions are all the authors' own statements. The report does not say whether the survey has undergone peer review, nor does it give its publication venue, so the parts below involving the framework and judgments are all treated as source_claim.

Autoregressive video generation continuously generates the next token, frame or video chunk based on existing history. The problem is that history grows ever longer, while the context the model can directly access is always limited: constrained by context window, VRAM, KV-cache storage and latency, real systems can usually retain only a limited local context. As the rollout advances, the early character identity, scene layout, motion state and interaction results gradually leave the current window.

From this, the survey defines Memory as a historical state continuously maintained across AR steps: even after the original evidence has left the current local context, it can still influence future generation. The key point of this definition is that Memory is not simply equivalent to Longer Context; the real question is how historical information is continuously retained, managed and reused.

There is an easily overlooked inference here: if the criterion for "memory" is "it can still influence generation after leaving the current context," then merely expanding the context window cannot automatically satisfy that criterion—it only postpones the point at which information leaves the window, and the survey itself gives no quantitative comparison between expanding the window and Memory mechanisms.