V-RAE: Rethinking Video Latent Spaces for Generation with Visual Representations

The core idea of V-RAE originates from the Representation Autoencoder (RAE) in the image domain: instead of learning a latent space from pixel reconstruction objectives, it directly uses high-level representations learned by pretrained vision models as the latent space for generative models. V-RAE extends this approach to video, systematically studying four pretrained visual representations from different paradigms: DINOv3, SigLIP2, EUPE, and V-JEPA 2.1.

Video poses unique challenges: per-frame encoding leads to token count growing with length, and simple temporal compression loses motion information. V-RAE introduces a lightweight temporal attention pooling module to achieve 4x temporal compression while preserving semantic and dynamic information, then reconstructs videos via a spatiotemporal Transformer decoder with 3D RoPE.

Experiments confirm the paradigm works for video: on Kinetics-600, V-RAE achieves a reconstruction rFVD of 2.13, outperforming the best result among evaluated large-scale pretrained video VAEs (3.58); on UCF101, reconstruction performance is close to the strongest VAE baseline. In semantic probing, V-RAE reaches 90.92% accuracy on UCF101, while the best traditional VAE only achieves 30.83%; on Something-Something V2, it reaches 72.91%.

The key question is: do these semantics make generation easier? With fixed generation backbone, token count, and training settings, all four V-RAE representations yield better downstream generation performance on both UCF101 and Kinetics-600, with best gFVD of 117.86 and 19.16, respectively.

Training efficiency improves significantly: on UCF101, V-RAE reaches the generation level of traditional VAE's ~150,000 updates in about 30,000 updates; on K600, up to ~6x convergence speedup is observed. This indicates that the representation latent space changes the problem the generator needs to solve—from rediscovering objects, actions, and scene structure to directly learning how visual states change over time.

Experiments reveal that the correlation between reconstruction quality (rFVD) of different autoencoders and final generation quality is weak: correlation coefficients are only 0.200 on UCF101 and 0.473 on K600. The reason is that reconstruction only tests the decoder's ability to restore the true encoder output, whereas during generation, predicted latent representations always deviate from the true trajectory. If the latent space is not smooth, small deviations can be amplified into artifacts like ghosting and flickering.

To address this, V-RAE proposes the tFVD metric: it locally interpolates between temporally adjacent latent variables to construct latent representations that slightly deviate from the true trajectory, and observes whether the decoder can restore them into natural, continuous videos. The correlation coefficients between tFVD and generation quality reach 0.621 and 0.919 on UCF101 and K600, respectively, significantly higher than traditional reconstruction metrics, indicating that temporal smoothness and robustness to generation errors are crucial.

V-RAE further validates the advantages of representation latent spaces in future video prediction. On Cityscapes, using the exact same prediction architecture and training budget, V-RAE reduces gFID from 15.02 (traditional VAE) to 11.52, and gFVD from 144.47 to 111.36.

Notably, V-RAE's reconstruction rFVD is actually worse, yet future prediction is significantly better. This again confirms: for prediction/generation modeling, good reconstruction is not the ultimate goal; a more semantic, smoother, and more predictable latent state space is the appropriate interface for understanding the dynamic world.

The significance of V-RAE lies not only in extending image RAE to video but also in exposing easily overlooked issues in latent space design: video representations need to handle temporal redundancy; semantic representations retain rich structure after compression; more structured latent spaces lead to better generation quality and up to ~6x convergence speedup; reconstruction quality cannot fully predict generation quality; temporal smoothness and prediction robustness are key for video generation; the advantages of representation latent spaces transfer to future prediction.

These results point to a fundamental question: how to evaluate and design latent spaces truly oriented toward generation? In the past, autoencoders were viewed as compression and reconstruction modules, but from a generation perspective, they define the 'coordinate system' of the learning problem. A good video latent space should not only accurately reconstruct the past but also make the future easy to generate and predict.