Audio-Video Generation Accelerated 54x! TurboT2VA: Joint Distillation of Distribution and Trajectory Distillation

The cost of joint text-to-audio-video generation comes from two directions: long sampling trajectories require the model to repeatedly execute forward computation, while high-resolution audio-video tokens make each forward pass heavy. Reducing only the number of sampling steps leaves single-step computation expensive; optimizing only the operators leaves the repeated computation from multi-step sampling in place. TurboT2VA's approach is to address both layers at once—compressing the number of steps on the training side, and compressing per-step overhead on the inference side.

According to the paper and official repository figures, under the test setting of a single NVIDIA H20, 1024×1792 resolution, and 121 frames, TurboT2VA cuts generator time from 318.74 seconds to 5.83 seconds, achieving a 54.67× generator speedup. This number is the result of stacking two layers of optimization, not the credit of a single technique.

The scope must be clarified first: the high-resolution numbers measure generator time, excluding model loading, audio-video decoding, packaging, and disk read/write. So the 54.67× corresponds to generator speedup under this test scope, and cannot be directly equated with the speedup multiple of end-to-end product experience.

TurboT2VA distills a 40-step teacher into a 4-step student on the LTX-2 model. The training order is dCM warmup, sCM refinement, and joint sCM+DMD optimization; all three stages use the same student model, and what changes is the training objective and when it is introduced.

The first stage uses discrete Consistency Distillation (dCM), where the student learns consistent predictions across discrete noise levels, first establishing a coarse-grained joint denoising structure. The paper argues that dCM does not require continuous-time trajectory tangent estimation, making it suitable as a warmup for distilling large models. The second stage introduces sCM, and the student further learns the teacher's continuous generation trajectory; for audio-video tasks, this stage needs to simultaneously preserve the evolution of motion in the picture, the development of sound events, and the temporal correspondence between the two. The third stage keeps sCM and adds Distribution Matching Distillation (DMD), pushing the generated results toward the teacher distribution.

The paper compares direct joint training with staged training: using sCM+DMD directly can converge and also generate reasonable results; the staged approach, after entering the joint training stage, achieves a higher Javis Score at the same number of training steps. The authors' explanation for this phenomenon is that when distribution matching is added too early, the student has not yet learned a sufficiently broad generation trajectory before being subjected to strong distribution-level constraints, which may weaken the diversity inherited from the teacher trajectory. This explanation is the authors' view, and the source gives no independent verification.

The paper also claims that pure sCM leans more toward preserving diversity, pure DMD leans more toward perceptual quality, and the staged joint approach achieves a better balance between the two. This judgment comes from the paper's own account and is a source claim rather than a third-party conclusion. Sample diversity itself needs to be judged by combining generation results under different random seeds with independent diversity evaluation; the source gives no complete diversity numbers.