Taobao TaoMate-H3 Open-Sourced: Three-Step Generation for Minute-Level Continuous Audio-Video Creation

The most direct change in TaoMate-H3 happens in the generation rhythm. The traditional approach of repeatedly denoising an entire video requires waiting for the whole thing to finish, whereas TaoMate-H3 advances in units of small chunks, with each chunk requiring only three denoising steps. Through few-step LoRA, the team compresses the denoising process of each small chunk into three steps, advancing along the time states 0→16→33→49. This means the model prioritizes completing the current chunk, then generates subsequent content, without waiting for the entire video to finish denoising.

Under the current five-second prompt configuration, each segment is divided into four chunks, with the main chunk at about 1.4 seconds and the tail chunk shorter. The model completes these chunks in sequence and uses historical context to continue generating. After each chunk is completed, the model updates the KV cache based on the final generated audio-video state for use by the next chunk. Few-step generation reduces the computation of repeated iterations, while chunked processing allows the model to produce the first segment of content earlier.

This change in rhythm maps directly to the production time of the first segment of content. Team self-test shows that on an 8×NVIDIA H20 96 GB machine, when outputting a 480×864 ten-second video, the ready time of the first segment's final latent shortens from 170.052 seconds to 6.148 seconds, a 27.66× speedup. The first-segment latent being ready corresponds to the time when the model completes the first segment of generation and can hand it to the VAE for decoding. For content production that needs timely feedback, this segment-by-segment output approach helps shorten waiting and provides a foundation for generating while playing.

TaoMate-H3 handles sound and picture in the same generation process, supporting not only human speech, singing, and character dialogue, but also environmental and action sound effects such as ocean waves, vehicle movement, and explosions. Sound participates in the model's generation process, providing temporal coordination for lip sync, facial expressions, and actions.

To give local chunks a segment-level audio reference, the system internally first prepares an audio denoising trajectory covering the current prompt segment, then uses the audio latent at the corresponding time position to guide each small chunk. The three denoising updates are respectively aligned with three states in the audio trajectory, and the final audio latent remains consistent with the clean endpoint of the guiding trajectory. Audio guidance runs through the generation process, helping stabilize the unfolding rhythm of dialogue and reducing repeated opening lines when continuing from short chunks.

To connect the timbre and tone across different segments, the system uses the last approximately one second of clean audio from the previous segment as a read-only reference. New audio continues to be generated under this reference, while the reference itself remains unchanged; finally, the audio latents of each segment are decoded uniformly to obtain continuous sound output. This design addresses the problem that after streaming generation splits video into short chunks, the pauses, speaking rate, and ending position of a line of dialogue need to be arranged within a complete segment.

With the continuously updated audio-video KV cache and segmented audio guidance, the model can continue previous content across prompt boundaries. Creators can adjust the explanation focus or performance content segment by segment, organizing more complete product introductions and character narratives.

The cache is organized as "initial visual reference + recent audio-video context." The initial reference preserves the initial information of characters and scenes, while the recent context scrolls and updates with generation progress, enabling the model to continue creating following the latest actions and sounds. This approach keeps the historical cache within a fixed scale, preventing it from growing continuously with video duration, and provides stable context support for minute-level continuous generation.

In long videos, prompts can change, but the media timeline still needs to be continuous. TaoMate-H3 uses globally continuous RoPE positional encoding, moves the current text's time coordinate along with audio-video progress, and aligns the text's right boundary to the current media start point, so that new prompts correspond to the content being generated. When switching prompts, the system retains the existing audio-video KV while distinguishing newly generated content from the tail context needed for encoding and decoding. Historical content is used only for connection, not regenerated. For brightness and color tone changes during long continuations, the model also uses the visual statistics of the first chunk as a reference to align the mean and variance of subsequent video latents, reducing drift in the visual experience.

The training stage adopts the autoregressive training approach of Self Forcing, letting the model condition on its own generated historical chunks to continue generating subsequent content. This allows the training process to encounter historical states from actual continuation, alleviating the training-inference distribution gap caused by relying only on real historical training. This training approach, together with few-step inference and the KV cache, forms the foundation of streaming continuation.

On the same 8×NVIDIA H20 96 GB machine, the team tested the original MiniMax H3 and TaoMate-H3, both outputting 480×864 ten-second videos. TaoMate-H3's pure DiT computation time drops from 169.572 seconds to 14.810 seconds, an 11.45× speedup; the ready time of the first segment's final latent shortens from 170.052 seconds to 6.148 seconds, a 27.66× speedup.

Under the above ten-second configuration, TaoMate-H3 generates a total of eight small chunks, executing 24 generation forwards and 8 KV updates. The single-machine eight-GPU setup uses TP2 × Ulysses4 parallelism, combined with efficient attention, operator fusion, and selective low-precision computation for some linear layers, reducing inference overhead. These optimizations both improve the generation efficiency of the entire content and shorten the production time of the first segment, providing a foundation for chunked decoding, playback, and interactive application development.

It must be made clear that the above performance data all come from the team's self-test on the same 8×H20 96 GB machine, with output being a 480×864 ten-second video, and lack third-party independent verification. Capability descriptions such as three-step denoising and minute-level continuation are also all team self-reports, with no external evaluation seen. Demo cases (backpack, wooden comb, singing by the window, girl and white fox under the moon, sunset coast, explosion, etc.) are all official demonstrations and provide no objective quality assessment.

This open-source release includes inference code, LoRA weights, and example prompts, downloadable from GitHub and Hugging Face. The project is based on MiniMax H3 and follows the MiniMax H3 Community License. Before use, you need to complete environment installation according to the README and download the MiniMax H3 base weights.

The runtime environment requires Linux, Python 3.10/3.11, CUDA 12.8, and PyTorch 2.8, and provides single-machine 4-GPU or 8-GPU inference entry points. The verified runtime configuration is 8×H20 96 GB. The public inference entry point automatically runs the entire pipeline and saves the final video.

The model supports 480p, 768p, and 1080p-level landscape and portrait output. The ten-second example in the repository can modify scenes, dialogue, and actions by replacing the prompt file; through duration and resolution parameters, you can further generate longer videos or switch between landscape and portrait formats.

The team says it is still continuously optimizing inference speed, further shortening the waiting time for the first segment of content and subsequent responses. The team also plans to release and open-source new FL2AV streaming models in the near future, further expanding to Ref2AV, and simultaneously opening the corresponding weights to support richer audio-video creation methods. However, the release timing and specific capabilities of the FL2AV and Ref2AV models are not yet clear.

On capability boundaries, the model's stability and consistency at longer durations or higher resolutions are not explained in the source; the impact of the fixed-scale audio-video KV cache on memory effectiveness for ultra-long videos also lacks quantitative evidence. These are key variables for judging whether TaoMate-H3 can move from demo to stable production.

From the application scenario perspective, TaoMate-H3 targets creative scenarios that require continuous generation of sound and picture. Users can describe a complete scene with one prompt, or arrange dialogue, actions, and plot segment by segment; the model continues generating while retaining historical context, so adjacent segments share character, sound, and scene information. The demos cover different content types such as product explanations, character singing, anime dialogue, scenery, and movie special effects, but none provide objective quality assessment.