Xiaohongshu Releases FireRedTTS3: Unified Speech Cloning, Design, and Editing

Over the past two years, zero-shot voice cloning has matured, but user needs are shifting from 'replicating a voice' to 'control': designing voices through natural language descriptions or making precise local modifications to existing speech. FireRedTTS3's release directly addresses this demand.

Achieving control requires overcoming three hurdles: understanding natural language instructions, precisely mapping them to speech attributes or segments, and preserving unspecified parts. The foundation is a speech representation that encodes both semantics (content/style/intent) and acoustic details (timbre/breathiness/pitch).

Existing approaches each have shortcomings: non-autoregressive Flow Matching struggles to leverage instruction-following from text LLMs; discrete token quantization erases fine-grained acoustic details; continuous autoregression (LLM-DiT) suffers from error accumulation leading to timbre drift. Previous remedies either required extra semantic modules or complicated the architecture.

The first cornerstone of FireRedTTS3 is the continuous speech tokenizer RedAE. The team first trained an audio understanding model, FireRedAudio, whose Audio Encoder simultaneously understands content semantics and speaker acoustic cues. During RedAE training, this encoder is frozen as a semantic teacher, and RedAE's output is aligned to its features via an MSE semantic distillation loss. After training, the teacher is no longer used in inference.

RedAE deliberately omits KL regularization to avoid over-compressing acoustic information, preserving timbre details. Input 24 kHz waveforms are segmented into 480-sample frames (50 Hz), passed through two cascaded Qwen3-style Transformers, and downsampled to 25 Hz. The overall training uses a GAN framework with losses including adversarial, multi-scale Mel reconstruction, feature matching, and semantic distillation.

Training data comprises 500,000 hours (50% clean speech, 25% noisy, 10% sound effects, 15% music), trained for 550,000 steps on 32 H800 GPUs. It uses single-stage joint training, with no extra semantic branches or multi-stage pipelines.

The generation framework consists of three components: an Aggregator that compresses the 25 Hz representation into 6.25 Hz latent patches; a Backbone Transformer initialized from Qwen3, inheriting text understanding and instruction following; and a DiT module that performs patch-level denoising conditioned on the Backbone, with input being the noisy current patch plus 12 frames of clean historical latents. During training, conditions are randomly dropped with probability 0.1 for CFG.

The Base version targets multilingual/multi-dialect cloning: the Backbone is initialized from Qwen3-1.7B-Base, uses CAM++ to extract speaker embeddings, and prepends language tags to the text. Training proceeds in two stages: the first uses 2.6 million hours of Chinese and English (170,000 steps), and the second uses 560,000 hours covering 24 languages plus 21 Chinese dialects.

The Instruct version unifies cloning/design/editing: the Backbone is initialized from the instruction-capable Qwen3-1.7B, uses ChatML format, and distinguishes tasks via system prompts. A key design is 'plan-then-synthesize': for voice design, free-form descriptions are decomposed into 12 acoustic attribute sequences; for editing, they are expanded into target transcription plus edit region masks. It does not use explicit speaker embeddings or language tags, and the second stage trains on 330,000 hours for 40,000 steps.

On Seed-TTS-Eval, the Base version achieves the lowest average error rate (3.04%) and highest average speaker similarity (78.8%), with best similarity on both Test-EN and Test-ZH. On MiniMax-MLS-Test across 24 languages, it achieves a 3.75% error rate and 84.8% similarity, ranking first or second in similarity for 22 of 24 languages. Portuguese and Ukrainian, not in RedAE's training data, still show competitive performance, demonstrating generalization.

On InstructTTSEval (6,000 samples, 3,000 each in Chinese and English), the Instruct version is best across all three instruction types: acoustic parameter specification (APS), style description (DSD), and role-play (RP), thanks to the 12-dimensional structured scheme reducing instruction ambiguity.

On Ming-Freeform-Audio-Edit, it leads in most acoustic edits (speed/pitch/volume) and semantic edits (insertion/deletion/replacement), remaining stable in open settings. Key metrics: low WER and NoEdit WER, high ACC, high SIM, and extremely low RDE/RAE, achieving 'accurate edits, unchanged rest, and no timbre drift'.

FireRedTTS3's key contribution is pushing error accumulation suppression down to the representation layer, using a frozen semantic teacher to inject semantics, making stability an inherent property of the representation. As a result, cloning, design, and editing are unified for the first time into a single simple LLM-DiT framework, with single-stage tokenizer training, no extra semantic modules, and no complex structures.

The team describes this as a 'sustainable growth foundation,' having built a growable speech generation infrastructure from FireRedTTS, FireRedTTS-1S, FireRedTTS-2, to FireRedTTS3. Future work will expand languages and dialects, and enrich instruction control and editing boundaries.

Note: This article is a promotional piece from Machine Intelligence, and all evaluation results are self-reported by the team or from official scripts, not independently verified by third parties. Statements like 'industry-leading' are claims from the source. The generalization conclusions for Portuguese/Ukrainian are based on a single evaluation, and language coverage details are not provided.