Alibaba Releases Qwen-Audio-3.0-Realtime Voice Interaction Model

Qwen-Audio-3.0-Realtime dynamically adjusts tone, rhythm, pitch, and emotional expression based on conversation context, achieving human-like voice interaction. It supports voice cloning and achieves SOTA results on the S2S voice instruction following public benchmark VStyle.

The model possesses emotion-aware generation, prosody dynamic adjustment, and paralinguistic information processing capabilities, understanding and generating non-verbal sound signals such as laughter, sighs, and hesitations. In complex contexts like debates, role-playing, and emotional companionship, it adjusts speaking style and emotional expression.

These capabilities enable applications in AI companions, intelligent customer service, educational assistants, and other scenarios, providing warm emotional support.

The model incorporates a multimodal-aware duplex control model that perceives rich information such as voice, environment, background, and speaker, enabling precise duplex rhythm judgment. It remains composed in noisy backgrounds, multi-person conversations, and speaker switching scenarios.

Its multimodal perception and intelligent interruption detection simultaneously analyze audio, semantics, and voiceprint features, distinguishing environmental noise from real interruptions. The Realtime-API reserves an audio_prompt field, allowing developers to implement voiceprint-level anti-interference and precisely target specific users.

Achieves SOTA results in Artificial Analysis's dialogue dynamics subcategory, leading in key metrics such as interruption detection accuracy and response latency.

The model possesses strong Agentic capabilities and a dynamic tool calling mechanism, intelligently deciding when to call tools and when to engage in casual conversation based on dialogue context. In TauBench tests (covering real business domains like retail, aviation, and telecom), it accurately decomposes complex spoken instructions and automatically calls external tools for multi-step reasoning.

Core features include dynamic tool routing, multi-tool coordination, unified capability access (based on FunctionCall standard protocol supporting MCP, API, knowledge base), and context awareness. It can perform tasks such as route planning, restaurant recommendations, real-time news queries, stock quotes, schedule management, and data analysis.

To address the IQ degradation issue in voice models, the On-Policy Distillation framework is adopted, distilling the complete reasoning ability of text large models into the voice model. Additionally, a multi-teacher distillation strategy is introduced, including spoken multi-turn preference teacher, general teacher, Agentic teacher, and audio understanding teacher, ensuring the model is well-rounded.

The model provides millisecond-level response for latency-sensitive scenarios like daily Q&A. It ranks first overall in Artificial Analysis's Speech Reasoning subcategory and performs excellently on benchmarks like VoiceBench and AudioMultiChallenge.

However, note that in VoiceBench, the plus version scores 92.5 on standard prompts and 90.5 on spoken prompts (a drop of 2.0); the flash version scores 89.8 on standard prompts and 87.5 on spoken prompts (a drop of 2.4). In AudioMultiChallenge, the plus version scores 44.0 on standard prompts and 37.6 on spoken prompts (a drop of 6.4); the flash version scores 43.6 on standard prompts and 38.1 on spoken prompts (a drop of 5.5). Performance decreases on spoken prompts.