Hugging Face announced the Speech Agent Arena, a new benchmark for evaluating speech-to-speech models in real-world scenarios. It measures conversational preference and task success rate through human comparisons, with initial results showing Google's Gemini 3.1 Flash Live Preview leading the leaderboard.
SynthePulse Insight · AI deep readingMembers
Version 1 · 1 source
Artificial Analysis releases Speech Agent Arena to evaluate speech-to-speech models in real-world scenarios. Gemini 3.1 Flash Live Preview leads in preference, but its task success rate is lower than Grok Voice and GPT-Realtime series, revealing the gap between 'sounding good' and 'doing well.'
On August 21, 2026, Artificial Analysis released Speech Agent Arena, aiming to evaluate speech-to-speech models in real-world scenarios. The benchmark covers 15 agentic scenarios (requiring tool calls, such as ordering food) and 20 non-agentic scenarios (no tool calls, such as asking for business hours). It collects preference votes from human participants in live conversations with two hidden models and fits Elo scores.
Task success rate is defined as the proportion of qualifying conversations where the model completes the request with the correct final tool call. To reduce overfitting, scenario prompts, tool schemas, and participant instructions are kept confidential except for one example; participants are paid, screened third-party individuals.
This report is based on the official release from Artificial Analysis, a primary source. All data comes from their announcement and has not been independently verified. Some conclusions (such as the reasons for the divergence between preference and task success) are inferences based on observations in the release.
Primary report
Primary source