Open ASR Leaderboard Adds Its First Global South Language

Maintainers of the Open ASR Leaderboard point out that benchmarks determine which models are adopted and iterated upon, and capabilities that are not measured often do not improve. Recent work has focused on making evaluation metrics more trustworthy: using private splits, analyzing benchmark fit, and narrowing normalizer gaps. But these improvements ultimately still reduce to a single number—word error rate (WER).

Research shows that ASR error rates are unevenly distributed across populations: commercial systems have error rates roughly twice as high for Black speakers as for white speakers, and gender, age, and accent also cause disparities. These differences are invisible on leaderboards because test sets record what was said, but rarely who said it.

To fill this gap, VoiceArena and Hugging Face collaborated to create the Monsoon evaluation sets. The design principle is: a test set can only expose failure modes along the axes it varies. Monsoon varies along nine axes: geography, age, gender, vocabulary, device, acoustic environment, speech type, speaking rate, and the possibility of multiple valid transcriptions for the same audio.

The data collection method deliberately differs from traditional benchmarks: contributors are recruited from hundreds of regions rather than recording for long periods in a few locations; contributors use their own phones and networks, recording indoors and outdoors; prompts guide everyday topics, encouraging contributors to express opinions, disagreements, narratives, and recollections, naturally eliciting named entities, numbers, and unscripted phrasing. Age and gender are verified and recorded.

Monsoon comprises four splits: Indian English public set (5.62 hours, 1,444 speakers), Indian English private set (5.58 hours, 1,405 speakers), Hindi public set (1.33 hours, 468 speakers), and Hindi private set (4.47 hours, 1,571 speakers). All splits are single-channel segments from spontaneous two-channel conversations, each segment corresponding to one speaker.

In addition to tabular fields, each segment records occupation, education, marital status, income bracket, phone brand, current city, and years of residence in the area. The Indian English public set covers 428 districts, 30 states and union territories; the Hindi sets cover 202 to 295 districts. Device models range from 315 to 582, with no single model accounting for more than 2.1% of any subset.

Despite Monsoon's small duration, the large number of speakers is its core value. The top ten contributors account for 2.8% to 6.8% of total duration, and over half of speakers appear only once. This means results are an average over hundreds of different voices, not long recordings from a few speakers.

Indian English is not a single accent: in the public set, 35% of segments come from southern speakers, with eastern and central each at 18%, northern at 16%, and western at 11%. This accent diversity is recorded in metadata, not subjective assertion.

The English sets use standard string references, with spelling variations handled by the leaderboard's normalizer. But Hindi spelling variants cannot be resolved by a normalizer with fixed mappings, so the Hindi sets use a 'lattice' reference: for each transcribed segment, a list of spellings accepted as correct is provided.

This design acknowledges the complexity of Hindi orthography, ensuring that correct prediction variants are not unfairly penalized due to insufficient normalization.

Monsoon provides 18 columns of data per segment, of which 12 are metadata, whereas most public ASR test sets only provide identifier, transcription, and duration. Demographic fields are complete or near-complete, and contributors have consented to use.

Metadata groupings include: segment (ID, audio, duration, language), reference (lattice or text), speaker (ID, gender, date of birth), background (occupation, education, marital status, income), and geography (birth region, birth state, current city, years of residence).