Music Arena v1.1 Update: New Prompt Set and Blind Panel Voting

Artificial Analysis launches Music Arena v1.1, comprising two new benchmarks: AA-Music-Vocal v1.1 and AA-Music-Instrumental v1.1. Unlike the typical "leaderboard update," this change replaces both the prompt set and the scoring source at once: the prompts are a newly constructed set of 1,000, 500 per benchmark; scoring comes entirely from blind-test votes by its recruited jury, with public Arena votes excluded from scoring.

Putting these two changes together, v1.1 reads more like a rewrite of the two premises—"what questions to use, and who judges." Artificial Analysis states that the past 3 months saw flagship model releases such as Suno v6, which forms the backdrop for this benchmark update (this statement is a source claim; the specific list and its impact are not elaborated by the source).

The new prompt set totals 1,000 prompts, curated and balanced across 17 genres, spanning from classical, jazz, and blues to EDM, metal, and reggae. This approach of "setting genre quotas first, then allocating questions" makes genre coverage a verifiable structure rather than a result tallied after the fact.

Artificial Analysis states that each prompt tests song structure, mood, instrumentation, or vocal style, and that the prompts are longer and more complex to reflect current prompting behavior and the emerging capabilities of frontier music models. This judgment about "longer and more complex" is a source claim; the source provides no length or complexity comparison data against the previous prompt set.

The post also showcases several sample prompts, such as style descriptions for Korean dance pop, Memphis deep Southern soul, tech house, late-sixties rock steady, Romantic choral requiem, and video game boss battle soundtrack. These examples illustrate the span of the prompts across genres and scenarios, but the source does not state whether the sample prompts are fully identical to the prompts ultimately used for scoring.

The most substantive institutional change in v1.1 is the scoring source: scoring comes entirely from blind-test votes by a recruited jury, and public Arena votes do not count toward scoring. This means public voting may still exist as a participation entry point, but it no longer affects rankings.

A distinction is needed: "public votes do not count toward scoring" is a confirmed rule setting; however, how the jury is recruited, how large it is, and how its members are composed are all unstated by the source. The statistical methodology of the blind-test votes, confidence intervals, and model ranking results are likewise not provided by the source. Therefore what can be confirmed is that the scoring scope has been narrowed; what cannot be confirmed is whether results under this scope are more stable.

The updated leaderboard launches with over 37,000 blind human preference votes, of which AA-Music-Vocal v1.1 received over 16,000 and AA-Music-Instrumental v1.1 received over 21,000. The vote counts of the two benchmarks sum consistently with the total.

Artificial Analysis also states that every model on the leaderboard received over 2,000 votes. This gives a lower bound for per-model vote counts, but the source does not state the specific number of models on the leaderboard, so the relationship between "over 2,000 votes/model" and the total vote count in terms of model count is unclear.

All the above numbers come from Artificial Analysis's own release, with no third-party independent verification. They can confirm "what was released," but cannot be treated as externally verified evaluation conclusions.

In AA-Music-Vocal v1.1, no lyrics are provided, so this benchmark measures each model's end-to-end songwriting capability. This setting brings "writing lyrics" into the evaluated scope, rather than having the model merely sing given text.

This statement is a source claim. The source does not explain the specific impact of this design on comparability, nor does it provide details on how the jury judges the match between vocals and text under the condition of missing lyrics.

Separating confirmed facts from source claims is the premise for understanding this update. What is confirmed: two new benchmarks launched, 1,000 prompts (500 per benchmark), 17 genres, scoring solely from blind-test jury votes, over 37,000 total votes with the 16,000/21,000 split, over 2,000 votes per model on the leaderboard, and the official publication of two leaderboard links with an invitation for users to vote on Music Arena.

What are source claims: that the prompts are longer and more complex, that the Vocal benchmark provides no lyrics and therefore measures end-to-end creation, and that "flagship model releases in the past 3 months" form the update backdrop. The source does not state jury recruitment and size, statistical methodology and confidence intervals, ranking results, nor whether the sample prompts are identical to the scoring prompts.

Additionally, the post was published on the X platform, with some content being reposts or duplicates, so information completeness is limited. Therefore what this article can confirm is the benchmark's composition and rules; what it cannot confirm is what rankings these rules produced, and whether the rankings can withstand independent review.