Artificial Analysis launches Decision Index to benchmark decision models

Artificial Analysis has released the Artificial Analysis Decision Index v1.0, used to compare the trade-offs between decision models and language models in capability, cost, and speed. According to the organization, decision models output structured answers or classifications with probabilities, respond faster, and cost less than using language models—this is the publisher's definition of the product category, not a third-party-verified conclusion.

Index v1.0 consists of four benchmarks: AA-Omniscience-Decision, HLE-MultiChoice, NVIDIA's When2Call, and WildGuardTest. The four benchmarks point to knowledge, reasoning, tool calling, and safety classification respectively, covering the main task types that decision models are expected to handle.

Artificial Analysis states it will continue to develop the index to help developers and enterprises understand when and how to use decision models. This statement itself also shows that v1.0 is not the final word: metrics and results may change.

AA-Omniscience-Decision is a 500-question private dataset adapted from the proprietary AA-Omniscience benchmark, measuring decision models' knowledge level and hallucination tendency. Private means the outside world cannot review the questions and scoring details, a point that runs through all the conclusions about "knowledge" below.

HLE-MultiChoice consists of 513 text-only multiple-choice questions from Humanity's Last Exam, testing decision models' reasoning ability. Text-only and multiple-choice means it does not cover multimodal or other question types.

When2Call is a tool-calling benchmark from NVIDIA, testing whether a model can correctly decide to call a tool, request clarification, refuse to answer, or answer directly, pointing to capabilities related to agent workflows and tool routing.

WildGuardTest is used to evaluate models' classification ability on harmful user requests, assistant refusals, and harmful responses. The publisher used the 835-row subset of the original 1,725-row test data where three human annotators were fully consistent across three labels, and excluded questions that would persistently trigger model refusals. The samples were filtered, and this is a premise that must be remembered when interpreting the scores.

Within decision models, Jev 1.13 ranks first on the Decision Index with 58 points, above GLiDE's 54 points and Microsoft-Decision-1's 54 points. This is the ranking result given by the publisher.

But when language models are brought in, the picture changes. Claude Haiku 5.5 (Low) scores 61 points on the index, 3 points higher than Jev 1.13. In other words, the leader of the decision model camp still scores lower overall than a language model used for comparison.

The cost of these 3 points is explicitly quantified by the publisher: Claude Haiku 5.5 takes 1.4 seconds per task, about 11 times that of Jev 1.13; it costs $184 per million tasks, about 4.4 times that of Jev 1.13. The score gap is small, the cost and latency gaps are large—this is the most central tension of the entire index.

It should be noted that the source does not disclose the weighting method of the Decision Index or the weights of each benchmark, so the comparability of 58 points and 61 points cannot be independently judged. This is an inference: with weights unknown, readers cannot confirm from the available material whether the 3-point gap is stable.

Decider v1.1 27B has the lowest cost per million tasks, at $17. Microsoft-Decision-1 ($31) and Jev 1.13 ($42) are also on the capability-cost Pareto frontier. The so-called frontier means that at these three points, it is difficult to further lower cost without sacrificing capability.

On the speed dimension, Celeris-1 Decision is the fastest per task, at 0.06 seconds, more than twice as fast as Jev 1.13's 0.13 seconds. The source does not give the provider of Celeris-1 Decision or its complete evaluation score, so its position on this chart can only be partially reconstructed.

These cost and time figures are given by the publisher, without specifying the specific measurement environment and hardware conditions, nor disclosing batch size and concurrency conventions. Therefore they are suitable for comparing orders of magnitude, not as reproducible engineering metrics.

On knowledge and reasoning tasks, almost all decision models' actual accuracy is lower than the probabilities they give at each probability level of the selected answer, indicating poor calibration. In other words, when a model says "I am 80% confident," the actual accuracy does not reach 80%.

This calibration deviation is largest on AA-Omniscience-Decision and HLE-MultiChoice, while most models are relatively better calibrated on WildGuardTest (classification) and When2Call (tool calling). The weakness is therefore not evenly distributed, but concentrated in the two task types of knowledge and reasoning.

This point directly affects the practical use of decision models: if downstream systems rely on the probabilities output by the model for threshold judgments or routing, then in knowledge and reasoning scenarios, this probability will be systematically overestimated. The source does not give a timetable for whether the calibration deviation will improve with version iterations.

On classification and tool-calling tasks, leading decision models perform comparably to language models, but are significantly faster and cheaper; on knowledge and reasoning, language models lead by a wide margin. This is the core conclusion given by the publisher, and the most operationally valuable point of the entire index.

Looking at this conclusion together with the calibration results, the dividing line becomes clearer: decision models are both on par and better calibrated on classification and tool calling, and both behind and worse calibrated on knowledge and reasoning. The gaps in the two task types point in the same direction rather than canceling each other out.

What requires restraint is that the specific gap values for "comparable performance" are not quantified in the source. Therefore this dividing line is a directional judgment, not a threshold that can be divided by scores.

All the above data and conclusions come from content published by Artificial Analysis itself, with no third-party independent verification provided. This is an inference: with only a single publisher, rankings and scores should be regarded as that organization's evaluation results, not industry consensus.

The index is version v1.0, and the publisher states it will continue to be developed, so metrics and results may change. AA-Omniscience-Decision is a private dataset that cannot be reviewed externally; WildGuardTest uses only the 835-row subset where annotators were fully consistent and excludes questions that persistently trigger refusals; HLE-MultiChoice is text-only multiple-choice. The three filters stacked together mean that the applicable scope of this report card is narrower than its title suggests.

The source also does not state whether the index will be reproduced or adopted by third parties. Before these conditions are filled in, the more prudent reading is: treat Decision Index v1.0 as a map of trade-offs, not a ruler that can be used directly for procurement decisions.