Artificial Analysis has released the "Search Index," a benchmark that rates search API providers for AI agents on quality, cost, and speed. Of seven providers tested, Parallel, Exa, and Firecrawl scored highest. The benchmark aims to help developers choose the best search API for their AI agents.
SynthePulse Insight · AI deep reading
Version 1 · 1 source
Artificial Analysis releases the Search Index benchmark, the first systematic evaluation of search API quality, cost, and speed in AI agents. Results show that quality is inversely related to total cost, and raw speed is not the key to total time.
Artificial Analysis has released a benchmark called 'Search Index' designed to measure the quality, cost, and speed of search API providers in AI agent scenarios. The first batch of test subjects includes seven providers: Parallel, Exa, Firecrawl, You.com, Tavily, Keenable, and Brave.
The test uses a standardized agent setup, with all providers using the same model, GPT-5.6 Luna, and only the search provider is changed. The agent runs on Artificial Analysis's open-source framework Stirrup, performing 25 searches and webpage fetches per task.
The benchmark consists of three equally weighted subtests: DeepSearchQA includes 900 research questions requiring multiple searches; the BrowseComp subset tests 200 hard-to-find facts requiring multi-step browsing; and AA-Omniscience covers 600 questions across 6 knowledge domains. Additionally, a no-tool baseline is set, where the model answers independently, serving as a comparison benchmark.
Results show that without search access, the model scores only 33 points, while with search, scores range from 65 to 75. Among them, Parallel, Exa, and Firecrawl lead with 75, 74, and 73 points, respectively.
This comparison highlights the significant impact of search tools on the quality of AI agent responses, and also shows that quality differences among search APIs can be as much as 10 points, which has a substantial effect on the end-user experience.
The benchmark found that better search quality reduces total cost. When the model gets good results in the initial stage, it uses fewer tokens. Specifically, Parallel Search (premium) uses over 40% fewer tokens than the basic version.
Although the premium version has a higher cost per search, the total cost is actually lower; for example, Parallel's premium version costs $0.084 in total, while the basic version costs $0.11. This indicates that when evaluating search APIs, one should not only look at the price per query but also consider the overall task cost.
Raw speed does not always mean faster overall results. Parallel Search (turbo) has the shortest response time per query at 0.51 seconds, compared to 1.03 seconds for the basic version, but its quality is lower (67 vs. 73), causing the agent to perform more search rounds.
Ultimately, the total task time for the turbo and basic versions is roughly the same. This shows that in agent scenarios, low-quality search may offset speed advantages due to multiple retries, so evaluation should focus on end-to-end time rather than per-query latency.
Artificial Analysis states that Parallel, Firecrawl, and Parallel (turbo) achieve the best balance between cost and performance. Other providers can apply to join the benchmark, and the full methodology is publicly available.
The release of this benchmark provides a quantifiable reference for AI agent developers, helping them balance quality, cost, and speed when choosing search APIs.
The information in this article is sourced from THE DECODER's report, which is a secondary source. All specific data (such as scores, costs, times) come from the benchmark published by Artificial Analysis, but have not been independently verified and should be regarded as source claims rather than confirmed facts.
The Search Index benchmark shows that the quality of search APIs directly affects the total cost and efficiency of AI agents, so developers should prioritize quality over raw speed or per-query price.
Primary report
Primary source