Armature Study: How Coding Agents Choose Tools, Old Brand Impressions May Hinder

Armature, a growth services company for developer tools, published an experimental analysis attempting to answer a question that few had systematically measured before: how coding agents view, discover, and select tools, and which tool ultimately wins in each category. The study observed 17,000 tool selection sessions, but the core data presented publicly comes from a smaller, verified subset of 5,292 sessions.

The backdrop of this research is AI shifting focus from search engine optimization (SEO) to answer engine optimization (AEO), which means optimizing content so it can be presented as agent-generated answers. Armature co-founder Theodore Otzenberger's judgment is that agents now truly determine which tools get integrated into codebases, which has a 'life-or-death impact' on developer tool vendors—if a tool cannot be mentioned and selected by agents, it may very well be excluded from tech stacks in the future. It should be noted that Armature itself is a developer tool growth services company with a very direct interest in this game, and it was founded in 2026, so this type of analysis is still only its early research attempt.

The study covered 75 mutually independent code repositories, designed 1,163 different prompts, and tested three coding agents: Claude Code, Codex, and Cursor. The focus was not just on which tools they recommend, but on observing what tools they ultimately select and integrate in real codebases.

The analysis was based on public GitHub repositories. The team recorded the project's programming language, framework, third-party services, deployment platform, team size, and codebase age, and adjusted the sample with reference to public data to more closely match a set target structure. To reduce bias, the team used fictional company names, fabricated Git commit histories, and forged API keys, and also added a simulated human component played by Gemini 3.7 Flash through an orchestrator.

This simulated human component is not decorative. Armature explains that the simulated human typically accepts the top-ranked option, or lets the coding agent choose the most suitable option and complete the implementation; but if the agent is initially asked not to ask questions and just start working, it tends to build all functionality itself, because there is no opportunity to ask whether a third-party solution could be adopted. After adding the 'human' component, the originally overly strong advantages of top tools and cloud platform native solutions were weakened, making the results closer to reality.

The sample structure itself has constraints: public open-source projects skew toward startups, and enterprise software projects often do not appear directly on GitHub, so the team could only adjust the sample with reference to public data. Therefore, whether the study's conclusions can be generalized to enterprise software projects remains uncertain.

Armature found that code repository context is crucial. When asked to choose the best email/communication service provider, four codebases written in different languages ultimately chose four different winners: Resend won in TypeScript code (winning 55 out of 89 runs), SendGrid won in Python (22 out of 24), Postmark won in Go (20 out of 24), and Azure ACS won in Java (22 out of 23).

The meaning of these numbers is that tool selection is not completed on a unified 'best tool' leaderboard, but is deeply shaped by the codebase's language and existing tech stack. For the same type of need, change the language, and the winner changes.

Differences in information sources among agents further amplify this divergence. Cursor made decisions based on web information in two-thirds of sessions; Codex almost always used web search (94% of sessions), and used search operators like site in nine out of ten queries; Claude Code mainly relied on its own existing knowledge, searching the web only about 30% of the time, but once it started browsing web pages, the number of pages visited was three times that of Codex.

As a result, the three agents chose the same tool in only 42% of test units; and Claude Code chose to 'build its own' at nearly twice the rate of Codex and Cursor (19% vs. 10%). This means there is no single answer to 'what will the agent choose'—it depends on which agent, what codebase, and whether it is willing to search online.

Jeet Pattanaik, founder and CTO of Glokal AI OÜ, pointed out that being mentioned and ultimately winning are not the same thing: PayPal was mentioned 139 times but never selected once; LangChain was the most mentioned framework, appearing 194 times, but was ultimately selected only 4 times. This gap is exactly why this type of growth services business exists.

Pattanaik's judgment is that what truly influences agent decisions are often seemingly trivial details, so what vendors need to optimize next, besides repository context itself, are the tool's accompanying documentation and pricing pages—these contents may increasingly be read by agents rather than humans. It does not skim, is not more favorably inclined because a logo looks good, and even takes footnotes like data retention periods literally and seriously.

It should be distinguished that the above observations about the mention-selection gap come from Pattanaik, a third-party commentator, and are source claims rather than confirmed conclusions of Armature's research itself; whether the gap between being mentioned and ultimately being selected stably applies to other tool categories has also not been evidenced by the source. Whether vendors optimizing documentation and pricing pages for agents can actually change agent choices is likewise still unverified.

Otzenberger offered a sharper metaphor: in 17,000 tool selection sessions, twenty years of brand building by tool vendors seemed frozen in time, with brand reputation entering model weights and being bound to the product that made the vendor famous.

His example is containers: once containers are mentioned, the agent immediately thinks of Docker, but it knows nothing about the sandbox features Docker currently offers, so in the end it actually does not choose this tool. In other words, what the model remembers is the old image that made the vendor famous, not its current capability list.

This judgment comes from the statement of Armature's co-founder and is a source claim. It points to a structural risk: the stronger and more rigid the brand perception, the greater the agent's cognitive bias about the product's current state may be, and it is difficult for vendors to directly correct the impression in model weights through traditional marketing.

This study publicly released the entire research, including all traces and prompts, for anyone to verify, and Otzenberger said the team will soon rerun the complete study on Astra and Fable 5.1. He emphasized that with each new agent or model, tool selection results may be reshuffled, and understanding how these agents make choices has become a problem that needs to be solved with engineering methods.

But the boundaries are equally clear: the core data presented publicly comes from a subset of 5,292 sessions rather than the full 17,000 sessions; the sample skews toward public open-source projects, and applicability to enterprise software scenarios is unknown; whether the conclusions have been independently verified externally is not stated by the source; whether results after rerunning on Astra and Fable 5.1 will change existing conclusions is currently unknown; and the specific magnitude of the simulated human component's impact on results is only qualitatively described by the source.

Therefore, a more prudent reading is: this study provides a verifiable methodology and a set of specific observations, demonstrating that agent tool selection has significant language dependence, inter-agent divergence, and a 'mention-selection' gap; but it is not yet sufficient to assert that a stable new order has formed in any tool category. The real variable is the iteration speed of models and agents—each new generation may put today's winners back in line.