AI Teams Still Fail About Half the Time: How AgentWorld Benchmarks Multi-Agent Collaboration

Individual agent capabilities continue to strengthen, but whether team-level collaboration improves in tandem is a question that requires separate verification. AgentWorld is positioned precisely to pull this question away from "plausible-sounding dialogue" back to "actions that must actually occur": it combines long-horizon tasks, role asymmetry, and black-box interaction into a single collaborative evaluation environment.

The task set consists of 100 human-designed tasks and 100 augmented variants, covering categories such as combat, crafting, gathering, trading, exploration, survival, building, and coordination. Tasks require 3 to 20 agents to participate, and many tasks involve sequential resource handoffs and action dependencies.

Role asymmetry means different agents possess different skills, items, and initial conditions, and the team must allocate work accordingly; black-box interaction means agents cannot directly read teammates' internal reasoning states and can only infer what teammates are doing through environment observations, chat, and action outcomes; multi-round execution requires the team to continuously update state within a budget of dozens of rounds. To reduce low-level operational interference, AgentWorld provides 13 high-level API tools; for example, gathering or attacking tools can encapsulate navigation and specific execution flows.

Under a unified prompt template, tool definitions, and interaction protocol, the paper evaluated Gemini 3 Flash, Claude Haiku 4.5, GPT-5 Mini, and DeepSeek R1-70B. On the main task set, the team task success rates driven by the four models were 52.0%, 45.0%, 36.0%, and 20.0%, respectively; on the augmented task set, the corresponding success rates were 24.0%, 26.0%, 21.0%, and 10.0%.

The highest success rate on the main task set was 52.0%. This number alone does not indicate that collaboration capability has approached some ceiling, but it does concretize the problem: when tasks require cross-role handoffs, continuous synchronization, and multiple dependencies, team-level completion rates are noticeably lower than intuitive expectations based on individual capabilities.

More noteworthy is the gap between "getting partway there" and "actually finishing." Gemini 3 Flash had a partial success rate of 71.5% on the main task set and a full task success rate of 52.0%. The team advancing some intermediate goals does not guarantee that the final resource handoff or crafting step can be completed.

There is no simple "more is better" relationship between communication volume and success rate. GPT-5 Mini sent an average of 44.1 chat messages per main task with a 36.0% success rate; Gemini 3 Flash sent an average of 11.0 messages with a 52.0% success rate.

This set of results does not prove that chatting less improves success rates, but it reminds us that message volume cannot directly substitute for collaboration quality. What deserves more attention is whether a message updates effective information, whether it eliminates ambiguity in the division of labor, and whether the recipient takes action based on it.

The paper's analysis of communication traces summarizes multiple failure types: repeatedly asking about already-resolved issues, misidentifying roles or resource recipients, reporting incorrect item information, and declaring the task finished before critical steps are complete. These phenomena indicate that multi-agent systems need to maintain not only the "task plan" but also the constantly changing team state.

Counting success rates alone is insufficient to describe how a team works. Two teams both complete a task—one with clear division of labor and smooth handoffs, the other experiencing extensive redundant gathering, ineffective waiting, and information misunderstandings. If only the final outcome is considered, both receive the same success label.

To address this, the paper proposes Causal Collaboration Effectiveness (CCE). For successful tasks, CCE equals the number of actions judged to contribute to success divided by the total number of actions executed by the team. It traces backward in time starting from the actions that completed the goal: which materials did this crafting depend on? Which transfer did the materials come from? Whose gathering or processing did the transfer depend on? The algorithm uses a large language model to judge causal dependencies between candidate actions, progressively constructing an action relationship graph.

In the paper's illustrative diagram, three agents execute a total of 27 actions, of which 12 are included in the contribution chain leading to task success, so the CCE of that illustrative trajectory is approximately 44.4%. This ratio does not represent the average result across the entire dataset. The paper scores failed tasks as CCE 0 by definition; this is the metric's scoring convention and does not mean that every attempt in a failed process was worthless.

CCE still relies on a large language model's judgment of causal dependencies between candidate actions and cannot alone prove that a system possesses general collaboration capability. It should be read in conjunction with success rate and partial success rate, and should not be treated as an independent "team chemistry score."

The paper's authors state that AgentWorld's experimental results correspond to specific models, prompts, tool interfaces, and execution protocols, and are not sufficient to attribute collaboration difficulties to a universal ceiling across all large models. Whether richer planning, memory, or communication designs can improve collaboration performance remains, according to the source, an open area for exploration.

The source does not provide an average CCE value across all tasks, only the 12/27 from a single illustrative trajectory; the source also does not explain the specific design details of how the augmented task set differs from the main task set. These gaps mean that the current results are better read as a reviewable starting point rather than a final judgment on multi-agent collaboration capability.

The paper has been accepted to COLM 2026 and was jointly completed by researchers from OpenAgents, Columbia University, the University of Pennsylvania, Seoul National University, and Pennsylvania State University, with co-first authors Raphael Shu, Yusen Zhang, and Young Min Cho.