NeurIPS 2026 | Don't Wait for Agents to Fail! 7B Auditor AgentForesight Catches Errors Online

A multi-agent failure does not necessarily show up as an error on the spot. The more troublesome case is this: one Agent gives a wrong premise, while other Agents continue to execute diligently. The later Agents may not have done their own subtasks wrong; they simply treated the error from the earlier step as the premise for continuing their work.

A location question-answering trajectory in the paper illustrates this risk: the system needs to find an old industrial town near Rivington Hall Barn, the case's annotated answer is Bolton, but the search Agent returns Horwich, and the Manager then carries that result forward and submits the wrong answer. Later Agents repeating an answer does not make it more reliable evidence.

This is exactly the starting point of AgentForesight: to raise an early warning when a local error has already appeared but has not yet propagated all the way into task failure. It moves post-hoc attribution forward into online auditing—after each completed step, an external auditor reads the execution record up to that point and decides whether to continue or raise an alarm, buying time for subsequent intervention.

Many past multi-agent failure analyses used post-hoc attribution: first read through a trajectory already known to have failed, then determine the responsible Agent and the critical error step. How later errors spread and why the final answer was wrong can all serve as clues for localization.

Online auditing has none of this hindsight. AgentForesight can only see the current trajectory prefix, that is, the record of what has already happened up to this point; future actions, tool results that have not yet returned, and final success or failure are all outside its view.

The difficulty is therefore not just finding an error, but distinguishing: is the system still exploring reasonably, or has a decisive error affecting task success already appeared? The former should not be interrupted without cause, while the latter should be identified as early as possible before downstream Agents carry it forward. An alarm also cannot just say something is wrong; the auditor must point out which step the error occurred at, which Agent introduced it, and give a brief reason.

Moving post-hoc attribution online means the training data must change too. With only failed trajectories, the model cannot learn when the task should continue; with only success-or-failure labels for the whole task, it cannot be told from which step the problem began.

To this end, the team built AFTraj-2K, covering code generation, mathematical reasoning, and Agentic tasks involving tools and retrieval, containing 2,272 annotated trajectories: 1,158 verified successful trajectories and 1,114 failed trajectories.

But getting the final answer right does not mean every step in the process is reliable. Trajectories that went wrong midway but were corrected by other Agents cannot be indiscriminately treated as usable success samples. The team therefore simultaneously checks the final result, the complete validity of tool calls, and the consistency of each step with the current subgoal, filtering out successful trajectories usable for prefix supervision.

Failure data comes from two paths: one injects errors into verified successful trajectories and checks whether the modification takes effect and whether the task indeed fails; the other proposes candidate locations from natural failures and then, through multiple independent verifications, checks whether the error truly exists, whether it substantially affects the task, whether it is decisive, and whether it is the earliest decisive error. What is sought here is not the most conspicuous anomaly, but the key step that can change the task's outcome.

With step labels in hand, why still train in stages? Because a general model faces two interrelated thresholds: first, distinguishing whether the current record has already crossed the failure boundary; then, within an erroneous record, localizing the specific step and responsible Agent. Directly asking it to learn all abilities at once makes the reward signal for precise attribution too sparse, and training may degenerate into always answering safe.

The first stage uses Boundary Pairwise Preference Optimization (BPPO) to learn the failure boundary. The team takes two adjacent prefixes from the same failed trajectory: one stops before the decisive error and should continue; the other adds only the step where the error just occurred and should raise an alarm. The task background and prior execution history remain unchanged, yet the judgment must flip with the key step, so the training signal focuses on the content that causes the risk to change.

The second stage starts from the first-stage model and gives rewards around the three dimensions of the audit conclusion. The step reward is not just all-right or all-wrong: the closer the predicted location is to the annotation, the higher the reward, providing a smooth signal for gradually pinpointing the error; both false alarms and missed alarms are penalized, preventing the model from taking shortcuts by always alarming or always letting things pass.

Training also uses the first-stage model as a reference for the KL constraint, limiting how far the second stage deviates from the risk judgment already learned. The two stages are not simply concatenated: the latter must, while preserving sensitivity to the failure boundary, refine something is wrong here into this step, this Agent has a problem.

The team evaluated auditing ability on 332 held-out test trajectories from AFTraj-2K. AgentForesight and the general models in the table all read prefixes step by step and are scored on the judgment in the first alarm. Exact-F1 examines both the detection and precise localization of the critical error, while ASS measures how many steps the reported error location deviates on average from the annotation.

The paper self-reports AgentForesight-7B's Exact-F1 reaching 66.44, 19.88 points higher than the strongest general baseline on this metric, DeepSeek-V4-Pro; the improvement over its base model Qwen2.5-7B-Instruct's 21.05 is also very clear. ASS drops from DeepSeek-V4-Pro's 1.77 to 0.59, about one third. What is measured here is the attribution location deviation on detected failed trajectories, not the waiting time from error to alarm.

By domain, Exact-F1 for math, code, and Agentic tasks is 77.36, 78.87, and 48.70 respectively; Exact-F1 and ASS lead the comparison methods in the main table across all three domains. On the external Who&When benchmark, which was not involved in training, AgentForesight-7B's accuracy for identifying the critical step and the responsible Agent is 57.69% and 73.08% respectively, with ASS at 1.62, all the best results among the comparison models listed in the paper.

The other half of the online auditing test lies in successful trajectories: it is not judging success once after the task ends, but at every not-yet-completed moment, not mistakenly halting a reasonable process. AgentForesight-7B's false alarm rate FAR on successful trajectories is 2.37%, and its Step Accuracy on failed trajectories is 59.51%; DeepSeek-V4-Pro's corresponding figures are 43.20% and 53.99%.

2.37% is calculated per whole trajectory: of the 169 successful trajectories in the test set, only 4 ever triggered a false alarm during step-by-step auditing. A trajectory counts as a false alarm if it alarms on any prefix at all, rather than averaging the error away with a large number of normal steps. These results therefore are not just fewer alarms, but low false alarms and better critical error identification appearing together.

The ablation experiments give a direct comparison: the base model's overall Exact-F1 is 21.05, training with only the first-stage BPPO is 35.63, training with only the second stage is 50.42, and training with the complete scheme reaches 66.44.

More telling is the Agentic task: with only the second stage, Exact-F1 is 19.05, lower than the 31.58 with only the first stage; only when the two stages are combined does it rise to 48.70. This echoes the division of labor between the two stages—critical errors in math and code are relatively easy to localize, and the localization reward alone can already yield good performance; whereas in retrieval, tool calling, and multi-role interaction, whether a decisive error has already occurred is harder to judge, making it especially important to first build the ability to recognize the failure boundary.

This also explains why one should not stare only at ASS: a model that alarms only in a few cases where it is confident may also obtain a very small location deviation. Whether localization is accurate must also be considered together with how many errors were missed.

AgentForesight explores an independent division of labor: execution Agents are responsible for advancing the task, while an external auditor specifically judges whether a critical deviation has already appeared in the process. It does not require retraining the underlying execution system; execution capability and auditing capability can be optimized separately, and the audit conclusion can serve as a basis for pausing, replanning, switching execution paths, or handing over to human inspection.

But the paper's experimental focus is still online identification and attribution. What action to take after an alarm, how many failures can be salvaged, and whether the final task success rate improves need to be verified in combination with specific intervention mechanisms.

The auditor itself can also err: the paper observes that processes that could have been corrected by a verification Agent on their own may still trigger false alarms, and detected errors may also have localization deviations. Step-by-step auditing increases invocation overhead, and its performance on longer workflows, embodied tasks, and open-ended scientific research tasks also remains to be further evaluated.

In addition, the metrics on AFTraj-2K and Who&When are all self-reported results from the paper, lacking independent third-party reproduction; metrics such as Exact-F1 and ASS depend on the paper's own annotation criteria, and the comparability of results under different annotation standards is unknown.