LoopsBench: A Benchmark for Long-Horizon Software Engineering with Coding Agents

Existing Coding Agent benchmarks (e.g., SWE-bench) typically judge success by whether final tests pass, but with new capabilities like continuous execution and Goal Mode, final results alone are insufficient to capture the execution process. LoopsBench argues that in long-horizon tasks, agents need to identify dependencies, progress in a reasonable order, and prevent regression of completed functionality.

LoopsBench represents tasks as a Dependency DAG: nodes are actual engineering units, and edges represent evidence-supported prerequisite dependencies. This DAG is not the only correct order but serves as an evaluation contract, allowing parallel or redo actions as long as dependencies are satisfied.

Task sources include university course labs, continuous PRs from open-source projects, and research code evolution, totaling 112 tasks: 57 Course Labs, 29 PR Sequences, and 26 Research Evolutions.

LoopsBench designs a Flow-aware Evaluation Runtime: only when prerequisites are met does a Development Unit enter the Ready Frontier, and tests advance along the DAG as the agent progresses. This allows observation of how far the agent advances, rather than merely counting pass rates.

Once a unit is completed, its tests are retained, becoming a Regression Obligation. Each step forward requires maintaining more completed functionality, accumulating regression pressure.

The Runtime separates editing and evaluation environments, recording state as code changes to generate a trajectory over time, capturing regression dynamics.

In benchmark tests, the best configuration (highest configuration + outer continuation) achieved a Resolve Rate of 25.00% and a Test Pass Rate of 53.05%, meaning about three-quarters of tasks were not fully resolved. Varying the Loop with a fixed model or varying the model with a fixed Loop produced differences, indicating that Model and Loop separately affect local reasoning and long-term organization.

Introducing continuation improved performance: one configuration's Resolve Rate increased from 16.96% to 25.00%, another from 14.29% to 21.43%, but it could not automatically solve subsequent task selection and progression path issues.

Analysis shows that agent plans cover only part of the dependencies, patches are typically longer than Gold References, and fewer agent-authored tests are established, leading to regression in completed units during subsequent modifications.

The paper compares Goal Mode, Dynamic Workflows, and loop mechanisms based on fresh invocation. Dynamic Workflows produce more independent context rounds and achieve higher Resolve Rates; Goal Mode advances by maintaining long-term goals; fresh invocation performs relatively weaker on complex tasks.

However, no mechanism completely eliminates regression, indicating that simply adding or refreshing context is insufficient to solve all problems in long-term execution.