Cognition Launches Leaderboard for FrontierCode 1.1

The core change in the FrontierCode 1.1 leaderboard is not in the models but in the scoring criteria. Artificial Analysis announced the launch of this private evaluation from Cognition, which tests not whether a model can write code that runs, but whether its changes can actually be merged by the maintainers of real-world repositories. Cognition released FrontierCode in June 2026 and updated it to version 1.1 in July; the dataset was built jointly by the company and open-source maintainers, with each task taking over 40 hours.

The task format determines where the difficulty comes from: each task requires the agent to make a single change to a real open-source repository, then checks both unit tests and the qualitative tests maintainers need for merging. This means "all tests green" is only the entry ticket, not the finish line. Artificial Analysis runs FrontierCode 1.1 Main, the primary 100-task set out of the benchmark's 150 tasks, and uses its Stirrup open-source reference harness to enable same-condition comparisons across models; all tasks are private to avoid data contamination.

FrontierCode's scoring rules are the premise for understanding its score distribution. Each attempt is scored simultaneously against blocking and non-blocking criteria: failing any blocking check yields zero, while passing all of them yields a score between 70% and 100%. This design turns the evaluation from continuous scoring into a tiered judgment with a hard threshold—a model is either shut out entirely or lands in the 70-plus range.

Unlike other coding evaluations, FrontierCode also assesses behaviors such as excessive changes, large diffs, modifying files outside the relevant scope, and adding useless tests. These metrics do not map directly to functional correctness, but they map directly to whether a maintainer is willing to press the merge button. The judgment Artificial Analysis offered in its announcement is that models are now very good at coding tasks, but their changes often still require substantial cleanup before maintainers can merge them. This judgment is the publisher's own statement, not an independently verified conclusion.

Among the models evaluated at launch, Claude Opus 5.5 (max with fallback) leads with a score of 52%, with 58% of its attempts passing the blocking checks. The gap between these two numbers is worth noting: even among attempts that pass the blocking checks, the final score does not necessarily land at the upper edge of the range, indicating that non-blocking criteria continue to deduct points.

The top is very dense: nine models from six labs score between 43% and 52%. In the open-weight camp, MiMo-V2.6-Pro leads at 45%, followed by DeepSeek V4.1 Flash (max) and Kimi K3 (max) at 44% each. Nine models packed into a nine-point range means that under this evaluation criteria, the capability gap among leading models has been compressed into a very small band.

Another set of numbers points to the capability boundary: 9 of the 100 tasks were not passed by any attempt from any model. The specific content of these tasks and the reasons for failure were not disclosed in the source, so it is impossible to determine whether they are task design issues, blind spots in model capability, or some extreme case of maintainer standards.

The third point the publisher offered is cost efficiency. GPT-6.1 Sol (max) achieves a score of 49% at $0.81 per task, tied with Claude Fable 5.1 (max with fallback); Artificial Analysis says it reached comparable performance at less than one-tenth the per-task cost of the leading Anthropic model.

It must be made clear that this cost comparison is the publisher's own statement: the source does not give the specific cost figure for the comparison model, so "less than one-tenth" cannot be independently verified. Likewise, all models' scores and cost data were run and published by Artificial Analysis itself, lacking independent third-party verification. These numbers can serve as a reference for observing direction, but should not be treated as confirmed cost conclusions.

From the standpoint of evidence strength, the mechanistic facts of the evaluation are relatively clear: the number of tasks, how they were built, the harness used, the scoring rules, the private anti-contamination design, and each model's score and blocking-check pass rate all come from a single post published by Artificial Analysis on X. That post is the publisher's own statement and provides no first-party evaluation report or third-party review.

Several key pieces of information are missing from the source, directly affecting the boundaries of interpretation. First, the statement that "leading models still have a failure rate above 40%" does not specify the exact statistical basis for the failure rate, making it impossible to determine whether it refers to task pass rate, attempt pass rate, or some other denominator. Second, which six labs the nine models come from and the specific model list are not fully enumerated in the source. Third, the details of the division of labor between Cognition and Artificial Analysis—who handles scoring and who handles running—are not explained in the source.

In addition, this leaderboard covers only the 100 tasks of FrontierCode 1.1 Main, not all 150 tasks. Artificial Analysis says it will add FrontierCode to its coding agent leaderboard soon, but "soon" is a vague statement with no timetable.