AI The new coding leaderboard tests whether code can be merged, and Opus 5.5 still has of attempts fail

Article

Beating AI Breaking news: Artificial Analysis has launched FrontierCode 1.1 coding leaderboard, focusing on evaluating whether AI code can meet open-source project maintainers’ merge standards. Claude Opus 5.5 scored 52.2%highest overall, but only 58%of attempts passed all hard checks, meaning more than still failed.

This benchmark was developed by Cognition together with maintainers of 36 open-source projects, with each task refined for more than 40 hours. It checks not only whether code runs, but also whether added tests can detect issues in the old code, whether unrelated files are modified arbitrarily, and whether existing functionality is broken. Artificial Analysis selected 100 of the more difficult private tasks, had all models use the same Agent framework, and ran each task 3 times. For 9 of the tasks, no evaluated model succeeded.

The cost gap is larger than the score gap.GPT-6.1 Sol scored 49.1%, only 5.5 percentage points below Opus, 3.1 but its average cost per task was only 0.81 USD, compared with 10.15 USD for the latter—a difference of more than 12 times. Xiaomi MiMo-V2.6-Pro led the open-source models with 45%, at a per-task cost of only 0.18 USD；DeepSeek V4.1 Flash and Kimi K3 both came in at 44%.

It is important to note that 52.2%is an overall score, not the actual PR merge rate. The benchmark used automated checks and reviews by the GPT-6 Luna model; tasks were not actually reviewed individually by maintainers. All models used a standardized Agent framework, which also means this ranking cannot directly represent the real-world performance of each company’s official coding tools.

Original link https://m.theblockbeats.info/flash/371531

Key points

Artificial Analysis has launched the FrontierCode 1.1 coding leaderboard, focusing on evaluating whether AI generated code can meet open-source project maintainers’ merge standards.
Claude Opus 5.5 The overall score is 52.2%, ranking first; 58%of attempts passed all hard checks.
The benchmark was developed by Cognition together with maintainers of 36 open-source projects, with each task refined for more than 40 hours.
The benchmark selected 100 private tasks; all models used the same Agent framework, and each task was run 3 times. Of these, 9 tasks had no successful submission from any evaluated model.
GPT-6.1 Sol scored 49.1%, with an average cost per task of 0.81 USD；Claude Opus 5.5 average cost per task 10.15 USD。
Xiaomi MiMo-V2.6-Pro led with 45%among the open-source models, while DeepSeek V4.1 Flash and Kimi K3 both came in at 44%。

AI-assisted interpretation

The following is analysis, separate from reported facts. Verify important claims independently.

This leaderboard looks beyond whether code runs: it also checks whether added tests can detect issues in old code, whether unrelated files are modified, and whether existing functionality is broken. The results show that the highest-scoring model and GPT-6.1 Sol have similar scores, but a large gap in per-task costs.

Why it matters to readers

It compares code mergeability, test quality, and cost within a single benchmark, helping readers understand AI programming models’ engineering performance and cost differences.

For beginners, a leaderboard score does not mean that code can definitely be merged directly; tests, the scope of changes, existing functionality, and actual review also matter.

Risks and unknowns

52.2%is an overall score, not the actual PR merge rate.
The benchmark uses automated checks and reviews by the GPT-6 Luna model; tasks were not actually reviewed individually by maintainers.
All models use a standardized Agent framework, so the ranking cannot directly represent the real-world performance of each company’s official coding tools.
The provided materials do not include Artificial Analysis’s original rankings or independent verification materials.

AI The new coding leaderboard tests whether code can be merged, and Opus 5.5 still has of attempts fail

Article

AI-assisted interpretation

Why it matters to readers

Risks and unknowns

Related Developments

Related concepts

Keep following

Security Center