Artificial Intelligence coding harness quality benchmarks

Research Question

What benchmarks, metrics, and evaluation methodologies are used to measure the quality of Artificial Intelligence (AI) coding harnesses, including Integrated Development Environment (IDE) plugins, agentic coding assistants, and code completion tools, and which vendors and open-source projects score highest on those measures as of 2025-2026?

Findings

Executive Summary

Public evidence for AI coding harness quality currently favors end-to-end software engineering benchmarks, with SWE-bench Verified carrying the most decision weight because it evaluates whether systems actually resolve real repository issues rather than merely emit plausible standalone code.

The strongest public leaderboard positions visible in accessible official sources are mostly held by simple or open harnesses paired with high-performing current models, while several branded assistants in this item are represented instead by controlled studies or internal evals rather than directly comparable leaderboard entries.

That means tool selection in 2025-2026 should weight benchmark family and evidence credibility before raw score, because HumanEval, Mostly Basic Programming Problems (MBPP), Aider, GitHub Copilot studies, and CursorBench all measure different slices of quality.

GitHub Copilot and Cursor both publish public evidence, but their retrieved official evidence is controlled-study or internal-eval evidence rather than first-party public leaderboard parity with benchmark-native harnesses such as mini-SWE-agent, OpenHands, Amazon Q Developer Agent, or Google Jules.

The best-supported answer to the research question is therefore that no single benchmark is authoritative, but SWE-bench Verified is the strongest public anchor, and the current visible leaders in directly comparable public leaderboard evidence are open or benchmark-native agent systems.

Key Findings

1. SWE-bench Verified is currently the most decision-useful public benchmark for comparing agentic coding harnesses, because it uses human-filtered real GitHub issue tasks and scores whether a system actually resolves repository problems rather than merely generating plausible standalone code.
2. HumanEval and Mostly Basic Programming Problems (MBPP) remain important coding benchmarks, but they are weak direct proxies for harness quality because they evaluate standalone Python problem solving and unit-test passing rather than repository navigation, multi-file editing, and regression-safe issue resolution.
3. LiveCodeBench and BigCodeBench extend benchmark coverage beyond older standalone-function tests by emphasizing contamination resistance, self-repair, code execution, tool or library use, and harder instructions, yet they still rank models more directly than branded coding products or end-to-end harnesses.
4. The dominant public metrics are pass@k for sampled code generation and percent resolved for software engineering agents, while productivity studies add unit-test pass rates, approval rates, and readability or maintainability ratings that capture important but different aspects of coding-tool quality.
5. The highest public SWE-bench Verified scores visible in accessible official sources are attached primarily to simple or open harnesses such as live-SWE-agent and mini-SWE-agent paired with current top-scoring models, with retrieved top entries at 79.2% for live-SWE-agent plus Claude 4.5 Opus medium and 76.8% for mini-SWE-agent plus Claude 4.5 Opus.
6. Among named open-source platform entries with direct public attribution, OpenHands has become a serious top-tier benchmark participant, appearing at 65.8% as a branded entry and 70.4% when paired with Claude 4 Sonnet on the retrieved SWE-bench Verified leaderboard.
7. Among named commercial products with directly attributable public Verified entries in retrieved official sources, Amazon Q Developer Agent presently has stronger public end-to-end benchmark evidence than Google Jules, at 65.4% versus 52.2% on SWE-bench Verified.
8. Devin's published 13.86% result on original SWE-bench was historically important because it normalized agent-style evaluation, but it should not be read as a current leaderboard position because it used a 25% subset of the older benchmark and not the current Verified leaderboard regime.
9. GitHub Copilot and Cursor both publish public evidence, but GitHub's evidence is primarily randomized controlled productivity and code-quality research while Cursor's evidence is primarily its internal CursorBench methodology, so neither currently offers first-party public leaderboard comparability with benchmark-native open harnesses.
10. Benchmark credibility is now part of the quality question itself, because contamination risk, narrow grading, and benchmark-workflow mismatch are explicit public concerns in both LiveCodeBench and CursorBench, and practitioner trust in complex-task performance remains mixed even as usage grows.

Assumptions

• If a popular branded tool did not appear in the retrieved official public leaderboard sources, this item treats that as missing public evidence, not as proof that the tool lacks any strong internal or private benchmark performance.
• When a score is reported for a harness paired with a frontier model, the analysis attributes the result to the combined system rather than claiming the harness alone deserves the full score.
• Aider benchmark results are used as evidence for harness-specific editing quality even though they are not directly comparable to SWE-bench Verified issue-resolution results.

Analysis

• End-to-end harness selection should weight benchmark families by how much real software engineering behavior they contain, not by historical fame alone.
• That weighting puts SWE-bench Verified at the top for public harness comparison, followed by adjacent repository-scale proxies such as SWE-bench Lite and harness-specific editing suites such as Aider, while HumanEval and MBPP become supporting evidence about base-model coding ability.
• The public leaderboard leaders now show that simple scaffolds plus strong models can outperform more elaborate branded products, which means a vendor's user interface or market visibility should not be mistaken for public benchmark leadership.
• At the same time, products such as GitHub Copilot and Cursor are not evidence-free; they simply publish different evidence types, namely controlled studies and internal eval loops, which are useful for workflow fit but weaker for external comparability.
• The resulting decision rule is to combine public benchmark strength, benchmark-family fit, and evidence credibility, and to treat any ranking that crosses those boundaries without adjustment as methodologically unsound.

Risks, Gaps, and Uncertainties

• Public benchmark scores move quickly, so any 2025-2026 leader table is time-sensitive.
• Several major products do not publish first-party public scores on shared harness benchmarks, which limits apples-to-apples comparison.
• Static public benchmarks may miss long-running, ambiguous, externally integrated, or highly collaborative engineering tasks.
• Official OpenAI explanatory pages about SWE-bench Verified were linked from the SWE-bench site but not directly fetchable in this runtime, so Verified methodology claims here rely on the accessible SWE-bench Verified page itself rather than on the linked OpenAI copy.

Open Questions

• What public benchmark can reliably measure multi-day or multi-session engineering work that crosses external services, review loops, and deployment boundaries?
• Which metric best captures post-generation review burden, not just whether tests eventually pass?
• Will major commercial IDE assistants converge on shared public harness benchmarks, or will internal evals such as CursorBench become the dominant decision surface?

sources

• [x] Princeton Natural Language Processing Group SWE-bench
• [x] Jimenez et al. (2024) SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
• [x] Princeton Natural Language Processing Group SWE-bench Verified
• [x] Princeton Natural Language Processing Group SWE-bench Lite
• [x] Chen et al. (2021) Evaluating Large Language Models Trained on Code
• [x] OpenAI HumanEval repository
• [x] Google Research Mostly Basic Programming Problems dataset
• [x] Jain et al. (2024) LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
• [x] LiveCodeBench leaderboard
• [x] BigCodeBench leaderboard
• [x] BigCodeBench OpenReview submission
• [x] Aider leaderboards
• [x] Stack Overflow 2024 Developer Survey, AI
• [x] Cognition Labs technical report, SWE-bench results for Devin
• [x] Cognition Labs introducing Devin
• [x] GitHub Copilot productivity study
• [x] GitHub Copilot code quality study
• [x] GitHub Copilot coding agent announcement
• [x] Amazon Q Developer
• [x] Google Developers Blog, The next chapter of the Gemini era for developers
• [x] Cursor blog, CursorBench
• [x] OpenHands