Learn
What is SWE-bench Verified?
Last reviewed July 16, 2026
SWE-bench Verified is a benchmark that hands a model a real GitHub issue in a real open-source repository and asks it to produce a patch that resolves it, graded by running the project's own test suite. The Verified subset was human-reviewed to remove broken or unsolvable tasks from the original set. Because success requires reading a codebase, localizing a bug, and editing without breaking other tests, it is the strongest single signal for agentic software engineering.
Why Verified exists
The original SWE-bench included tasks with underspecified issues, impossible setups, or tests unrelated to the described bug, so failures did not always mean incapability. Human annotators screened tasks for solvability and fair grading, producing the Verified subset that vendors now report. It measures the same skill with far less noise.
What a score does and does not mean
The task format is end-to-end engineering, so scores correlate with real agentic coding ability better than contest benchmarks do. Caveats: the repositories are Python-heavy open-source projects, harnesses and scaffolding differ across reported numbers, and resolving an issue is not the same as writing production code a reviewer would accept unchanged.
Top models on SWE-bench Verified
Full leaderboardPeak scores from the compiled benchmark data; reasoning models use their highest effort tier.
idapt's coding leaderboard ranks by SWE-bench Verified, and agents in idapt do the same kind of work: read a codebase from Drive, edit files, and run its tests on a cloud computer.
Frequently asked
See it in practice
The fastest way to understand it is to use it. Start free, no credit card.
Try idapt free