SWE-bench
also: SWE-bench Verified
The benchmark behind every coding-agent claim: resolve real GitHub issues in real repos, graded by real tests.
SWE-bench evaluates coding agents on 2,000+ real GitHub issues from popular Python repos: given the repo and the issue text, produce a patch, judged by the project’s own (held-out) tests. It measures what toy puzzles can’t — navigating a large codebase, localizing the fault, editing without breaking everything else. SWE-bench Verified is the human-validated 500-issue subset most frontier-lab claims cite. When a model announcement says “X% on SWE-bench,” this is that.