SWE-bench

SWE-bench evaluates coding agents on 2,000+ real GitHub issues from popular Python repos: given the repo and the issue text, produce a patch, judged by the project’s own (held-out) tests. It measures what toy puzzles can’t — navigating a large codebase, localizing the fault, editing without breaking everything else. SWE-bench Verified is the human-validated 500-issue subset most frontier-lab claims cite. When a model announcement says “X% on SWE-bench,” this is that.

Worked example: a benchmark measuring whether an AI system can resolve REAL GitHub issues from open-source repos — given the issue text and codebase, produce a patch that makes the project’s hidden tests pass. Gotcha: it is hard because it needs multi-file understanding, not single-function coding, and scores are sensitive to the harness/scaffolding (retrieval, test-running, retries) as much as the model — so compare numbers only within the same setup, and beware contamination from issues in the training data.