SWE-bench & SWE-agent,
explained.

Rungs 1 and 2 both evaluate a function written from a specification in a vacuum. This rung changes the task to something recognisably the job: a real issue from a real repository, where the model must find the relevant code among thousands of files, understand conventions it did not write, and produce a patch. Success is defined operationally and strictly — the patch must apply, the previously-failing tests must pass, and the previously-passing tests must still pass. That last clause is what makes it a software engineering benchmark rather than a coding one: no credit for a fix that breaks something else.

The agent side then produced a finding that is more useful than any score on it. Giving a capable model a shell and a repository works poorly. Giving it a small, purpose-built set of operations works dramatically better — a file viewer that shows a bounded window and remembers position, an edit command that checks syntax before accepting a change and refuses it otherwise, scoped search that returns manageable results. Same model, same task, large difference from the interface alone.

The reason is the transferable part: an agent is not a human user. A human at a terminal benefits from general power and unbounded output, and recovers from a mess by looking at the screen. A model has a limited context that unbounded output destroys, cannot see, and will confidently build on a broken edit for many turns. So the ergonomics invert — narrow operations, small outputs, and immediate error feedback at the moment of the mistake beat general-purpose tools, because the cost of an undetected error compounds.

Benchmarks like HumanEval measure whether a model can write a small, self-contained function from a clear spec. Useful — but a poor proxy for what software engineers actually do. Real work means operating inside a large, unfamiliar codebase: interpreting a vague bug report, locating the relevant code among thousands of files, making a correct, minimal change, and confirming with the test suite that the bug is gone and nothing else broke. None of that is captured by "write this function."

SWE-bench closed the gap by building its tasks from reality. Each instance is a genuine GitHub issue that was actually resolved by a pull request in a popular open-source Python project, packaged with the repository as it was before the fix and the tests that validate the fix. The model gets the issue and the repo; it must produce a patch. It's the difference between a coding quiz and a real ticket in a real backlog.

How do you score a code change objectively? By running the tests — but SWE-bench is careful about which tests. It splits them into two sets and requires both to come out right for an issue to count as resolved:

The FAIL_TO_PASS tests are the ones that failed before the fix — they prove the bug is actually fixed, not merely that the patch runs. The PASS_TO_PASS tests already passed — requiring they still pass is a regression check, so a "fix" that quietly breaks something else does not count. And the patch must apply cleanly to begin with. All three, or it's not resolved. This is a strict, execution-based grade with no partial credit — exactly the honesty you want for real engineering. The runnable version below encodes this rule and the regression trap.

SWE-bench sets the task; SWE-agent asks how a model can actually do it. A language model can't magically edit a repo — it needs a way to look around and act. The naive approach is to hand it a normal shell. The paper's key finding is that this is a mistake: interfaces built for humans serve models poorly. Models get lost in verbose output, mangle multi-line edits, and waste turns on malformed commands.

The fix is a purpose-built Agent-Computer Interface (ACI) — a small set of commands designed for how a language model reads and writes:

The result was measurable: a well-designed ACI raised the resolve rate substantially over a plain shell, holding the model fixed. The lesson generalized far beyond this paper — how you let an agent act, not just how smart the model is, is a first-class design variable. Guardrails (like the edit linter) that catch mistakes before they compound turned out to matter enormously.

With the ACI in place, SWE-agent runs the familiar agent loop — the same reason-then-act cycle, specialized for coding. The model observes, decides on a command, runs it, reads the result, and repeats until it submits a patch:

It might search for the error message, open the offending file, scroll to the function, edit it (the linter catching a botched edit before it lands), run the tests, see two still failing, adjust, and finally submit. Every step is a turn where the model reasons about what it sees and picks the next action. The whole trajectory is bounded — limited turns and cost — so the agent must localize and fix efficiently. When it submits, SWE-bench applies the strict three-part grade from section 02.

The page covers two papers: SWE-bench (Jimenez and colleagues, 2023), which defines the task, and SWE-agent (Yang and colleagues, 2024), which builds an agent for it. The benchmark came out of three filtering stages, and the agent works through a purpose-built command set.

Building SWE-bench

1. Scrape pull requests. About 90,000 pull requests from 12 popular open-source Python repositories, including django, sympy, scikit-learn, matplotlib, pytest, sphinx and astropy.
2. Filter by attributes. Keep merged pull requests that resolve a linked GitHub issue and that change the repository's tests, since those tests are what will check a fix.
3. Filter by execution. Install each repository at the pull request's base commit, apply the pull request's test changes, and run the tests before and after its code changes. Keep an instance only if at least one test goes from failing to passing and nothing errors during installation or running. 2,294 task instances survive.
4. Pose the task. The model sees the issue text and the repository at the base commit, and must return a patch. Grading applies the patch and runs the fail-to-pass and pass-to-pass tests.

The original paper's baselines wrote a patch in one shot. Retrieval chose which files went into the context: BM25 over the repository, filling a budget of up to about 50,000 tokens, or an "oracle" that supplied exactly the files the reference patch edited. The authors also fine-tuned CodeLlama-Python 7B and 13B into SWE-Llama on about 19,000 issue and pull-request pairs from 37 other repositories.

How SWE-agent works an issue

1. Search with bounded output. find_file, search_file and search_dir return a summary of matching files and lines, and ask for a narrower query when there are more than 50 results.
2. View a window, not a file. open shows 100 lines with line numbers and a header stating the file and the position; scroll_up, scroll_down and goto move the window.
3. Edit with a guard. edit replaces a range of lines with new text, shows the result, and runs a linter first: if the change introduces a syntax error it is rejected, and the agent sees the error alongside the code before and after.
4. Reproduce and test through the shell. Ordinary bash commands remain available, so the agent creates a reproduction script, runs it and runs the tests. A command that prints nothing returns an explicit message saying it succeeded with no output.
5. Manage context. Each turn is a thought plus one command. Older observations are collapsed to a single line, with only the last five kept in full, and the prompt includes a demonstration trajectory.
6. Submit. submit turns the repository's current diff into the patch that SWE-bench grades.

The SWE-agent paper distills these choices into four design principles for an agent-computer interface: commands should be simple enough to learn from a short description; important operations such as navigating and editing should take as few actions as possible; feedback should be informative but concise; and guardrails, like the edit linter, should stop errors before they compound.

• F2Pi — fail-to-pass tests: tests from the reference pull request that failed before the fix and pass after it, so passing them shows the issue is fixed.
• P2Pi — pass-to-pass tests: tests that passed both before and after the reference fix; requiring them keeps regressions from counting as solutions.
• D — the evaluation set: the full 2,294 tasks, the 300-task Lite subset or the 500-task Verified subset.
• applies — the patch must apply cleanly to the repository at the base commit.

Every term is binary and there is no partial credit, so a patch that fixes the bug but breaks one unrelated test scores the same as an empty patch. At the extremes, an empty patch resolves nothing, because the fail-to-pass tests still fail, and a patch that deletes the failing functionality fails too, because those same tests check the new behavior. Because D varies, a percentage means nothing without its subset; the same system scores differently on the full set, on Lite and on Verified.

The SWE-bench paper evaluated single-shot, retrieval-augmented baselines on all 2,294 tasks. The SWE-agent paper ran its agent with GPT-4 Turbo and Claude 3 Opus on the full set and on SWE-bench Lite, a 300-task subset of more self-contained bug fixes, and ran its interface ablations on Lite.

Two negative results came from the benchmark paper. Even when handed the files the reference patch edits, the best model resolved under 5 percent, so finding the code is not the only hard part. And giving Claude 2 more BM25-retrieved context made it worse, not better: longer inputs distracted the model from the code that mattered.

The SWE-agent ablations isolate the interface. Holding the model fixed, the purpose-built commands beat a plain shell by 7 points on Lite. Removing the linting guard on edits cost points; showing whole files instead of a 100-line window did worse, and so did a 30-line window; and search that returned raw matches iteratively did worse than summarized search. The same interface also reached 87.7 percent on HumanEvalFix, a short bug-fixing benchmark.

Later work questioned how much of this needs an agent loop. Agentless (Xia and colleagues, 2024) used a fixed localize, repair and validate pipeline with no free-form tool use and matched or beat the open-source agents of its time on Lite at low cost. SWE-bench Verified, a 500-task subset screened by human developers for clear issues and fair tests, became the standard reporting set.

The benchmark's own limitations are substantial and well documented, several by its authors and more by later analyses. Contamination is structural: the issues and their fixes are public and predate the models being evaluated, so exposure cannot be excluded. Tests are an imperfect oracle — a patch that passes them is not necessarily the right fix, and analyses have found instances where the tests are weak enough to admit wrong solutions or where the issue text or comments essentially give the answer away. And the tasks are drawn from a handful of Python repositories, so "real software" means one language and one ecosystem's conventions.

A practical consequence worth flagging: filtered subsets and variants proliferated, and results reported on different subsets with different scaffolds are routinely compared as though they were the same number. When a model is claimed to score some percentage here, the subset and the harness are part of the claim.

How it held up: it became the standard agentic coding benchmark, which is both its success and the source of its problems — a benchmark this influential is optimised against. The interface finding is the durable contribution, and it generalises past coding to every agent on the agents ladder: how much capability you get out of a model depends on the tools you hand it at least as much as on the model, and designing those tools for a reader that cannot see is a distinct skill.

SWE-bench and SWE-agent set the agenda for the coding-agent era. The benchmark is now the headline metric quoted for every serious coding model and agent, and its verified subset is a trusted standard. SWE-agent's ACI insight — design the interface for the model — shaped how agent frameworks are built across the industry, well beyond code.

Its deepest contribution is a reframing of what "coding AI" is measured on. HumanEval asked "can it write code?"; SWE-bench asked "can it engineer?" — navigate, localize, edit, and verify in a real repository, graded by whether the software actually works afterward. That execution-grounded, whole-task evaluation is far closer to the real job, and it turned agent design (interfaces, guardrails, verification loops) into a research frontier alongside the models themselves. When a coding agent reports a SWE-bench score today, it's speaking the language this work created.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Completion, then competition, then a benchmark made of real GitHub issues — the point at which "can it code?" got an answer you could audit.

1. 1Codex2021
2. 2AlphaCode2022
3. 3SWE-bench & SWE-agent2023

All 110 papers, in 19 ladders →