SWE-bench & SWE-agent,
explained.

Benchmarks like HumanEval measure whether a model can write a small, self-contained function from a clear spec. Useful — but a poor proxy for what software engineers actually do. Real work means operating inside a large, unfamiliar codebase: interpreting a vague bug report, locating the relevant code among thousands of files, making a correct, minimal change, and confirming with the test suite that the bug is gone and nothing else broke. None of that is captured by "write this function."

SWE-bench closed the gap by building its tasks from reality. Each instance is a genuine GitHub issue that was actually resolved by a pull request in a popular open-source Python project, packaged with the repository as it was before the fix and the tests that validate the fix. The model gets the issue and the repo; it must produce a patch. It's the difference between a coding quiz and a real ticket in a real backlog.

How do you score a code change objectively? By running the tests — but SWE-bench is careful about which tests. It splits them into two sets and requires both to come out right for an issue to count as resolved:

The FAIL_TO_PASS tests are the ones that failed before the fix — they prove the bug is actually fixed, not merely that the patch runs. The PASS_TO_PASS tests already passed — requiring they still pass is a regression check, so a "fix" that quietly breaks something else does not count. And the patch must apply cleanly to begin with. All three, or it's not resolved. This is a strict, execution-based grade with no partial credit — exactly the honesty you want for real engineering. The runnable version below encodes this rule and the regression trap.

SWE-bench sets the task; SWE-agent asks how a model can actually do it. A language model can't magically edit a repo — it needs a way to look around and act. The naive approach is to hand it a normal shell. The paper's key finding is that this is a mistake: interfaces built for humans serve models poorly. Models get lost in verbose output, mangle multi-line edits, and waste turns on malformed commands.

The fix is a purpose-built Agent-Computer Interface (ACI) — a small set of commands designed for how a language model reads and writes:

The result was measurable: a well-designed ACI raised the resolve rate substantially over a plain shell, holding the model fixed. The lesson generalized far beyond this paper — how you let an agent act, not just how smart the model is, is a first-class design variable. Guardrails (like the edit linter) that catch mistakes before they compound turned out to matter enormously.

With the ACI in place, SWE-agent runs the familiar agent loop — the same reason-then-act cycle, specialized for coding. The model observes, decides on a command, runs it, reads the result, and repeats until it submits a patch:

It might search for the error message, open the offending file, scroll to the function, edit it (the linter catching a botched edit before it lands), run the tests, see two still failing, adjust, and finally submit. Every step is a turn where the model reasons about what it sees and picks the next action. The whole trajectory is bounded — limited turns and cost — so the agent must localize and fix efficiently. When it submits, SWE-bench applies the strict three-part grade from section 02.

SWE-bench's first scores were humbling — resolving only a low single-digit percentage of issues — and that was the point:

The benchmark did what a good benchmark does: it made a fuzzy goal ("can AI do software engineering?") into a concrete, execution-graded number that everyone could chase. And it repeatedly showed that closing the gap took two things advancing together — stronger models and better agent scaffolding around them.

SWE-bench and SWE-agent set the agenda for the coding-agent era. The benchmark is now the headline metric quoted for every serious coding model and agent, and its verified subset is a trusted standard. SWE-agent's ACI insight — design the interface for the model — shaped how agent frameworks are built across the industry, well beyond code.

Its deepest contribution is a reframing of what "coding AI" is measured on. HumanEval asked "can it write code?"; SWE-bench asked "can it engineer?" — navigate, localize, edit, and verify in a real repository, graded by whether the software actually works afterward. That execution-grounded, whole-task evaluation is far closer to the real job, and it turned agent design (interfaces, guardrails, verification loops) into a research frontier alongside the models themselves. When a coding agent reports a SWE-bench score today, it's speaking the language this work created.