Back to paper library
arXiv 2023ICLR 2024Model evaluationPaper notes

SWE-bench

Live fire

Original paper

SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

Paper authors

Carlos E. Jimenez · John Yang · Alexander Wettig …

Show all 7 authors

Carlos E. Jimenez · John Yang · Alexander Wettig · Shunyu Yao · Kexin Pei · Ofir Press · Karthik Narasimhan

Publication
ICLR 2024Conference
First submitted to arXiv
arXiv:2310.06770
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{modelevaluationswebench2023,
  title = {{SWE-bench: Can Language Models Resolve Real-World GitHub Issues?}},
  author = {Carlos E. Jimenez and John Yang and Alexander Wettig and Shunyu Yao and Kexin Pei and Ofir Press and Karthik Narasimhan},
  year = {2023},
  eprint = {2310.06770},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.06770}
}
On this page

In 2023, the coding ability of language models was mostly measured with benchmarks like HumanEval: given a function signature and a description, write the function body and run a few unit tests. Every problem stands on its own and can be solved in a few lines of code.

Real software work isn't like that. Fixing a bug can mean finding the relevant code in a project with thousands of files, figuring out how functions in different files call each other, tracking down the one wrong line in a tangle of logic, and then changing it without breaking anything else.

One benchmark after another was being pushed close to a perfect score, and they could no longer measure what the strongest models were actually capable of. Jimenez, Yang and colleagues at Princeton turned to GitHub: open-source projects are full of real problem reports (issues) and the pull requests that actually fixed them, along with the projects' own tests.

Defining the problem

A good benchmark has to be hard enough to stump existing models, while its answers stay easy to verify. Coding problems have the advantage that you can run tests, but the existing coding benchmarks were too easy and too clean.

The authors wanted a benchmark that meets several conditions: the problems come from real work, the input is long and messy, the changes needed may be spread across several files, correctness is judged by the project's own tests, and new problems can keep being added so models haven't already seen the answers during training.

Solving the problem

Original Figure 1. The SWE-bench task. On the left is the input: a GitHub issue (for example, a scikit-learn report of data leakage in GBDT during warm start) and a snapshot of the whole codebase (sklearn/, examples/, README.rst, setup.py and so on). In the middle is the change generated by the language model, for example adding 20 lines and deleting 12 in gradient_boosting.py and helper.py. On the right are the unit tests: some tests fail before the change and pass after it (such as join_struct_col, vstack_struct_col and dstack_struct_col), while others must pass both before and after (such as matrix_transform and euclidean_diff).
Original Figure 1, Jimenez et al., from PDF page 1, CC BY 4.0.

The task is straightforward. The model gets the text of an issue and the whole codebase, and has to produce a change, expressed as a patch file saying which lines to change. For evaluation, the Unix patch program applies the change, and then the tests relevant to that problem are run; a problem counts as resolved only if the change applies cleanly and all the tests pass. The score for the whole benchmark is the fraction of problems resolved.

From 90,000 pull requests down to 2,294 problems

The authors filter in three stages:

  1. Pick projects and scrape the data: choose 12 popular open-source Python projects and scrape about 90,000 pull requests in total. Popular projects tend to be better maintained, with clear contribution guidelines and more complete tests. The version of the codebase each pull request applies to is its base commit.
  2. Filter by attributes: keep only pull requests that were merged, resolved an issue, and modified test files. A change to the test files suggests the author probably added tests to confirm the problem was fixed.
  3. Filter by execution: first apply only the test changes from the pull request and record the test results; then apply the rest of the code changes and record them again. A pull request is kept only if at least one test goes from failing to passing; those that fail to install or error out at run time are dropped too.

That leaves 2,294 problems.

What the problems look like

Original Figure 3. A pie chart of how SWE-bench's 2,294 problems are distributed across the 12 projects. django has the most at 850, followed by sympy 386, scikit-learn 229, sphinx 187, matplotlib 184, pytest 119, xarray 110, astropy 95, pylint 57, requests 44, seaborn 22 and flask 11.
Original Figure 3, Jimenez et al., from PDF page 4, CC BY 4.0.
ItemMeanMax
Issue text length (words)195.14,477
Files in the codebase (excluding tests)3,0105,890
Lines in the codebase (excluding tests)438K886K
Lines edited by the gold patch32.85,888
Files edited by the gold patch1.731
Functions edited by the gold patch336
Tests that fail before the change and pass after9.11,633
Total tests120.89,459

Compiled from Table 1 of the paper; the gold patch is the change that was actually merged.

On average, each problem's codebase has 3,000 files and 438,000 lines of code, yet only 1.7 files and 30-odd lines need to change. The model has to find the few lines that really need changing in a vast expanse of code.

There are two kinds of tests. Fail-to-pass (F2P) tests fail before the change and pass after it; they check whether the problem reported in the issue was fixed. Each problem has at least one, and 40% of problems have two or more. Pass-to-pass (P2P) tests must pass both before and after the change; they check whether existing functionality was broken. The median is 51 per problem.

A problem counts as resolved only if both kinds pass. To take a simple example, say an issue reports that 1900 is wrongly treated as a leap year, so the F2P test checks 1900. If the fix only adds "not divisible by 100," 1900 is fixed, but 2000 also becomes a non-leap year, and a P2P test that used to pass catches the mistake.

What this design buys

  • Realistic: both the problems and the solutions were actually submitted by users, and they call for the skills of a senior engineer.
  • Can keep being updated: the collection pipeline needs almost no human work and can be applied to any Python project. Test with issues created after a model's training cutoff, and you can be sure the answers aren't in its training data.
  • Long, messy input: issues average 195 words, and codebases easily run to thousands of files.
  • Changes across files: unlike HumanEval, where you change a single function, or fill-in-the-blank problems, nothing here tells you where to make the change.
  • Open-ended solutions: the model's change doesn't have to match the gold patch; passing the tests is what counts. Retrieval, long context, agents and other approaches can all be compared on the same standard.

Running all 2,294 problems once eats a lot of time and API fees, so the authors also picked 300 problems to form SWE-bench Lite, leaning toward more self-contained functional bugs and covering 11 of the 12 projects.

SWE-Llama

Among open-source models at the time, only CodeLlama could handle such long contexts, but the off-the-shelf versions weren't good at following instructions to produce a change to a whole codebase, and often output only placeholder text or unrelated code. Using the same pipeline, the authors collected 19,000 issue and pull request pairs from another 37 Python projects. This time a test change wasn't required, and the projects don't overlap with the benchmark at all.

Fine-tuning CodeLlama-Python 7B and 13B on this data gives SWE-Llama. To save memory, only the attention layers are fine-tuned with LoRA, and training sequences longer than 30,000 tokens are excluded, leaving about 10,000 examples actually used. The input is the instructions, the issue and the relevant code files, and the target is the gold patch.

Fitting a codebase into the context

A codebase averaging 438,000 lines can't fit into the contexts of the time in its entirety. The authors choose files in two ways:

  • BM25 retrieval: use the issue text as the query to retrieve relevant files, filling up to a limit of 13,000, 27,000 or 50,000 tokens. With code paired with a natural-language query, and both the query and the documents being long, the authors judged dense retrieval a poor fit.
  • Oracle retrieval: give the model exactly the files the gold patch touches. This is unrealistic, since an engineer doesn't know at the start which files need changing either, but it's useful for analysis.

At the 27,000-token limit, BM25 finds all of the oracle files for about 40% of problems, but for nearly half of problems it finds none of them.

The models evaluated are ChatGPT-3.5 (16K context), GPT-4 (32K), Claude 2 (100K) and SWE-Llama. The shorter the context, the fewer the problems whose oracle files fit in completely; that's 84.1% of problems for GPT-4, but only 58.1% for ChatGPT-3.5.

Discussion

Almost nothing gets solved

ModelResolvedPatch appliesLite resolved
Claude 3 Opus3.7946.564.33
Claude 21.9743.073.00
ChatGPT-3.50.1726.330.33
GPT-4-turbo1.3126.902.67
SWE-Llama 7b0.7051.741.33
SWE-Llama 13b0.7053.621.00

Compiled from Table 5 of the paper; BM25 retrieval, in %.

The best model resolves under 4%, and Claude 2, the best when the paper came out, under 2%. More than half of the changes aren't even well-formed and can't be applied at all.

Oracle files help a little. Claude 2 rises from under 2% to 4.80%, SWE-Llama 13b reaches 3.97%, and GPT-4 (tested only on a random quarter of the problems) 1.74%.

Difficulty varies a lot between projects

Original Figure 4. A bar chart of resolve rates for ChatGPT-3.5, Claude 2 and SWE-Llama 13b across the 12 projects, with oracle retrieval. The three models follow similar ups and downs; requests is the highest, at about 15% for Claude 2 and about 13% for SWE-Llama 13b; on flask, only SWE-Llama 13b solves anything, about 9%; on seaborn all three models are at 0; ChatGPT-3.5 is near 0 on most projects.
Original Figure 4, Jimenez et al., from PDF page 6, CC BY 4.0.

The three models show similar highs and lows across the projects, but the problems they solve overlap little. In the oracle setting, Claude 2 solves 110 problems and SWE-Llama 13b solves 91, yet of the problems SWE-Llama solves, Claude 2 solves only 42%.

Some projects' issues include images, embedded in the text as Markdown image links. 32% of matplotlib problems include images and 10% of seaborn's, against only 2% overall. Problems like these may need a multimodal model, or tools that can handle images.

The longer the context, the worse the performance

Claude 2's performance drops clearly as the total input length grows, and the same goes for the other models. Relaxing the BM25 limit from 13,000 to 50,000 tokens raises the fraction of oracle files found, yet Claude 2's resolve rate falls from 1.96% to 1.22%. When the model sees too much code unrelated to the problem, it can't find the place that actually needs changing.

The authors also ran an "oracle-collapsed" experiment: the model still gets the oracle files, but only the lines the gold patch touches are kept, with 15 lines above and below, and everything else is collapsed. Claude 2 rises from 4.8% to 5.9%, and GPT-4 to 3.40%.

Other observations

  • No link to the issue's date: splitting the problems into those from before 2023 and after, most models perform about the same (GPT-4 was tested on only a quarter of the problems, and its after-2023 group is 0). The authors take this to mean that even if models have seen older versions of the codebases, they can hardly cheat by memorizing answers.
  • Fine-tuned models are sensitive to a change of context: SWE-Llama was trained on oracle files and learned that "every file I'm given needs changing"; switch to the pile of unrelated files BM25 provides and it performs poorly.
  • Writing a patch is easier than rewriting the whole file: models mostly see complete code files and rarely see patches, so the authors tried having the model output the entire modified file directly. That turned out even worse, with Claude 2 under oracle retrieval dropping from 4.8% to 2.2%.
  • Models change less: counting the total lines of the patch file (context lines included), changes that applied successfully average only 17 to 30 lines, while the gold patches for the same problems are 34 to 44 lines, and all gold patches average 74.5 lines. Models almost always change just one file.

An example that didn't get fixed

Original Figure 6. Sphinx problem sphinx-8713. On the left is the model's input: the instructions, the issue (napoleon_use_param should also affect the output of the "Other Parameters" section) and the relevant code. Top right is the gold patch: inside _parse_other_parameters_section it first checks self._config.napoleon_use_param, and if it is set, uses _format_docutils_params instead; otherwise it keeps the original _format_fields. Middle right is SWE-Llama's change: in the same function, it switches to _format_docutils_params unconditionally. Bottom right are the test results: 45 passed and 2 failed, one of them test_parameters_with_class_reference.
Original Figure 6, Jimenez et al., from PDF page 8, CC BY 4.0.

The issue says that in Sphinx's napoleon extension, when napoleon_use_param is set to True, the "Other Parameters" section is formatted incorrectly. The issue is very detailed and even includes a snippet of the code that might be at fault.

In the oracle setting, the input totals 1,558 lines and 20,882 tokens. SWE-Llama changed the right function, but made it treat napoleon_use_param as True whatever the setting; the gold patch checks the setting first, following the pattern of the neighboring _parse_parameters_section. One test happens to compare the output with the setting at False, and it caught the mistake right away.

The authors looked at 11 outputs from SWE-Llama and Claude 2 and identified a few tendencies. The code the models write is primitive; they rarely use existing libraries or utilities already in the codebase. Their solutions are "greedy," fixing only the case described in the issue and paying little attention to code style or project conventions, such as whether to use relative or absolute imports. The human-written gold patches often make structural improvements along the way, and deal with related problems that might come up as well.

Limitations

  • Python only: the authors hope to apply the same collection pipeline to other languages in the future.
  • Baselines only: the paper uses the simplest approach, retrieving a few files and generating the change in one go. The authors encourage trying agents, tool use and other approaches next.
  • What tests can't catch: code that passes the tests can still be incomplete, inefficient or hard to read, and model-written changes often fall short of human-written ones in these respects.

What came next

The agent approach the authors suggested in their limitations was soon built by the same team. SWE-agent, from May 2024, designed a dedicated interface for the language model to operate through, letting it browse the whole codebase, open files, edit and run tests, and it resolved 12.5% of the problems on the full SWE-bench.

On August 13, 2024, OpenAI released SWE-bench Verified. They had 93 experienced Python developers check 1,699 randomly drawn problems, removed those with unclear descriptions or with tests that were unfair to correct solutions, and kept 500. With the best open-source scaffold of the time, GPT-4o resolved 33.2%.

After that, SWE-bench Verified became the coding metric most often listed when companies released new models. In October 2024, the upgraded Claude 3.5 Sonnet went from 33.4% to 49.0%; in May 2025, Claude Opus 4 scored 72.5%. All of these are on the human-filtered Verified set and can't be compared directly with numbers on the original 2,294 problems, but going from under 2% to 70% took just a year and a half.

Conclusion

SWE-bench filters 2,294 real issues out of 90,000 pull requests from 12 popular Python projects and uses each project's own tests to judge whether they were fixed. When the paper came out, the best model, Claude 2, resolved under 2%, and only 4.8% even when given the right files.

The hard part isn't just writing code. The model has to find the 30 lines to change in a 438,000-line codebase, and the more code it's given, the worse it gets at finding them; and the change must not break anything else, with a median of 51 P2P tests per problem watching from the sidelines.

The benchmark was realistic enough that it wasn't pushed to a perfect score quickly. A year and a half later, the leaderboard numbers had climbed from single digits to 70%, thanks not only to stronger models but also to models learning to work like engineers, digging around the codebase, running tests, reading the error messages and trying again.