SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
- Publication
- ICLR 2024Conference
- First submitted to arXiv
- arXiv:2310.06770
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{modelevaluationswebench2023,
title = {{SWE-bench: Can Language Models Resolve Real-World GitHub Issues?}},
author = {Carlos E. Jimenez and John Yang and Alexander Wettig and Shunyu Yao and Kexin Pei and Ofir Press and Karthik Narasimhan},
year = {2023},
eprint = {2310.06770},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2310.06770}
}
In 2023, the coding ability of language models was mostly measured with benchmarks like HumanEval: given a function signature and a description, write the function body and run a few unit tests. Every problem stands on its own and can be solved in a few lines of code.
Real software work isn't like that. Fixing a bug can mean finding the relevant code in a project with thousands of files, figuring out how functions in different files call each other, tracking down the one wrong line in a tangle of logic, and then changing it without breaking anything else.
One benchmark after another was being pushed close to a perfect score, and they could no longer measure what the strongest models were actually capable of. Jimenez, Yang and colleagues at Princeton turned to GitHub: open-source projects are full of real problem reports (issues) and the pull requests that actually fixed them, along with the projects' own tests.
Defining the problem
A good benchmark has to be hard enough to stump existing models, while its answers stay easy to verify. Coding problems have the advantage that you can run tests, but the existing coding benchmarks were too easy and too clean.
The authors wanted a benchmark that meets several conditions: the problems come from real work, the input is long and messy, the changes needed may be spread across several files, correctness is judged by the project's own tests, and new problems can keep being added so models haven't already seen the answers during training.
Solving the problem

The task is straightforward. The model gets the text of an issue and the whole codebase, and has to produce a change, expressed as a patch file saying which lines to change. For evaluation, the Unix patch program applies the change, and then the tests relevant to that problem are run; a problem counts as resolved only if the change applies cleanly and all the tests pass. The score for the whole benchmark is the fraction of problems resolved.
From 90,000 pull requests down to 2,294 problems
The authors filter in three stages:
- Pick projects and scrape the data: choose 12 popular open-source Python projects and scrape about 90,000 pull requests in total. Popular projects tend to be better maintained, with clear contribution guidelines and more complete tests. The version of the codebase each pull request applies to is its base commit.
- Filter by attributes: keep only pull requests that were merged, resolved an issue, and modified test files. A change to the test files suggests the author probably added tests to confirm the problem was fixed.
- Filter by execution: first apply only the test changes from the pull request and record the test results; then apply the rest of the code changes and record them again. A pull request is kept only if at least one test goes from failing to passing; those that fail to install or error out at run time are dropped too.
That leaves 2,294 problems.
What the problems look like

| Item | Mean | Max |
|---|---|---|
| Issue text length (words) | 195.1 | 4,477 |
| Files in the codebase (excluding tests) | 3,010 | 5,890 |
| Lines in the codebase (excluding tests) | 438K | 886K |
| Lines edited by the gold patch | 32.8 | 5,888 |
| Files edited by the gold patch | 1.7 | 31 |
| Functions edited by the gold patch | 3 | 36 |
| Tests that fail before the change and pass after | 9.1 | 1,633 |
| Total tests | 120.8 | 9,459 |
Compiled from Table 1 of the paper; the gold patch is the change that was actually merged.
On average, each problem's codebase has 3,000 files and 438,000 lines of code, yet only 1.7 files and 30-odd lines need to change. The model has to find the few lines that really need changing in a vast expanse of code.
There are two kinds of tests. Fail-to-pass (F2P) tests fail before the change and pass after it; they check whether the problem reported in the issue was fixed. Each problem has at least one, and 40% of problems have two or more. Pass-to-pass (P2P) tests must pass both before and after the change; they check whether existing functionality was broken. The median is 51 per problem.
A problem counts as resolved only if both kinds pass. To take a simple example, say an issue reports that 1900 is wrongly treated as a leap year, so the F2P test checks 1900. If the fix only adds "not divisible by 100," 1900 is fixed, but 2000 also becomes a non-leap year, and a P2P test that used to pass catches the mistake.
What this design buys
- Realistic: both the problems and the solutions were actually submitted by users, and they call for the skills of a senior engineer.
- Can keep being updated: the collection pipeline needs almost no human work and can be applied to any Python project. Test with issues created after a model's training cutoff, and you can be sure the answers aren't in its training data.
- Long, messy input: issues average 195 words, and codebases easily run to thousands of files.
- Changes across files: unlike HumanEval, where you change a single function, or fill-in-the-blank problems, nothing here tells you where to make the change.
- Open-ended solutions: the model's change doesn't have to match the gold patch; passing the tests is what counts. Retrieval, long context, agents and other approaches can all be compared on the same standard.
Running all 2,294 problems once eats a lot of time and API fees, so the authors also picked 300 problems to form SWE-bench Lite, leaning toward more self-contained functional bugs and covering 11 of the 12 projects.
SWE-Llama
Among open-source models at the time, only CodeLlama could handle such long contexts, but the off-the-shelf versions weren't good at following instructions to produce a change to a whole codebase, and often output only placeholder text or unrelated code. Using the same pipeline, the authors collected 19,000 issue and pull request pairs from another 37 Python projects. This time a test change wasn't required, and the projects don't overlap with the benchmark at all.
Fine-tuning CodeLlama-Python 7B and 13B on this data gives SWE-Llama. To save memory, only the attention layers are fine-tuned with LoRA, and training sequences longer than 30,000 tokens are excluded, leaving about 10,000 examples actually used. The input is the instructions, the issue and the relevant code files, and the target is the gold patch.
Fitting a codebase into the context
A codebase averaging 438,000 lines can't fit into the contexts of the time in its entirety. The authors choose files in two ways:
- BM25 retrieval: use the issue text as the query to retrieve relevant files, filling up to a limit of 13,000, 27,000 or 50,000 tokens. With code paired with a natural-language query, and both the query and the documents being long, the authors judged dense retrieval a poor fit.
- Oracle retrieval: give the model exactly the files the gold patch touches. This is unrealistic, since an engineer doesn't know at the start which files need changing either, but it's useful for analysis.
At the 27,000-token limit, BM25 finds all of the oracle files for about 40% of problems, but for nearly half of problems it finds none of them.
The models evaluated are ChatGPT-3.5 (16K context), GPT-4 (32K), Claude 2 (100K) and SWE-Llama. The shorter the context, the fewer the problems whose oracle files fit in completely; that's 84.1% of problems for GPT-4, but only 58.1% for ChatGPT-3.5.
Discussion
Almost nothing gets solved
| Model | Resolved | Patch applies | Lite resolved |
|---|---|---|---|
| Claude 3 Opus | 3.79 | 46.56 | 4.33 |
| Claude 2 | 1.97 | 43.07 | 3.00 |
| ChatGPT-3.5 | 0.17 | 26.33 | 0.33 |
| GPT-4-turbo | 1.31 | 26.90 | 2.67 |
| SWE-Llama 7b | 0.70 | 51.74 | 1.33 |
| SWE-Llama 13b | 0.70 | 53.62 | 1.00 |
Compiled from Table 5 of the paper; BM25 retrieval, in %.
The best model resolves under 4%, and Claude 2, the best when the paper came out, under 2%. More than half of the changes aren't even well-formed and can't be applied at all.
Oracle files help a little. Claude 2 rises from under 2% to 4.80%, SWE-Llama 13b reaches 3.97%, and GPT-4 (tested only on a random quarter of the problems) 1.74%.
Difficulty varies a lot between projects

The three models show similar highs and lows across the projects, but the problems they solve overlap little. In the oracle setting, Claude 2 solves 110 problems and SWE-Llama 13b solves 91, yet of the problems SWE-Llama solves, Claude 2 solves only 42%.
Some projects' issues include images, embedded in the text as Markdown image links. 32% of matplotlib problems include images and 10% of seaborn's, against only 2% overall. Problems like these may need a multimodal model, or tools that can handle images.
The longer the context, the worse the performance
Claude 2's performance drops clearly as the total input length grows, and the same goes for the other models. Relaxing the BM25 limit from 13,000 to 50,000 tokens raises the fraction of oracle files found, yet Claude 2's resolve rate falls from 1.96% to 1.22%. When the model sees too much code unrelated to the problem, it can't find the place that actually needs changing.
The authors also ran an "oracle-collapsed" experiment: the model still gets the oracle files, but only the lines the gold patch touches are kept, with 15 lines above and below, and everything else is collapsed. Claude 2 rises from 4.8% to 5.9%, and GPT-4 to 3.40%.
Other observations
- No link to the issue's date: splitting the problems into those from before 2023 and after, most models perform about the same (GPT-4 was tested on only a quarter of the problems, and its after-2023 group is 0). The authors take this to mean that even if models have seen older versions of the codebases, they can hardly cheat by memorizing answers.
- Fine-tuned models are sensitive to a change of context: SWE-Llama was trained on oracle files and learned that "every file I'm given needs changing"; switch to the pile of unrelated files BM25 provides and it performs poorly.
- Writing a patch is easier than rewriting the whole file: models mostly see complete code files and rarely see patches, so the authors tried having the model output the entire modified file directly. That turned out even worse, with Claude 2 under oracle retrieval dropping from 4.8% to 2.2%.
- Models change less: counting the total lines of the patch file (context lines included), changes that applied successfully average only 17 to 30 lines, while the gold patches for the same problems are 34 to 44 lines, and all gold patches average 74.5 lines. Models almost always change just one file.
An example that didn't get fixed

The issue says that in Sphinx's napoleon extension, when napoleon_use_param is set to True, the "Other Parameters" section is formatted incorrectly. The issue is very detailed and even includes a snippet of the code that might be at fault.
In the oracle setting, the input totals 1,558 lines and 20,882 tokens. SWE-Llama changed the right function, but made it treat napoleon_use_param as True whatever the setting; the gold patch checks the setting first, following the pattern of the neighboring _parse_parameters_section. One test happens to compare the output with the setting at False, and it caught the mistake right away.
The authors looked at 11 outputs from SWE-Llama and Claude 2 and identified a few tendencies. The code the models write is primitive; they rarely use existing libraries or utilities already in the codebase. Their solutions are "greedy," fixing only the case described in the issue and paying little attention to code style or project conventions, such as whether to use relative or absolute imports. The human-written gold patches often make structural improvements along the way, and deal with related problems that might come up as well.
Limitations
- Python only: the authors hope to apply the same collection pipeline to other languages in the future.
- Baselines only: the paper uses the simplest approach, retrieving a few files and generating the change in one go. The authors encourage trying agents, tool use and other approaches next.
- What tests can't catch: code that passes the tests can still be incomplete, inefficient or hard to read, and model-written changes often fall short of human-written ones in these respects.
What came next
The agent approach the authors suggested in their limitations was soon built by the same team. SWE-agent, from May 2024, designed a dedicated interface for the language model to operate through, letting it browse the whole codebase, open files, edit and run tests, and it resolved 12.5% of the problems on the full SWE-bench.
On August 13, 2024, OpenAI released SWE-bench Verified. They had 93 experienced Python developers check 1,699 randomly drawn problems, removed those with unclear descriptions or with tests that were unfair to correct solutions, and kept 500. With the best open-source scaffold of the time, GPT-4o resolved 33.2%.
After that, SWE-bench Verified became the coding metric most often listed when companies released new models. In October 2024, the upgraded Claude 3.5 Sonnet went from 33.4% to 49.0%; in May 2025, Claude Opus 4 scored 72.5%. All of these are on the human-filtered Verified set and can't be compared directly with numbers on the original 2,294 problems, but going from under 2% to 70% took just a year and a half.
Conclusion
SWE-bench filters 2,294 real issues out of 90,000 pull requests from 12 popular Python projects and uses each project's own tests to judge whether they were fixed. When the paper came out, the best model, Claude 2, resolved under 2%, and only 4.8% even when given the right files.
The hard part isn't just writing code. The model has to find the 30 lines to change in a 438,000-line codebase, and the more code it's given, the worse it gets at finding them; and the change must not break anything else, with a median of 51 P2P tests per problem watching from the sidelines.
The benchmark was realistic enough that it wasn't pushed to a perfect score quickly. A year and a half later, the leaderboard numbers had climbed from single digits to 70%, thanks not only to stronger models but also to models learning to work like engineers, digging around the codebase, running tests, reading the error messages and trying again.