Evaluating Large Language Models Trained on Code
- Publication
- Publication venue not yet verified
- First submitted to arXiv
- arXiv:2107.03374
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{modelevaluationcodex2021,
title = {{Evaluating Large Language Models Trained on Code}},
author = {Mark Chen and Jerry Tworek and Heewoo Jun and Qiming Yuan and Henrique Ponde de Oliveira Pinto and Jared Kaplan and Harri Edwards and Yuri Burda and Nicholas Joseph and Greg Brockman and Alex Ray and Raul Puri and Gretchen Krueger and Michael Petrov and Heidy Khlaaf and Girish Sastry and Pamela Mishkin and Brooke Chan and Scott Gray and Nick Ryder and Mikhail Pavlov and Alethea Power and Lukasz Kaiser and Mohammad Bavarian and Clemens Winter and Philippe Tillet and Felipe Petroski Such and Dave Cummings and Matthias Plappert and Fotios Chantzis and Elizabeth Barnes and Ariel Herbert-Voss and William Hebgen Guss and Alex Nichol and Alex Paino and Nikolas Tezak and Jie Tang and Igor Babuschkin and Suchir Balaji and Shantanu Jain and William Saunders and Christopher Hesse and Andrew N. Carr and Jan Leike and Josh Achiam and Vedant Misra and Evan Morikawa and Alec Radford and Matthew Knight and Miles Brundage and Mira Murati and Katie Mayer and Peter Welinder and Bob McGrew and Dario Amodei and Sam McCandlish and Ilya Sutskever and Wojciech Zaremba},
year = {2021},
eprint = {2107.03374},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2107.03374}
}
At the end of June 2021, GitHub launched a technical preview of Copilot. Write a function name and a comment in your editor, and it fills in the body of the function. A week later, OpenAI published this paper introducing the model behind it, Codex; the paper's opening paragraph states that Copilot runs on a distinct production version of Codex.
The idea came from GPT-3. The authors had found earlier that after reading a Python docstring, GPT-3 could write some simple programs. The capability was rudimentary, but GPT-3 had never been trained specifically for code. So they took GPT and kept training it on code from GitHub, to see how far it could get.
Where this paper puts most of its effort, though, is the question of "how do you tell whether the model wrote good code?" The HumanEval benchmark and the pass@k metric it left behind became the standard exam and scoring method for code models, and they have outlived Codex itself.
Defining the problem
Before this, models that generated code were mostly scored by matching. You compare the model's program with a reference solution, and if it's an exact match or the BLEU score is high enough, it counts as good.
The trouble is that there are countless ways to write a program. return [i + 1 for i in l] and a for loop that appends one element at a time do exactly the same thing, yet look very different on the page; conversely, two programs that differ by a single character can be one right and one wrong. Later in the paper the authors actually check this, and Codex's incorrect programs often have higher BLEU scores than its correct ones.
The authors argue for looking at functional correctness instead, where a program counts as correct only if it passes the unit tests. This is also how engineers normally judge code; test-driven development means writing the requirements as tests first and the implementation second.
To score this way, three things have to be dealt with first:
- The problems can't appear in the training data. Codex has read most of GitHub, and on GitHub there are more than ten public repos of Codeforces solutions alone.
- How to compute the score fairly when many answers are sampled for each problem.
- The code the model writes can't be trusted, so it needs somewhere safe to run.
The third is an engineering problem. The authors use the gVisor container runtime to isolate the host from the containers, plus eBPF firewall rules that block every network connection except those needed to control the experiment. The first two are HumanEval and pass@k.
HumanEval
The authors hand-wrote 164 problems, each with a function signature, a docstring, a reference solution and an average of 7.7 unit tests. The problems cover language comprehension, reasoning, algorithms and simple math, and some are about as hard as simple software interview questions.

The white background is the prompt given to the model, and the yellow background is the answer the model wrote. The three problems go from easy to hard, and the probability that Codex-12B gets each one right in a single sample is 0.9, 0.17 and 0.005. The hardest one requires first understanding how encode_cyclic rotates every group of three characters, and then writing the reverse operation.
The prompt contains only the header, the function signature and the docstring. The model keeps on writing, so generation stops when it hits \nclass, \ndef, \n#, \nif or \nprint; otherwise it would go on to write the next function or some test code.
pass@k
You can let the model take many shots at one problem. pass@k means sampling k answers; if any one of them passes the tests, the problem counts as solved, and the result is averaged over all problems.
The most direct way to compute it is to actually draw k samples per problem, but that has high variance. The authors instead draw samples per problem (the paper uses and ), count the number that pass, and estimate:
The fraction is the probability that samples drawn at random from the are all wrong; one minus that is the probability that at least one of the is right.
The binomial coefficients get enormous if you compute them directly, so the paper provides a numerically stable version that expands the product term by term. Let's use it on one problem:
import numpy as np
from math import comb
def pass_at_k(n, c, k):
"""The paper's Figure 3 implementation: estimate pass@k when c of n samples pass"""
if n - c < k:
return 1.0
return 1.0 - np.prod(1.0 - k / np.arange(n - c + 1, n + 1))
n, c = 200, 10 # 200 samples for one problem, 10 pass the tests
for k in (1, 10, 100):
exact = 1 - comb(n - c, k) / comb(n, k) # computed directly with binomial coefficients
naive = 1 - (1 - c / n) ** k # naively extrapolated from pass@1
print(f"k={k:3d} unbiased={pass_at_k(n, c, k):.4f} comb={exact:.4f} naive={naive:.4f}")
# k= 1 unbiased=0.0500 comb=0.0500 naive=0.0500
# k= 10 unbiased=0.4085 comb=0.4085 naive=0.4013
# k=100 unbiased=0.9992 comb=0.9992 naive=0.9941
The stable version gives the same result as computing the binomial coefficients directly. The intuitive comes out slightly low, and the paper proves in the appendix that it always underestimates. It amounts to drawing times from the samples "with replacement," whereas the unbiased estimate draws without replacement.
There's an even easier trap to fall into. pass@k has to be estimated for each problem separately and then averaged; you can't extrapolate it from the pass@1 of the whole dataset. Take three problems of very different difficulty, with single-sample pass rates of 0.9, 0.17 and 0.005, which are the three HumanEval examples above. Suppose each problem is sampled 200 times and passes 180, 34 and 1 times respectively.
The average pass@1 of the three problems is 0.358, and extrapolating pass@100 from it gives 1.0. In reality, the hardest problem was solved only once in 200 tries, so if you pick 100 at random, there's only a fifty-fifty chance of getting that one, and the average over the three problems is 0.833. When problems vary widely in difficulty, the scores of the easy ones paper over the hard ones.
Solving the problem
Codex: GPT, trained further on GitHub
The training data is Python files collected in May 2020 from 54 million public repos on GitHub: 179GB of unique files under 1MB. The authors then filtered out files that were likely auto-generated, had an average line length over 100, had a longest line over 1,000, or had too low a proportion of alphanumeric characters, leaving 159GB.
The models are fine-tuned starting from weights in the GPT-3 family, up to 12 billion parameters. The authors originally expected that starting from GPT would help, since it already has strong natural-language representations; in the end the results were about the same as training from scratch, and the authors guess this is because the fine-tuning dataset is so large. Starting from GPT does converge faster, though, so all later experiments do so. Training covers 100 billion tokens in total, with learning rates carried over from the corresponding GPT models.
The tokenizer is also carried over from GPT-3, but the distribution of words in code is very different from natural language, and the biggest waste is the whitespace used for indentation. The authors added an extra set of tokens representing runs of whitespace of different lengths, so the same code uses about 30% fewer tokens.
Sampling uses nucleus sampling with top p = 0.95, while the temperature has to be tuned to k. For the 679M model, the best temperature for pass@1 is 0.2, and for pass@100 it is 0.8. When k is large, the more the samples differ from each other the better, since all it takes is one right answer.

This figure puts the whole paper's results together. The horizontal axis is the number of non-embedding parameters. The blue line is GPT-3's pass@1, flat at 0 the whole way; the orange line is Codex, and the green line is Codex-S from the next section. Every sample in the figure uses temperature 0.8, so these two pass@1 lines sit below the best scores reported in the paper; with each model's own best temperature, the 12B Codex solves 28.8% in one sample and Codex-S 37.7%. The red and purple lines both have Codex-S draw 100 samples per problem, and differ in how one is picked. The red line picks the one with the highest mean log-probability, 44.5%; the purple line lets an oracle that knows the test results pick, so a problem counts if any of the 100 passes, 77.5%.
Compared with other models:
| Model | pass@1 | pass@10 | pass@100 |
|---|---|---|---|
| TabNine | 2.58% | 4.35% | 7.59% |
| GPT-Neo 2.7B | 6.41% | 11.27% | 21.37% |
| GPT-J 6B | 11.62% | 15.74% | 27.74% |
| Codex-300M | 13.17% | 20.37% | 36.27% |
| Codex-2.5B | 21.36% | 35.42% | 59.50% |
| Codex-12B | 28.81% | 46.81% | 72.31% |
Compiled from Table 1 of the paper.
GPT-Neo and GPT-J are open-source models that EleutherAI trained on The Pile, about 8% of which is GitHub code; GPT models of the same sizes score almost 0. GPT-J 6B performs about the same as Codex-300M, with 20 times the parameters. TabNine was the leading code autocomplete tool of the time, and its largest free model is roughly on par with Codex-12M.
Codex-S: making the training data look more like the exam
The Python on GitHub isn't just standalone functions. There are also classes, config files, scripts, even files used to store data, all a long way from "read a docstring, write a function." The authors collected a separate batch of correctly implemented standalone functions and ran another round of supervised fine-tuning on them, producing Codex-S.
The data has two sources:
- Competitive programming and interview practice websites: the problem description serves as the docstring, together with a function signature and a solution. The full tests are usually not public, so the authors built tests from the examples in the problem statements, or deliberately submitted incorrect solutions to dig more test cases out of the feedback. 10,000 problems in total.
- Open-source projects with continuous integration (CI): the authors used Python's
sys.setprofileto trace the inputs and outputs of every function during integration tests, then turned them into unit tests. They built environments and ran the tests following travis and tox config files, and also used packages from PyPI. Most of these functions are parts of command-line tools, so what they test is implementing functionality as the docstring describes, not algorithms. About 40,000 problems in total.
Automatically collected problems vary in quality. Some have incomplete descriptions, and the tests can mark even a correct answer as wrong; some are stateful, giving different results on each run. The authors had Codex-12B sample 100 answers for each problem, and any problem where not a single one passed was treated as broken or too hard and dropped; this check was rerun several times to also screen out problems with unstable results.
During training, the loss is computed only on the reference solution, with the prompt masked out. The learning rate is a tenth of Codex's, and training runs until the validation loss stops falling, using fewer than 10 billion tokens.
Across model sizes, Codex-S's pass@1 is on average 6.5 percentage points higher than Codex's, and its pass@100 15.1 percentage points higher.
Picking without tests
pass@k assumes someone knows which answer is right. In actual use as autocomplete, the user sees only one suggestion and has no unit tests at hand.
The authors compared several ways of picking. Picking the sample with the highest "mean log-probability per token" beats picking at random; picking the one with the highest "total log-probability" instead can come out slightly worse than random.
Another idea is to work backwards. The authors trained Codex-D, which reads code and writes docstrings, then used it to compute the probability of "writing the original docstring back from this code," and picked the highest. This method beats random picking but loses to mean log-probability, and it overfits quickly.
Codex-D is interesting in its own right. The authors graded its docstrings by hand, counting one as correct only if it uniquely and accurately describes the code; its pass@1 is 20.3%, against 32.2% for Codex-S writing code under the same conditions. One possible reason the authors mention is that developers don't put much care into docstrings to begin with; the model even learned to write things like "I just found this function online" and "This test is not correctly written and it's not my solution."
Discussion
The longer the docstring, the more it goes wrong
The authors took 13 simple string operations as building blocks, such as "convert the string to lowercase" and "remove every third character," chained them into longer and longer docstrings, and checked whether Codex-12B could write the code.

Each extra building block cuts the pass rate to roughly a half or a third of what it was, and from five blocks on it is 0. Human engineers don't work like this. Someone who can write two steps can write seven; it just takes longer.
Another problem is binding operations to the right variables. The paper gives this example, asking to "add 3 to y, subtract 4 from both x and w, and return the product of the four numbers":
def do_work(x, y, z, w):
""" Add 3 to y, then subtract 4
from both x and w. Return the
product of the four numbers. """
t = y + 3
u = x - 4
v = z * w
return v
Codex-12B didn't subtract 4 from w, and in the end returned the product of only two numbers.
It knows better, and copies the bugs anyway
This is the most farsighted experiment in the paper. The authors put a few examples in the prompt first, one set with correct code and another with subtle bugs, such as off-by-one boundary errors or a single mistyped character, and then asked the model to solve the next problem.

The green line saw correct examples, the orange line saw buggy examples, and the gray line saw no examples. When the model is small, the three lines are about the same; the larger the model, the further the orange line falls behind, at about 0.22 against 0.29 at 12B. The authors also tried explicitly asking for "correct code" in the prompt, which helped only a little.
Codex's training objective is to imitate its training data, so when the context has bugs, it writes buggy code too. It is capable of getting it right, yet doesn't do what the user really wants. The paper calls this misalignment and keeps it separate from a lack of capability. Being unable to do something is a capability problem; being able to and not doing it is misalignment. The troubling part is that this gap widens as the model grows. Among the directions for improvement the paper lists in its appendix is reinforcement learning from human feedback.
Hazard analysis
Section 7 of the paper is a hazard analysis, and it runs about as long as the experiments before it.
- Over-reliance: Codex can produce code that looks right but doesn't actually do what the user asked, and beginners are especially likely to fall for it. The more capable the model, the more easily people let their guard down.
- Security: the model may write code with vulnerabilities. After testing, the authors concluded that with its capabilities at the time, Codex did not noticeably lower the barrier to writing malware.
- Economics: engineers don't spend their whole day writing code, so the short-term impact is limited; but Codex imports different packages at different rates, so the authors of the packages it recommends may benefit.
- Legal: generated code identical to the training data occurs in fewer than 0.1% of cases, and mostly consists of idioms that recur throughout the language.
What came after
HumanEval quickly became the standard exam for code models, and nearly every LLM technical report lists its pass@1. But with 164 problems and an average of 7.7 tests, the holes start to show with use. In 2023, EvalPlus automatically added a large number of tests to each problem, about 80 times as many as before in total; models' pass@k dropped by as much as 19.3% to 28.9%, and the rankings shifted along with it. Quite a few programs that had "passed" had simply slipped past the tests.
Scores were also pushed to the ceiling quickly. GPT-4 scored 67% in 2023, and by 2024, frontier models' pass@1 was above ninety percent. Everyone had to move to evaluations closer to real work, such as SWE-bench, which asks models to fix GitHub issues.
As for the Codex model itself, OpenAI retired the Codex series from its API back in March 2023. The name didn't go to waste, though. In April 2025, OpenAI launched Codex CLI, which writes code in the terminal, and in May a cloud-based coding agent, both called Codex.
Conclusion
Codex showed that once GPT is trained further on a large amount of code, it can write working functions from docstrings. The 12B model solves 28.8% of HumanEval in a single sample; with 100 samples per problem, more than seventy percent of the problems get at least one correct answer.
What has lasted longest from this paper are the scoring rules. Whether code is right is decided by tests, not by literal comparison with a reference answer; pass@k is estimated per problem and then averaged. Code models over the following years have been compared by almost exactly these rules.
So if your docstring is sloppy, don't expect the model to take its code much more seriously. Among the docstrings it read on GitHub, there's even "I just found this function online."