LLaMA: Open and Efficient Foundation Language Models
- Publication
- Publication venue not yet verified
- First submitted to arXiv
- arXiv:2302.13971
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{transformersllama2023,
title = {{LLaMA: Open and Efficient Foundation Language Models}},
author = {Hugo Touvron and Thibaut Lavril and Gautier Izacard and Xavier Martinet and Marie-Anne Lachaux and Timothée Lacroix and Baptiste Rozière and Naman Goyal and Eric Hambro and Faisal Azhar and Aurelien Rodriguez and Armand Joulin and Edouard Grave and Guillaume Lample},
year = {2023},
eprint = {2302.13971},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2302.13971}
}
At the end of February 2023, ChatGPT had been live for less than three months, and nearly all of the strongest language models were locked behind APIs.
The weights of GPT-3, Gopher, Chinchilla and PaLM were not public, and the training data was often described in a single line, such as "Books – 2TB" or "Social media conversations." Models with public weights did exist. OPT, GPT-NeoX, BLOOM and GLM had all been released, but in the authors' assessment, none of them could match PaLM-62B or Chinchilla.
In this paper, Touvron, Lavril, Izacard, Grave, Lample and colleagues at Meta AI present LLaMA, a set of foundation models from 7 billion to 65 billion parameters. Two claims drew attention as soon as it came out. First, the 13-billion-parameter LLaMA-13B beats the 175-billion-parameter GPT-3 on most benchmarks, and a single V100 is enough to run it for inference. Second, all of the training data comes from public datasets, and the weights were released to the research community ("released" meant filling out an application form; a week later, the weights were all over the internet anyway).
Defining the problem
The year before, Chinchilla had shown that for a given training compute budget, the best results come not from the biggest model but from a smaller model paired with more data. Its goal is to use training compute as efficiently as possible.
The authors point out that this goal ignores inference cost, and once a model is deployed, the total cost of inference is often far greater than that of training. To reach a given level of performance, the better deal is the model that is fastest at inference, not the one that is fastest to train. Training a large model to some level may be cheaper, but a small model trained for longer ends up saving much more at inference time.
The authors give a concrete comparison. Following Chinchilla's recommendation, a 10-billion-parameter model should be trained on roughly 200 billion tokens; but they found that a 7-billion-parameter model was still improving even after seeing 1 trillion tokens.
So the goal of this paper is to train the best-performing model for each of several inference budgets, by having the models see far more tokens than usual.
Let's work out "how many tokens each parameter saw" for a few models. Training FLOPs are about and inference is about per token, where is the parameter count and the number of training tokens:
models = { # parameters, training tokens
"GPT-3 175B": (175e9, 300e9),
"Chinchilla 70B": (70e9, 1.4e12),
"LLaMA-7B": (6.7e9, 1.0e12),
"LLaMA-13B": (13.0e9, 1.0e12),
"LLaMA-65B": (65.2e9, 1.4e12),
}
for name, (n, d) in models.items():
train = 6 * n * d # training FLOPs ≈ 6ND
infer = 2 * n # inference ≈ 2N FLOPs per token
print(f"{name:15s} tokens/param {d / n:6.1f} train {train:.1e} infer/token {infer:.1e}")
# GPT-3 175B tokens/param 1.7 train 3.2e+23 infer/token 3.5e+11
# Chinchilla 70B tokens/param 20.0 train 5.9e+23 infer/token 1.4e+11
# LLaMA-7B tokens/param 149.3 train 4.0e+22 infer/token 1.3e+10
# LLaMA-13B tokens/param 76.9 train 7.8e+22 infer/token 2.6e+10
# LLaMA-65B tokens/param 21.5 train 5.5e+23 infer/token 1.3e+11
LLaMA-65B's ratio is about the same as Chinchilla's, roughly 20 tokens per parameter. The small models are the real point of this paper. LLaMA-7B saw nearly 150 tokens per parameter, far beyond Chinchilla's "optimal" ratio. If LLaMA-13B really can keep up with GPT-3, its training compute is only a quarter of GPT-3's, and its inference cost per token only about a thirteenth.
Solving the problem
Public data only
| Dataset | Sampling proportion | Epochs at 1.4T training tokens | Disk size |
|---|---|---|---|
| CommonCrawl | 67.0% | 1.10 | 3.3 TB |
| C4 | 15.0% | 1.06 | 783 GB |
| GitHub | 4.5% | 0.64 | 328 GB |
| Wikipedia | 4.5% | 2.45 | 83 GB |
| Books | 4.5% | 2.23 | 85 GB |
| ArXiv | 2.5% | 1.06 | 92 GB |
| StackExchange | 2.0% | 1.03 | 78 GB |
Compiled from Table 1 of the paper.
Each source gets its own processing:
- CommonCrawl: five snapshots from 2017 to 2020, run through the CCNet pipeline to deduplicate line by line, filter out non-English pages with fastText, and filter low-quality content with an n-gram language model. On top of that, a linear classifier was trained to tell "pages that would be cited as references by Wikipedia" from random pages, and only the former were kept.
- C4: the authors found that adding CommonCrawl processed in a different way helped, so they also added T5's C4. Its quality filtering relies mainly on rules such as punctuation, word count and sentence count, unlike CCNet.
- GitHub: only projects under Apache, BSD and MIT licenses were kept, filtered by line length and the proportion of alphanumeric characters, then deduplicated exactly at the file level.
- Wikipedia: dumps from June to August 2022, covering 20 languages that use Latin or Cyrillic scripts.
- Books: public-domain books from Project Gutenberg plus Books3 from The Pile; where books overlapped by more than 90% in content, only one copy was kept.
- ArXiv: LaTeX source files, with everything before the first section, the bibliography and the comments removed, and author-defined macros expanded.
- StackExchange: questions and answers from the 28 largest sites, with answers sorted by score from highest to lowest.
Tokenization uses SentencePiece BPE. Numbers are always split into individual digits, and unknown UTF-8 characters fall back to bytes. That comes to about 1.4 trillion tokens in total; most of the data is seen only once, while Wikipedia and books are seen about twice.
Architecture: three borrowed tweaks
LLaMA is a standard Transformer decoder plus three modifications proposed earlier and already used in large models; the paper notes where each one came from in parentheses:
- Pre-normalization (GPT-3): normalize the input of each sub-layer instead of its output, to improve training stability; the normalization function is RMSNorm.
- SwiGLU (PaLM): the ReLU in the feed-forward layer is replaced with SwiGLU.
- Rotary position embeddings, RoPE (GPT-Neo): absolute position embeddings are removed, and RoPE is added at every layer.
There's a detail hiding in the SwiGLU change. The paper says the feed-forward width is rather than PaLM's . The reason is that SwiGLU has three matrices while a standard feed-forward layer has only two; multiplying the width by two-thirds keeps the parameter count about the same as before. The official code then rounds it up to a multiple of 256. Let's run it for LLaMA-7B's :
d, multiple_of = 4096, 256 # d_model of LLaMA-7B
hidden = int(2 * (4 * d) / 3) # two-thirds of 4d
hidden = multiple_of * ((hidden + multiple_of - 1) // multiple_of) # round up to a multiple of 256
print(hidden) # 11008
relu_ffn = 2 * d * (4 * d) # two matrices: d→4d, 4d→d
swiglu_ffn = 3 * d * hidden # three matrices: W, V and the output matrix
print(f"{relu_ffn / 1e6:.1f}M {swiglu_ffn / 1e6:.1f}M") # 134.2M 135.3M
The feed-forward parameters per layer come out nearly identical, and in return you get SwiGLU's quality gain.
The settings for the four sizes are below; every model uses a batch of 4 million tokens:
| Parameters | Attention heads | Layers | Learning rate | Training tokens | |
|---|---|---|---|---|---|
| 6.7B | 4,096 | 32 | 32 | 1.0T | |
| 13.0B | 5,120 | 40 | 40 | 1.0T | |
| 32.5B | 6,656 | 52 | 60 | 1.4T | |
| 65.2B | 8,192 | 64 | 80 | 1.4T |
Compiled from Table 2 of the paper.
The optimizer is AdamW (, ), with a cosine learning rate schedule decaying to 10% of the maximum, weight decay of 0.1, gradient clipping at 1.0 and 2,000 warmup steps.
Making training fast
The authors used several engineering optimizations:
- An efficient implementation of causal attention: taken from the xformers library, it does not store the attention weights and does not compute the key/query scores that the causal mask blocks out.
- Less recomputation during the backward pass: activations that are expensive to compute, such as the outputs of linear layers, are saved. To make this possible, the authors hand-wrote the backward pass for the Transformer layers instead of relying on PyTorch autograd.
- Model parallelism and sequence parallelism, with the computation of activations overlapped as much as possible with communication between GPUs.
When training 65B, each GPU processes about 380 tokens per second. On 2,048 80GB A100s, getting through 1.4 trillion tokens takes about 21 days.

This figure backs up the earlier claim. All four training loss curves are still heading down at the end, and 7B hasn't flattened out even at 1 trillion tokens.
Discussion
Main results
The authors evaluate on 20 benchmarks, comparing against the closed GPT-3, Gopher, Chinchilla and PaLM, as well as the open OPT, GPT-J and GPT-Neo. All numbers are taken from the respective papers.
Commonsense reasoning (zero-shot, 8 benchmarks). LLaMA-65B beats Chinchilla-70B on all but BoolQ, and beats PaLM-540B on all but BoolQ and WinoGrande. LLaMA-13B beats GPT-3 on most benchmarks.
Closed-book question answering. On Natural Questions, 65B scores 23.8 zero-shot and 39.9 at 64-shot, against 21.2 and 39.6 for PaLM-540B. LLaMA-13B also holds its own against GPT-3 and Chinchilla on both closed-book QA benchmarks, and the authors make a point of noting that it needs only a single V100 for inference.
Math. On GSM8K, LLaMA-65B scores 50.9 with a single answer, slightly below Minerva-62B (52.4), which was fine-tuned on 38.5 billion tokens of arXiv papers and math web pages. With majority voting, that is, sampling several answers to the same question and taking a vote, LLaMA-65B scores 69.7, beating Minerva-62B's 68.5. LLaMA was not fine-tuned on any math data.
Code. On HumanEval pass@1, LLaMA-65B scores 23.7, PaLM-62B 15.9 and PaLM-540B 26.2. LLaMA models of 13B and up beat LaMDA-137B on both HumanEval and MBPP. All of these are general-purpose models, none of them trained or fine-tuned specifically for code.
MMLU is where LLaMA clearly falls behind. 65B's 5-shot average is 63.4, against 67.5 for Chinchilla-70B and 69.3 for PaLM-540B. The authors attribute this to too few books and academic papers. ArXiv, Gutenberg and Books3 add up to only 177GB, while Gopher, Chinchilla and PaLM used up to 2TB of books. They also use this to explain why Gopher beats GPT-3 on MMLU while being about even on other benchmarks.
Performance during training

The authors tracked several benchmarks throughout training. Most rise steadily, improving along with the training loss. Two exceptions are worth noting. SIQA scores fluctuate a lot, and the authors suspect this benchmark may be unreliable. WinoGrande scores correlate less strongly with perplexity, and 33B and 65B are nearly identical throughout training.
A simple round of instruction tuning
The authors also ran a small experiment, following the procedure of Chung et al. (Flan-PaLM) to give 65B a brief round of instruction tuning, and called the result LLaMA-I. MMLU rises from 63.4 to 68.9, beating mid-sized instruction-tuned models such as OPT-IML-Max 30B (43.2) and Flan-PaLM 62B (59.6), but still some distance from the best at the time, code-davinci-002 at 77.4. The authors stress that this is not the focus of the paper and that they ran only a single experiment.
Bias, toxicity and truthfulness
Toxicity. On RealToxicityPrompts, toxicity rises as the model grows, and more noticeably with prompts that ask to "complete the sentence in a polite, respectful, and unbiased manner," where 65B scores 0.141 and 7B 0.081.
Stereotypes. On CrowS-Pairs, LLaMA-65B's average bias score is 66.6, slightly better than GPT-3 (67.2) and OPT-175B (69.5), but in the religion category it is about 10 percentage points higher than OPT, followed by age and gender. The authors suspect these biases come from CommonCrawl.
Gender and occupation. On WinoGender, the model resolves "they/them/someone" pronouns noticeably more accurately than "he" and "she." It makes more errors on the "gotcha" questions, where the pronoun doesn't match the majority gender of the occupation, which shows that the model relies on the occupation's stereotyped gender rather than the evidence in the sentence.
Truthfulness. On TruthfulQA, 57% of 65B's answers are truthful and 53% are both truthful and informative, higher than GPT-3 175B's 28% and 25%. The authors also say the rate of correct answers is still low, and the model is quite likely to make up wrong answers.
Compute and carbon
Training LLaMA-65B took about 1.02 million A100 GPU-hours. The authors estimate that the whole development process ran about 5 months on 2,048 A100s, consuming about 2,638 MWh in total, which converts to about 1,015 tonnes of CO2 equivalent at the US average grid carbon intensity. Their hope is that releasing the models will spare others from having to train them all over again.
After the release
LLaMA's weights were made available to researchers under a non-commercial license, by application. A week later, the weights leaked on 4chan as a torrent. Meta then issued takedown requests for copies on Hugging Face and other platforms, but the files had already spread widely.
What happened over the next few weeks shows how much these weights mattered. On March 13, 2023, researchers at Stanford released Alpaca. They used OpenAI's text-davinci-003 to generate 52,000 instruction demonstrations, fine-tuned LLaMA-7B on them, and claimed the whole replication cost less than $600. In their own blind test, Alpaca went 90 wins to 89 losses against text-davinci-003. The evaluation was small, but Alpaca turned "open weights plus a little instruction data" into a recipe that many people followed.
The paper's transparency about its data sources had a flip side, too. Books3 contains a large number of copyrighted books. In July 2023, several authors including Sarah Silverman sued Meta for copyright infringement, alleging that LLaMA was trained on their books, and Books3 was one of the focal points of the suit.
Five months later, Meta released Llama 2, which allowed commercial use; a year after that, Llama 3 pushed further in the same direction, raising pretraining data to about 15 trillion tokens.
Conclusion
LLaMA didn't invent any new components; it changed the question. Chinchilla asked how to spend training compute most cost-effectively, and LLaMA asked how to save the most after deployment. The answer was to have small models read a lot. LLaMA-13B read a trillion tokens and overtook GPT-3 on most benchmarks.
Keep this trade-off in mind when picking a model. The extra tokens read in training cost you only once, while inference runs the whole model for every request. For a service that will stay online for a long time, a smaller model that has read more is usually the better deal.
By the way, by the second generation the name had changed from LLaMA to Llama, so everyone could finally stop counting capital letters.