Gemma 2: Improving Open Language Models at a Practical Size
- Publication
- Publication venue not yet verified
- First submitted to arXiv
- arXiv:2408.00118
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{transformersgemma22024,
title = {{Gemma 2: Improving Open Language Models at a Practical Size}},
author = {Gemma Team and Morgane Riviere and Shreya Pathak and Pier Giuseppe Sessa and Cassidy Hardin and Surya Bhupatiraju and Léonard Hussenot and Thomas Mesnard and Bobak Shahriari and Alexandre Ramé and Johan Ferret and Peter Liu and Pouya Tafti and Abe Friesen and Michelle Casbon and Sabela Ramos and Ravin Kumar and Charline Le Lan and Sammy Jerome and Anton Tsitsulin and Nino Vieillard and Piotr Stanczyk and Sertan Girgin and Nikola Momchev and Matt Hoffman and Shantanu Thakoor and Jean-Bastien Grill and Behnam Neyshabur and Olivier Bachem and Alanna Walton and Aliaksei Severyn and Alicia Parrish and Aliya Ahmad and Allen Hutchison and Alvin Abdagic and Amanda Carl and Amy Shen and Andy Brock and Andy Coenen and Anthony Laforge and Antonia Paterson and Ben Bastian and Bilal Piot and Bo Wu and Brandon Royal and Charlie Chen and Chintu Kumar and Chris Perry and Chris Welty and Christopher A. Choquette-Choo and Danila Sinopalnikov and David Weinberger and Dimple Vijaykumar and Dominika Rogozińska and Dustin Herbison and Elisa Bandy and Emma Wang and Eric Noland and Erica Moreira and Evan Senter and Evgenii Eltyshev and Francesco Visin and Gabriel Rasskin and Gary Wei and Glenn Cameron and Gus Martins and Hadi Hashemi and Hanna Klimczak-Plucińska and Harleen Batra and Harsh Dhand and Ivan Nardini and Jacinda Mein and Jack Zhou and James Svensson and Jeff Stanway and Jetha Chan and Jin Peng Zhou and Joana Carrasqueira and Joana Iljazi and Jocelyn Becker and Joe Fernandez and Joost van Amersfoort and Josh Gordon and Josh Lipschultz and Josh Newlan and Ju-yeong Ji and Kareem Mohamed and Kartikeya Badola and Kat Black and Katie Millican and Keelin McDonell and Kelvin Nguyen and Kiranbir Sodhia and Kish Greene and Lars Lowe Sjoesund and Lauren Usui and Laurent Sifre and Lena Heuermann and Leticia Lago and Lilly McNealus and Livio Baldini Soares and Logan Kilpatrick and Lucas Dixon and Luciano Martins and Machel Reid and Manvinder Singh and Mark Iverson and Martin Görner and Mat Velloso and Mateo Wirth and Matt Davidow and Matt Miller and Matthew Rahtz and Matthew Watson and Meg Risdal and Mehran Kazemi and Michael Moynihan and Ming Zhang and Minsuk Kahng and Minwoo Park and Mofi Rahman and Mohit Khatwani and Natalie Dao and Nenshad Bardoliwalla and Nesh Devanathan and Neta Dumai and Nilay Chauhan and Oscar Wahltinez and Pankil Botarda and Parker Barnes and Paul Barham and Paul Michel and Pengchong Jin and Petko Georgiev and Phil Culliton and Pradeep Kuppala and Ramona Comanescu and Ramona Merhej and Reena Jana and Reza Ardeshir Rokni and Rishabh Agarwal and Ryan Mullins and Samaneh Saadat and Sara Mc Carthy and Sarah Cogan and Sarah Perrin and Sébastien M. R. Arnold and Sebastian Krause and Shengyang Dai and Shruti Garg and Shruti Sheth and Sue Ronstrom and Susan Chan and Timothy Jordan and Ting Yu and Tom Eccles and Tom Hennigan and Tomas Kocisky and Tulsee Doshi and Vihan Jain and Vikas Yadav and Vilobh Meshram and Vishal Dharmadhikari and Warren Barkley and Wei Wei and Wenming Ye and Woohyun Han and Woosuk Kwon and Xiang Xu and Zhe Shen and Zhitao Gong and Zichuan Wei and Victor Cotruta and Phoebe Kirk and Anand Rao and Minh Giang and Ludovic Peran and Tris Warkentin and Eli Collins and Joelle Barral and Zoubin Ghahramani and Raia Hadsell and D. Sculley and Jeanine Banks and Anca Dragan and Slav Petrov and Oriol Vinyals and Jeff Dean and Demis Hassabis and Koray Kavukcuoglu and Clement Farabet and Elena Buchatskaya and Sebastian Borgeaud and Noah Fiedel and Armand Joulin and Kathleen Kenealy and Robert Dadashi and Alek Andreev},
year = {2024},
eprint = {2408.00118},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2408.00118}
}
The most common way to make a small model stronger is to feed it more data.
Llama 3's 8B model saw 15T tokens, far more than scaling laws recommend. This works, but with diminishing returns. The report points out at the very start that, under Chinchilla's scaling laws, performance grows only about logarithmically with the amount of data; the latest small models use vastly more data to gain just 1 to 2 percentage points on most benchmarks.
This also shows that small models are still undertrained. The problem is that data runs out eventually, and every doubling of data doubles the training cost as well.
Google DeepMind's Gemma team released Gemma 2 in June 2024 and went in a different direction. Rather than have a small model see more tokens, they let it learn more from each one. The method is knowledge distillation, which replaces the original ground-truth answer with a large model's predicted distribution.
Gemma 2 comes in three sizes: 2B, 9B, and 27B. 2B and 9B are trained with distillation; 27B is trained from scratch.
Defining the problem
Standard language-model training tells the model only one thing at each position: what the next token is.
Take "Today I'd like to drink some ___." If the next word in the training data is "coffee," the target is a one-hot vector that is 1 for "coffee" and 0 everywhere else. But "tea" and "water" are perfectly reasonable too, and the one-hot target says nothing about them.
A large model that has already been trained might say "coffee 0.6, tea 0.3, water 0.1." With this whole distribution as the target, the small model learns one more thing at the same position: "tea works too." The authors state the purpose of distillation plainly, using it to "simulate training beyond the number of available tokens."
The distillation loss is the cross-entropy between the teacher distribution and the student distribution :
is the context, and the sum runs over the entire vocabulary. Replace the teacher distribution with a one-hot vector and this expression reduces to ordinary next-token prediction.
The idea is not new; Gemini 1.5 also used distillation. If distillation is used in large-scale pretraining, with small models seeing far more tokens than the optimal amount, how well does it work? That is what this report sets out to answer.
Solving the problem
Architecture
Like the first generation, Gemma 2 is a decoder-only Transformer with a context length of 8,192, using RoPE and approximated GeGLU. There are four new design choices:
- Alternating local and global attention. Layers alternate between the two types. Local layers use a 4,096-token sliding window, and global layers see the full 8,192 tokens.
- Logit soft-capping. In the attention layers and the final output layer, logits pass through , with the cap set to 50 for attention layers and 30 for the output layer.
- Normalization before and after. Both the input and the output of every sublayer go through RMSNorm, to make training more stable.
- Grouped-query attention (GQA), with 2 groups.

The two masks on the left show what local layers save. In a global layer, each token sees all previous tokens, so computation grows with the square of the length; in a local layer, each token sees at most the tokens inside the window, a fixed amount. Alternating the two saves computation in half the layers while keeping the ability to look at the whole context every other layer.
The right plot is soft-capping. Small logits are barely affected, since tanh is close to a straight line near 0; very large logits are held near the cap, so that no single value grows large enough to make softmax attend almost entirely to one position. This design helps training stability, but as we will see later, it also caused a bit of trouble on the inference side.
The configurations of the three models (from the report's Table 1):
| 2B | 9B | 27B | |
|---|---|---|---|
| Hidden dimension | 2,304 | 3,584 | 4,608 |
| Layers | 26 | 42 | 46 |
| Feedforward dimension | 18,432 | 28,672 | 73,728 |
| Attention heads / KV heads | 8 / 4 | 16 / 8 | 32 / 16 |
| Vocabulary | 256,128 | 256,128 | 256,128 |
The vocabulary reuses Gemini's 256K tokens, originally designed to support a large number of languages, so the embedding layer has a lot of parameters. The 2B model has about 590 million parameters in its embedding layer and about 2 billion non-embedding parameters.
Pretraining data and hardware
27B is trained on 13T tokens, 9B on 8T, and 2B on 2T. The data is mainly English and includes web documents, code, and science articles. Gemma 2 is not a multimodal model, and it was not specifically trained for multilingual ability.
For both 2B and 9B, the amount of data far exceeds what scaling laws recommend. Using the common rough estimate of "about 20 tokens per parameter," the optimum for 2B is about 40B tokens, but it used 2T, 50 times as much; for 9B it is about 180B tokens, but it used 8T, still more than forty times as much.
Training ran on Google's own TPUs: 512 TPUv5e chips for 2B, 4,096 TPUv4 chips for 9B, and 6,144 TPUv5p chips for 27B. The report estimates the carbon emissions of all Gemma 2 pretraining at about 1,248 tonnes of CO2 equivalent.
Post-training: averaging is the last step
Post-training has three steps:
- SFT: on synthetic and real prompts, with responses mostly generated by a larger teacher model; distillation is also applied on the student's own output distribution.
- RLHF: the reward model is an order of magnitude larger than the policy model and puts more weight on multi-turn conversation. The prompts include LMSYS-Chat-1M, but only its prompts are used, not the responses in it.
- Model merging: average the models obtained by running the whole pipeline with different hyperparameters.
The third step cites WARP by Ramé et al. from the same team. The report covers it in a single sentence and gives no before-and-after comparison.
Discussion
How much does distillation actually help?
The report has few ablations, but each one targets a design decision. The most important concerns distillation:
| 2B model, 500B training tokens | From scratch | Distilled from a 7B teacher |
|---|---|---|
| Average over three benchmarks | 60.3 | 67.7 |
500B tokens is already 10 times the optimum for a 2B model, and distillation still comes out 7.4 points higher. A 7B teacher is used here to mimic the size ratio of "27B teaching 9B."
Does the benefit of distillation disappear as models grow? The authors fix the 7B teacher and train students of three sizes, 200M, 400M, and 1B:
| Validation perplexity | 200M | 400M | 1B |
|---|---|---|---|
| From scratch | 23 | 19 | 17 |
| Distilled | 21 | 17 | 15 |
Perplexity is consistently 2 lower at all three sizes, and the gap does not shrink.
The comparison between released models points the same way. Gemma 2's 2B used 2T tokens, against 3T for the first-generation 2B; with less data, the new model is clearly better across benchmarks, and 9B is about 10% higher than the first-generation 7B on some benchmarks. The authors credit this to distillation.
Keep in mind, though, that distillation was not the only change between the two generations. The architecture, data, and post-training changed too, so this comparison cannot credit distillation alone. The truly clean evidence is the two ablation tables above, and both were run in settings much smaller than the released models.
Other architecture ablations
- GQA vs. multi-head attention: on the 9B model, the average over four benchmarks is 50.3 for multi-head attention and 50.8 for GQA, a small difference. The authors chose GQA because it has fewer parameters and faster inference.
- Deep vs. wide: at the same 9B parameter count, the deeper version averages 52.0 and the wider one 50.8. The gap is small but consistent, so Gemma 2 is deeper than the first generation.
- Shrinking the sliding window at inference: shrinking the 9B model's window from 4,096 to 2,048 and 1,024 moves perplexity only from 1.63 to 1.63 and 1.64. If needed, a smaller window can buy a little inference speed.
- Format robustness: across 12 combinations of prompt and evaluation formats, the standard deviation of MMLU scores is 0.9 for Gemma 2 9B and 1.0 for 27B, against 6.9 for Mistral 7B.
Results
On Hugging Face's evaluation suite, the pretrained 27B scores 75.2 on MMLU to Llama 3 70B's 79.2, 74.0 to 76.9 on GSM8K, and 86.4 to 88.0 on HellaSwag. The model is 2.5 times smaller, and its 13T training tokens are also fewer than Llama 3's 15T, yet it trails by only a few points. The authors also say frankly that from-scratch models improve only about logarithmically with size, so 27B may simply sit on the same Pareto curve as Llama 3.
The post-trained versions' results on LMSYS Chatbot Arena drew the most attention. 27B has an Elo of 1218, higher than Llama 3 70B's 1206; 9B is at 1187, on par with GPT-4-0314's 1186; 2B (strictly speaking, 2.6B) is at 1126, higher than GPT-3.5-Turbo-0613's 1116.
All these comparisons count total parameters. The authors specifically explain that this is because, when open models run on ordinary devices, the limit is usually total memory use rather than active parameters.
Human evaluation and memorization
Besides Arena, the authors ran two further sets of human evaluations. On single-turn prompts, they compared safety and instruction following against GPT-4o. For multi-turn, raters scored overall satisfaction and goal achievement from 1 to 5 across 500 multi-turn conversation scenarios. Compared with the first-generation Gemma 1.1 7B, all three Gemma 2 sizes did better on both measures; for 27B, for example, user satisfaction rises from 3.32 to 4.20.
The report also measured verbatim memorization of training data. Give the model the first 50 tokens of a passage from the training data and see whether it reproduces the following 50 tokens word for word. Gemma 2's verbatim memorization rate is below 0.1%, clearly lower than comparably sized models. Approximate memorization, using a 10% edit-distance threshold, is higher, but still lower than the first-generation Gemma's verbatim memorization. Broken down by data source, code, Wikipedia, and science articles are memorized more.
The real cost of soft-capping
Soft-capping takes up just two lines in the paper, yet it caused plenty of trouble in the community.
When Gemma 2 was released, none of the mainstream FlashAttention implementations supported soft-capping on attention scores. Hugging Face's release notes warned that you had to either give up FlashAttention's speedup or sacrifice performance. When llama.cpp added the soft-capping fix, loading Gemma 2 simply switched FlashAttention off, and GGUF files that had already been converted had to be regenerated. Soon after release, Hugging Face users also reported typos and hallucinations in 27B's outputs, and the fix pointed to in those threads was adding this step on the inference side.
A seemingly harmless stability trick took the whole inference ecosystem several weeks to catch up with.
What the report leaves out
The report does not detail the composition of the pretraining data, nor does it say which model served as the teacher for 2B and for 9B. Going by the ablations, the authors targeted a "27B teaching 9B" ratio, but the actual teacher is not named. Post-training data, hyperparameters, and model-merging details likewise get only a few sentences.
Conclusion
What Gemma 2 has to say is not complicated. Rather than having small models see more data, make each token carry more information. With a large model's probability distribution as the training target, 2B and 9B were distilled on far more data than the optimum and got the best results in their class in return.
The architectural changes, alternating local and global attention, soft-capping, and normalization before and after, are not new inventions, but they combine in a very practical way.
Before deploying Gemma 2, check whether your inference framework implements soft-capping. Without that one tanh, the model starts making typos.