Back to paper library
arXiv 2022JMLR 2023Language & TransformersPaper notes

PaLM

A whale of a joke

Original paper

PaLM: Scaling Language Modeling with Pathways

Paper authors

Aakanksha Chowdhery · Sharan Narang · Jacob Devlin …

Show all 67 authors

Aakanksha Chowdhery · Sharan Narang · Jacob Devlin · Maarten Bosma · Gaurav Mishra · Adam Roberts · Paul Barham · Hyung Won Chung · Charles Sutton · Sebastian Gehrmann · Parker Schuh · Kensen Shi · Sasha Tsvyashchenko · Joshua Maynez · Abhishek Rao · Parker Barnes · Yi Tay · Noam Shazeer · Vinodkumar Prabhakaran · Emily Reif · Nan Du · Ben Hutchinson · Reiner Pope · James Bradbury · Jacob Austin · Michael Isard · Guy Gur-Ari · Pengcheng Yin · Toju Duke · Anselm Levskaya · Sanjay Ghemawat · Sunipa Dev · Henryk Michalewski · Xavier Garcia · Vedant Misra · Kevin Robinson · Liam Fedus · Denny Zhou · Daphne Ippolito · David Luan · Hyeontaek Lim · Barret Zoph · Alexander Spiridonov · Ryan Sepassi · David Dohan · Shivani Agrawal · Mark Omernick · Andrew M. Dai · Thanumalayan Sankaranarayana Pillai · Marie Pellat · Aitor Lewkowycz · Erica Moreira · Rewon Child · Oleksandr Polozov · Katherine Lee · Zongwei Zhou · Xuezhi Wang · Brennan Saeta · Mark Diaz · Orhan Firat · Michele Catasta · Jason Wei · Kathy Meier-Hellstern · Douglas Eck · Jeff Dean · Slav Petrov · Noah Fiedel

First submitted to arXiv
arXiv:2204.02311
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{transformerspalm2022,
  title = {{PaLM: Scaling Language Modeling with Pathways}},
  author = {Aakanksha Chowdhery and Sharan Narang and Jacob Devlin and Maarten Bosma and Gaurav Mishra and Adam Roberts and Paul Barham and Hyung Won Chung and Charles Sutton and Sebastian Gehrmann and Parker Schuh and Kensen Shi and Sasha Tsvyashchenko and Joshua Maynez and Abhishek Rao and Parker Barnes and Yi Tay and Noam Shazeer and Vinodkumar Prabhakaran and Emily Reif and Nan Du and Ben Hutchinson and Reiner Pope and James Bradbury and Jacob Austin and Michael Isard and Guy Gur-Ari and Pengcheng Yin and Toju Duke and Anselm Levskaya and Sanjay Ghemawat and Sunipa Dev and Henryk Michalewski and Xavier Garcia and Vedant Misra and Kevin Robinson and Liam Fedus and Denny Zhou and Daphne Ippolito and David Luan and Hyeontaek Lim and Barret Zoph and Alexander Spiridonov and Ryan Sepassi and David Dohan and Shivani Agrawal and Mark Omernick and Andrew M. Dai and Thanumalayan Sankaranarayana Pillai and Marie Pellat and Aitor Lewkowycz and Erica Moreira and Rewon Child and Oleksandr Polozov and Katherine Lee and Zongwei Zhou and Xuezhi Wang and Brennan Saeta and Mark Diaz and Orhan Firat and Michele Catasta and Jason Wei and Kathy Meier-Hellstern and Douglas Eck and Jeff Dean and Slav Petrov and Noah Fiedel},
  year = {2022},
  eprint = {2204.02311},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2204.02311}
}
On this page

"Google just hired an eloquent whale for their TPU team. It showed them how to communicate between two different pods!"

You need to know two things to get this pun. A pod is a group of TPU chips, and it is also a group of whales. In April 2022, Google had the model explain this joke in the very first figure of the PaLM paper:

Original Figure 1. Two examples of explanations generated by PaLM 540B. On the left, "Explaining a Joke": the input is that Google hired an eloquent whale for its TPU team, which showed them how to communicate between two pods, and the model explains that a pod is both a group of TPUs and a group of whales. On the right, "Logical Inference": the input is that Shelley, from Virginia, is visiting the city with that famous market where they throw the fish, and the model works out that the city is Seattle, which is on the Pacific coast, so it answers that she is likely to be near the Pacific this weekend.
Original Figure 1, Chowdhery et al., from PDF page 4, CC BY 4.0.

On the left is the joke. The model explains that the whale can communicate between two groups of whales, and the speaker is pretending it can communicate between two groups of TPUs. On the right is a different kind of question, where the model first has to recognize that "the market where they throw the fish" is in Seattle before it can answer whether Shelley will be near the Pacific this weekend. Both examples use greedy decoding, with two exemplars that include explanations in the prompt; that is the chain-of-thought prompting proposed earlier that same year.

The joke is also about this paper. PaLM (Pathways Language Model) is a 540-billion-parameter dense language model trained on 6,144 TPU v4 chips across two TPU v4 Pods, made possible by Google's new Pathways system.

Ugh, this is so annoying! A joke that takes this much explaining isn't funny at all.

Defining the problem

After GPT-3, GLaM, Gopher, Chinchilla, Megatron-Turing NLG and LaMDA kept pushing few-shot results forward. The authors boil these advances down to four directions:

  1. scaling up the depth and width of the model;
  2. increasing the number of training tokens;
  3. using cleaner data from more diverse sources;
  4. adding capacity with sparse modules without adding computation.

PaLM mainly takes the first route, scaling up a dense model in which every parameter takes part in the computation, to see whether few-shot ability has already saturated.

Doing that means solving engineering problems first. The largest training runs at the time used 4,096 TPU v3 chips. Megatron-Turing NLG used 2,240 A100s and Gopher used four TPU v3 Pods, and both relied on pipeline parallelism to span multiple machines. Pipeline parallelism suffers from "bubbles," where many devices sit idle while the pipeline fills and drains, and it also adds load on memory bandwidth. The authors wanted to scale training to two Pods without pipeline parallelism.

Solving the problem

Architecture: a set of practical tweaks

PaLM is a decoder-only Transformer with a few modifications:

  • SwiGLU: the activation function in the feed-forward layer becomes Swish(xW)⋅xV\text{Swish}(xW)\cdot xV. This takes three matrix multiplications, but Shazeer showed that it improves quality in comparisons at equal compute.
  • Parallel layers: the standard formulation runs attention first and then the feed-forward layer; PaLM has both read the same input and adds their outputs:
Standard: y=x+MLP(LN(x+Attn(LN(x))))Parallel: y=x+MLP(LN(x))+Attn(LN(x))\text{Standard:}\ y = x + \text{MLP}(\text{LN}(x + \text{Attn}(\text{LN}(x)))) \qquad \text{Parallel:}\ y = x + \text{MLP}(\text{LN}(x)) + \text{Attn}(\text{LN}(x))

The input matrix multiplications of attention and the feed-forward layer can be fused, making training about 15% faster at large scale. The authors' ablations show a slight quality drop at 8B and no difference at 62B, so they infer that 540B is unaffected as well.

  • Multi-Query Attention: each attention head has its own query, but all heads share one set of keys and values. This has little effect on training quality or speed, but saves a great deal of cost during autoregressive decoding. (Llama 2's 34B and 70B later switched to GQA, which sits between the two.)
  • RoPE position embeddings and shared input-output embeddings.
  • No biases: none of the dense layers or LayerNorms have bias terms, which the authors found makes large-model training more stable.
  • A 256k SentencePiece vocabulary: fully reversible, preserving all whitespace (important for code). Out-of-vocabulary Unicode characters are split into UTF-8 bytes, and numbers are always split into individual digits, so "123.5" becomes "1 2 3 . 5". LLaMA followed suit the next year.

Here are the configurations of the three sizes. The feed-forward width is fixed at 4×dmodel4\times d_{\text{model}}, and each attention head is fixed at 256 dimensions:

ModelLayersAttention headsdmodeld_{\text{model}}ParametersBatch size
PaLM 8B32164,0968.63B256 → 512
PaLM 62B64328,19262.5B512 → 1,024
PaLM 540B1184818,432540.35B512 → 1,024 → 2,048

Compiled from Table 1 of the paper.

Data: 780 billion tokens, seen once

The training data builds on the data used for LaMDA and GLaM, 780 billion tokens in total: social media conversations 50%, filtered web pages 27%, English books 13%, GitHub code 5%, Wikipedia 4% and English news 1%. Web pages are sampled according to a quality classifier's score. The higher the score, the more often a page gets picked, but low-scoring pages are not excluded entirely. Code comes from open-source GitHub repositories. Excluding copyleft licenses and keeping 24 common languages leaves 196GB of source code, from which duplicate files are then removed based on Levenshtein distance. Non-English data makes up about 22%.

All three models see this data exactly once, and the mix of sources is set so that no part of it gets repeated.

Infrastructure: two Pods, no pipeline parallelism

Each TPU v4 Pod has 3,072 chips and 768 hosts. Within a Pod, every weight tensor is split 12 ways for model parallelism, combined with 256-way fully sharded data parallelism; each Pod holds a complete copy of the model.

Across the two Pods, Pathways handles data parallelism. A Python client splits the batch in half and sends one half to each Pod. Each Pod computes its forward and backward passes, the two exchange gradients, and each then sums them and updates its parameters, ending up with bitwise-identical weights.

The hard part is the network between the Pods. Gradient transfer can only begin once every chip has finished computing, so all 1,536 hosts start sending data at the same instant. Each pair of hosts exchanges about 1.3GB of gradients per step, for an aggregate burst of 81 Tbps. The authors split the gradients into small chunks and spread them across many data-center network links, and in the end two Pods achieve 1.95 times the throughput of one. The remaining gap to the theoretical 2x comes from the backward pass not overlapping with the cross-Pod gradient transfer.

Measuring efficiency with model FLOPs utilization

The authors argue that the commonly used "hardware FLOPs utilization" (HFU) is a poor basis for comparison. It depends on the implementation. When rematerialization trades compute for memory, for example, hardware FLOPs go up, but that is not what training is really after. What training cares about is how many tokens it processes per second.

So the authors propose model FLOPs utilization (MFU), the observed token throughput divided by the theoretical throughput the hardware would reach running at peak and doing only the required computation. Required computation counts only the forward and backward passes, not rematerialization. A decoder with NN parameters needs 6N6N matmul FLOPs per token (2N2N forward, 4N4N backward), plus 12LHQT12LHQT for attention. Let's run the numbers for PaLM 540B:

python
N = 540.35e9                     # parameter count
L, H, Q, T = 118, 48, 256, 2048  # layers, attention heads, dims per head, sequence length
chips, peak = 6144, 275e12       # TPU v4 chips, peak matmul FLOP/s per chip
tokens_per_sec = 238.3e3         # measured training throughput

flops_per_token = 6 * N + 12 * L * H * Q * T   # forward 2N + backward 4N, plus attention
mfu = tokens_per_sec * flops_per_token / (chips * peak)
print(f"{6 * N:.3e} {12 * L * H * Q * T:.3e}")  # 3.242e+12 3.563e+10
print(f"{mfu:.1%}")                              # 46.2%

The attention term is only about 1% of 6N6N; for a large model, nearly all the computation goes into the matrix multiplications with the weights. Computed the same way, GPT-3 comes to 21.3%, Gopher 32.5% and Megatron-Turing NLG 30.2%. PaLM's 46.2% comes from its parallelism strategy, XLA compiler optimizations and the parallel layers. Counting the rematerialization FLOPs as well, hardware utilization is 57.8%.

Training setup and loss spikes

A few of the less usual settings:

  • The optimizer is Adafactor without factorization, which is roughly equivalent to Adam with "parameter scaling." The learning rate is held at 10−210^{-2} for the first 10,000 steps, then decays as 1/k1/\sqrt{k}. The second-moment coefficient is set to β2=1−k−0.8\beta_2 = 1 - k^{-0.8}, where kk is the step number; the authors found this more stable than a fixed 0.99, because the second moment for the embeddings of rare tokens can't be estimated well over a short window.
  • An extra z-loss term, 10−4⋅log⁡2Z10^{-4}\cdot\log^2 Z, keeps the softmax normalizer log⁡Z\log Z close to 0 and improves stability.
  • The batch size doubles as training goes on. 540B uses 512 sequences (about 1 million tokens) up to step 50,000, then 1,024 up to step 115,000, and finally 2,048 (about 4 million tokens) until training ends at step 255,000. Small batches early on are more sample-efficient; large batches later give more accurate gradient estimates, and the larger matrices make the TPUs more efficient.
  • Pretraining uses no dropout, and restarting from any checkpoint reproduces the run bit for bit.

Even with gradient clipping, the 540B run hit about 20 loss spikes during training, at very irregular intervals and sometimes late in training; the smaller models had no such problem. The authors found no principled fix. In the end, they restarted from a checkpoint about 100 steps before the spike and skipped the 200 to 500 batches around it.

They also checked whether "bad data" was the cause. Training on the batches from around a spike, starting from an earlier checkpoint, produced no spike. So a spike happens only when particular data meets a particular parameter state. The bitwise reproducibility mentioned above also makes checks like this clean, since rerunning from the same checkpoint gives exactly the same result.

Discussion

English NLP and MMLU

On the 29 English benchmarks used in the GPT-3 paper, PaLM 540B's few-shot results set new highs on 28, compared against the best results of GLaM, GPT-3, Megatron-Turing NLG, Gopher, Chinchilla and LaMDA. Its 5-shot MMLU average is 69.3, against the previous best of 67.5 from Chinchilla; of the four major categories, only "Other" loses to Chinchilla.

BIG-bench: above the human average, but not on every task

Original Figure 3. Normalized BIG-bench score against non-embedding parameter count. The left plot covers 58 common tasks, comparing Gopher, Chinchilla, GPT-3 and PaLM 0/1/5-shot, with horizontal lines for average human and best human performance; PaLM 5-shot at 540B is about 54, above the human average of about 48 and far below the best human score of about 90. The right plot shows PaLM on 150 tasks.
Original Figure 3, Chowdhery et al., from PDF page 15, CC BY 4.0.

BIG-bench is a collaborative benchmark of more than 150 tasks. The human baseline was obtained by having crowdworkers (usually 10) read the same task descriptions and examples and answer, with any tools they liked, such as search engines and calculators. "Average human" is the average over everyone, and "best human" takes the highest score on each question.

The left plot covers only the 58 tasks on which all three model families were evaluated. PaLM 540B 5-shot is above the human average, and it beats the previous best on 44 of those tasks. The three PaLM sizes fall roughly on a log-linear line, which the authors take as a sign that scaling further would do even better. The right plot shows PaLM on 150 tasks, with a similar trend.

It is still a long way from the best human. Task by task, the average human still beats PaLM 540B on 35% of them.

Discontinuous improvements

Original Figure 5. 5-shot normalized scores at 8B, 62B and 540B on six BIG-bench tasks, with average human and best human performance marked. (a) goal_step_wikihow and logical_args rise log-linearly with scale; (b) english_proverbs and logical_sequence jump sharply between 62B and 540B; (c) navigate and mathematical_induction stay nearly flat.
Original Figure 5, Chowdhery et al., from PDF page 17, CC BY 4.0.

The authors picked six tasks to show three different scaling curves. (a) rises steadily with scale. (c) barely improves, with 540B only slightly better than 62B and far from the best human. (b) improves only a little from 8B to 62B, then makes a sudden big jump from 62B to 540B.

Here is how the authors quantify "discontinuity." Take english_proverbs, where the three sizes score about 13%, 25% and 87%. Going from 8B to 62B multiplies the parameters by about 8, and going from 62B to 540B does roughly the same. Extrapolating the log-linear trend of the first step, 540B should land at 25%+(25%−13%)=37%25\% + (25\% - 13\%) = 37\%, but it actually reaches 87%, 50 percentage points higher. Across all 150 tasks, 25% show a discontinuity of more than 10 percentage points, and 15% more than 20 percentage points.

english_proverbs requires understanding the metaphors behind proverbs, which the authors consider to demand a very high level of abstract reasoning. mathematical_induction is the opposite case. Its questions give premises that may contradict real-world math, such as "2 is an odd integer," and then ask whether the inductive argument is correct. Every PaLM size, and the humans who did the same task, had trouble with the questions whose premises are false.

Reasoning and code

PaLM also uses chain-of-thought prompting, proposed a little earlier by the same team. With 8-shot chain-of-thought prompting plus an external calculator, it reaches 58% on GSM8K:

Model and techniqueGSM8K accuracy
PaLM 540B, chain of thought + calculator58%
PaLM 540B, chain of thought54%
PaLM 540B, without chain of thought17%
PaLM 62B, chain of thought33%
GPT-3, fine-tuning + chain of thought + calculator34%
GPT-3, fine-tuning + chain of thought + calculator + verifier55%

Compiled from Table 10 of the paper.

The earlier 55% needed fine-tuning, a calculator and a specially trained verifier on top; PaLM beats it with few-shot prompting alone.

On code, PaLM 540B, which was not trained specifically for code, reaches 76.2% pass@100 on HumanEval, comparable to the specialized Codex 12B (72.3%). PaLM's training data contains only about 2.7 billion Python tokens, less than a thirtieth of Codex's (about 100 billion). The authors read this as transfer from other programming languages and from natural language, combined with the higher sample efficiency of large models. PaLM-Coder 540B, obtained by further fine-tuning on code, reaches 88.4% pass@100 on HumanEval.

Memorization and bias

The authors randomly sampled 100-token spans from the training data, gave the model the first 50 tokens, and checked whether greedy decoding could reproduce the remaining 50 word for word. 8B could reproduce 1.6% of the spans and 540B 2.4%; the bigger the model, the more it remembers. For spans that appear only once, 540B's memorization rate is 0.75%; for spans that appear more than 500 times, it is over 40%. The code corpus, full of license text and copy-pasted snippets, is the easiest to memorize; the book corpus, mostly unique text, is the hardest.

In the bias analysis, the model's continuations of religion-related prompts tie Islam to words such as terrorism, extremism and violence, a tendency present at every size. On toxicity, the overall toxicity of 62B and 540B is slightly higher than that of 8B. The toxicity of the model's continuations is highly correlated with the toxicity of the prompt itself, while human continuations show no such strong correlation.

Questions the authors leave open

The paper's "Open Questions in Scaling" section is an interesting read. PaLM mainly scales up model size, but the authors admit they ran no ablations to separate the contribution of "a bigger model" from that of "more tokens." They ask how, on the same training budget, a 62B model trained on 7 trillion tokens, or a 120B model trained on 3.6 trillion, would compare with the 540B trained on 780 billion. If a smaller model performed just as well, inference would be much cheaper.

DeepMind's Chinchilla had tackled exactly this question just a week before PaLM came out. The authors put the comparable results from the two side by side:

ModelParameters (B)Tokens (B)Training FLOPs (ZFLOPs)Average over 9 English benchmarks
PaLM 62B62795295.764.8
Chinchilla701,400588.065.2
Gopher280300504.059.5
PaLM 540B5407802,527.270.3

Compiled from Table 21 of the paper.

PaLM 62B, trained with about half of Chinchilla's FLOPs, trails Chinchilla's average by only 0.4 points, while Gopher falls clearly behind. The authors agree that this supports Chinchilla's conclusion that Gopher was undertrained for its size. But PaLM and Chinchilla were trained on different data, so the authors say these results can't answer "would a model of size X trained on Y tokens beat PaLM 540B?"

A year later, the PaLM 2 technical report answered part of it. Following Chinchilla's conclusion, it scales data and model size at roughly a 1:1 ratio; the largest model, PaLM 2-L, is much smaller than PaLM 540B yet uses more training compute.

Conclusion

Push a dense model to 540 billion parameters, and few-shot ability is still climbing. It took 28 of 29 English benchmarks, beat fine-tuning plus a verifier on GSM8K with prompting alone, and on a quarter of BIG-bench tasks something suddenly clicked between 62B and 540B.

If you want to know how efficient your own training run is, MFU is a ready-made yardstick. Multiply the tokens processed per second by 6N6N, divide by the chips' peak compute, and you can put the result next to PaLM's 46.2%.

Still, at Chinchilla's ratio of roughly 20 tokens per parameter, a 540-billion-parameter model should have read about 11 trillion tokens, and PaLM read only 780 billion. The whale is plenty smart. It just hasn't read quite enough.