Back to paper library
arXiv 2022NeurIPS 2022Language & TransformersPaper notes

Chain-of-Thought

Think before you answer

Original paper

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

Paper authors

Jason Wei · Xuezhi Wang · Dale Schuurmans …

Show all 9 authors

Jason Wei · Xuezhi Wang · Dale Schuurmans · Maarten Bosma · Brian Ichter · Fei Xia · Ed Chi · Quoc Le · Denny Zhou

Publication
NeurIPS 2022Conference
First submitted to arXiv
arXiv:2201.11903
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{transformerschainofthought2022,
  title = {{Chain-of-Thought Prompting Elicits Reasoning in Large Language Models}},
  author = {Jason Wei and Xuezhi Wang and Dale Schuurmans and Maarten Bosma and Brian Ichter and Fei Xia and Ed Chi and Quoc Le and Denny Zhou},
  year = {2022},
  eprint = {2201.11903},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2201.11903}
}
On this page

"The cafeteria had 23 apples. If they used 20 to make lunch and bought 6 more, how many apples do they have?"

For a grade-schooler this is easy. Put it to a large language model from early 2022, show it one "question, answer" example in the prompt first, and it copies the format and answers right away: 27.

In this paper, Wei, Wang, Schuurmans, Zhou and colleagues at Google change only one thing. Before the answer to the example question, they add a few lines showing how it was solved:

text
Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls.
   Each can has 3 tennis balls. How many tennis balls does he have now?
A: Roger started with 5 balls. 2 cans of 3 tennis balls each is 6 tennis balls.
   5 + 6 = 11. The answer is 11.

The model then follows suit, writing out the working before the answer, and gets 23 − 20 = 3 and 3 + 6 = 9. The authors call these intermediate steps a chain of thought, and this way of prompting chain-of-thought prompting.

The method needs no fine-tuning and no new model. The paper measures the phenomenon carefully, including which tasks it works on, how big the model has to be, where the gain comes from, and which plausible-sounding explanations fall apart.

Defining the problem

Scaling up language models steadily improves performance on many tasks, but arithmetic, commonsense and symbolic reasoning are the exceptions. DeepMind's Gopher paper had already pointed out that scaling alone does not lift performance on tasks like these.

The authors draw on two existing lines of work. The first has models generate intermediate steps. In 2017, for example, Ling et al. trained a model to solve math problems using natural-language rationales, and in 2021 Cobbe et al. at OpenAI released GSM8K, a dataset of grade-school math problems, and fine-tuned GPT-3 on data that included worked solutions. The trouble is that writing large numbers of high-quality solutions is expensive, far harder to collect than ordinary input-output pairs.

The second is the few-shot prompting that GPT-3 popularized: put a few examples in the prompt, and there is no need to fine-tune the model for each task. It works well on simple question-answering tasks but falls apart on tasks that require reasoning, and making the model bigger doesn't help much.

Each approach has its strengths and its limits. The authors want to know whether prompting alone, with just a few examples that include the reasoning, can get a model to carry out multi-step reasoning.

Solving the problem

Original Figure 1. A comparison of standard prompting and chain-of-thought prompting. On the left, the exemplar in standard prompting gives the answer 11 directly, and the model outputs the wrong answer 27 for the apple question. On the right, the exemplar in chain-of-thought prompting first writes out the working 5 + 6 = 11, and for the same question the model first computes 23 − 20 = 3, then 3 + 6 = 9, arriving at the correct answer 9.
Original Figure 1, Wei et al., from PDF page 1, CC BY 4.0.

The prompts on the two sides differ only in the blue text before the exemplar's answer. The model on the right is never told to "work it out step by step." It simply imitates the exemplar's format, writing out the working first and then giving the answer.

In standard few-shot prompting, each exemplar is an (input, output) pair. Chain-of-thought prompting turns it into an (input, chain of thought, output) triple. The authors wrote 8 exemplars by hand for math word problems and applied them to almost every arithmetic benchmark; only AQuA, which is multiple choice, gets its own 4 exemplars taken from the training set. The authors stress that this set of exemplars did not go through any prompt engineering.

The authors list four advantages of chains of thought:

  1. A multi-step problem can be broken into intermediate steps, and harder problems can be given more computation.
  2. It lets people see how the model arrived at its answer, which also makes it easier to find the step where the reasoning went wrong. The authors add that how the model actually computes internally remains an open question.
  3. In principle, it applies to any task that humans can solve through language.
  4. Any off-the-shelf large model can use it; you just add chains of thought to the exemplars in the prompt.

Experimental setup

The authors tested three kinds of reasoning tasks:

  • Arithmetic: five math word problem datasets, GSM8K, SVAMP, ASDiv, AQuA and MAWPS.
  • Commonsense: CSQA, StrategyQA, date understanding and sports understanding from BIG-bench, and SayCan, which turns natural-language instructions into sequences of robot actions.
  • Symbolic: concatenating the last letter of each word in a name, and tracking whether a coin is still heads up after several people either flip it or don't.

There are five groups of models: four GPT-3 models from the OpenAI API (text-ada-001 through text-davinci-002, which the authors guess correspond to InstructGPT models of 350 million to 175 billion parameters; the paper says "presumably" because OpenAI never disclosed this), LaMDA (422 million to 137 billion), PaLM (8 billion, 62 billion and 540 billion), UL2 20B, and Codex. All results use greedy decoding. LaMDA results are averaged over 5 exemplar orders; for cost reasons, the other models were run with a single order.

Discussion

Arithmetic: it only works at scale

Original Figure 4. Solve rate against model scale for three model families, LaMDA, GPT and PaLM, on three datasets, GSM8K, SVAMP and MAWPS. The black line is standard prompting, the blue line is chain-of-thought prompting, and the orange dashed line is the best result from prior supervised methods. On small models the two lines nearly overlap, and the blue line is sometimes even lower; at the largest models, the blue chain-of-thought line pulls clearly ahead, and PaLM 540B rises above the orange dashed line on GSM8K.
Original Figure 4, Wei et al., from PDF page 5, CC BY 4.0.

The horizontal axis is parameter count. The three columns are LaMDA, GPT and PaLM, and the three rows are the three datasets. The black line is standard prompting, the blue line is chain-of-thought prompting, and the orange dashed line is the best prior result obtained with fine-tuning.

On the small models at the far left, the two lines sit on top of each other, and the blue line is sometimes lower. When the authors looked into it, they found that small models can write fluent chains of thought, but the logic is wrong, which ends up dragging the answer down. Chain of thought only starts to pull ahead at around 100 billion parameters. The authors call it an ability that "emerges" with model scale.

The second observation is that the harder the problem, the bigger the gain. GSM8K is the dataset with the lowest baseline scores, and with chain of thought the largest GPT and PaLM models more than doubled their scores. On SingleOp, the subset of MAWPS that can be solved in a single step, the gain is tiny or even negative.

ModelStandard promptingChain-of-thought promptingChain of thought + external calculator
LaMDA 137B6.514.317.8
GPT-3 175B (text-davinci-002)15.646.949.6
Codex (code-davinci-002)19.763.165.4
PaLM 540B17.956.958.6

GSM8K solve rate (%), compiled from Table 1 in the paper's appendix.

With only 8 exemplars, PaLM 540B goes from 17.9% to 56.9% on GSM8K, beating the previous best of 55%. That earlier result came from Cobbe et al. fine-tuning GPT-3 175B on the GSM8K training set and adding a verifier to pick the answer.

The last column is a small after-the-fact patch. Some chains of thought had correct reasoning and correct equations but got the arithmetic wrong. So the authors used Python's eval to recompute every equation in the chain of thought, then substituted each result into the following equations by string matching. Most models picked up a few more points this way.

Right answer, right reasoning?

The authors manually inspected LaMDA 137B's chains of thought on GSM8K. Of the 50 questions it answered correctly, only 2 had wrong reasoning that happened to land on the right answer; in the other 48, both the reasoning and the arithmetic were correct.

Of the 50 questions it got wrong, 46% were near misses, such as a calculation error, a symbol mapping error or a missing step; the other 54% had major errors in semantic understanding or coherence. The authors also compared the errors made by PaLM 62B and 540B. Going from 62B to 540B fixed a large share of the "missing step" and "misunderstood the question" errors.

Ablations: where the gain comes from

Original Figure 5. Ablations of LaMDA 137B and PaLM 540B on GSM8K. The five bars are, in order, standard prompting, equation only, variable compute only (outputting a string of dots), reasoning after the answer, and chain-of-thought prompting. For LaMDA, the first four are about 5 to 7 and chain of thought is about 14; for PaLM, the first four are about 18 to 22 and chain of thought is about 57.
Original Figure 5, Wei et al., from PDF page 6, CC BY 4.0.

There are several possible explanations for why chain of thought works. The authors designed one variant to test each:

  1. Equation only. Maybe the key is just getting the model to write out the equation it needs to compute? On GSM8K this barely helps; LaMDA goes from 6.5 to 5.4. The authors think GSM8K questions are too complex to translate directly into equations without going through reasoning steps in natural language. On datasets that can be solved in one or two steps, though, writing only the equation does help; SVAMP, for example, rises from 29.5 to 35.1.
  2. Variable compute only. Maybe the key is getting the model to produce more tokens, which amounts to computing for a few more steps? The authors had the model first output a string of dots, as many dots as there are characters in the equation. The result is about the same as standard prompting (6.4).
  3. Reasoning after the answer. Maybe a chain of thought just helps the model call up knowledge it picked up during pretraining? The authors moved the reasoning to after the answer, so that the answer could not depend on it. This result is also about the same as standard prompting (6.1).

None of the three variants comes close to chain of thought's 14.3. The authors conclude that writing out the intermediate steps in order, in natural language, and writing them before the answer are both necessary.

How much the wording of the exemplars matters

Prompting methods are often criticized for being sensitive to the exemplars. The authors asked two other co-authors to independently write chains of thought for the same 8 questions. The person who wrote the original chains also wrote a deliberately concise version in the style of the GSM8K solutions. On top of that, they randomly drew three sets of 8 questions from the GSM8K training set and used the solutions written by the dataset's crowdworkers as is.

On GSM8K with LaMDA 137B, these versions scored between 11.1 and 17.6. They do differ from one another, but all of them are clearly above standard prompting's 6.5. The effect of chain of thought does not hinge on one particular author's writing. The authors add that the results are just as robust when the order and number of exemplars change.

Commonsense and symbolic reasoning

In commonsense reasoning, both standard prompting and chain of thought get better as models grow, and the gain from chain of thought is largest on PaLM 540B. StrategyQA requires the model to work out a multi-step reasoning strategy on its own; PaLM 540B with chain of thought reaches 75.6%, above the previous best of 69.4%. Sports understanding asks whether a sports-related sentence is plausible, for example a soccer player catching a pass in an American football game. PaLM 540B reaches 95.4%, higher than the 84% of an unaided sports enthusiast. CSQA, on the other hand, shows almost no gain.

The two symbolic reasoning tasks are very easy for humans. The exemplars only demonstrate two-word names, or two people flipping a coin. At test time, besides questions of the same length, there are longer ones, such as names of three or four words.

With chain of thought, LaMDA 137B goes from 5.8% to 77.5% on letter concatenation and from 49.0% to 99.6% on the coin task. PaLM 540B gets close to 100% on questions of the same length. The authors also admit that the structure of the solution for these questions is already fully written out in the exemplars, and the model only has to apply the same steps to new symbols; even so, small models still cannot do it. On questions longer than the exemplars, standard prompting fails completely, while chain of thought improves as the model grows, though its scores stay below those for questions of the same length.

Limitations the authors list

The authors make four points in their discussion. First, a chain of thought imitates the human thought process, but that leaves open whether the neural network is actually "reasoning" (people are still arguing about that one). Second, writing a few exemplars by hand for few-shot prompting is cheap, but annotation becomes very expensive if you want chains of thought for fine-tuning. Third, the reasoning is not guaranteed to be correct; a wrong process can lead to a right answer, or to a wrong one. Fourth, it only works with large models, which are very expensive to deploy.

What came next

Within a few months, several follow-up papers built directly on this one.

In Large Language Models are Zero-Shot Reasoners, Kojima et al. at the University of Tokyo and Google found that you don't even need to write exemplars. Just add the sentence "Let's think step by step" before the answer, and text-davinci-002 rises from 10.4% to 40.7% on GSM8K.

In Self-Consistency, Wang et al. from the same team replaced greedy decoding with repeated sampling, each sample producing a different chain of thought, and then voted on the final answer. GSM8K rose by another 17.9 percentage points. The chain-of-thought paper itself mentions this approach where it describes its decoding.

The second advantage, "seeing how the model arrived at its answer," has been put to the test. In their 2023 paper Language Models Don't Always Say What They Think, Turpin et al. at NYU and Anthropic reordered the answer options in the few-shot exemplars so that the correct answer was always (A). The models then leaned toward choosing (A), and the chains of thought they wrote never mentioned this; instead, they made up plausible reasons for wrong answers. Across 13 BIG-Bench Hard tasks, this lowered the accuracy of GPT-3.5 and Claude 1.0 by as much as 36%. A chain of thought can read as perfectly sensible and still not match the cues the model actually relied on.

Later still, reasoning models simply turned "write the reasoning first, then give the answer" into a training objective, using reinforcement learning so that models teach themselves to write very long reasoning.

Conclusion

Add a few lines of working before the exemplar answers, and PaLM 540B jumps from 17.9% to 56.9% on GSM8K. Three ablations rule out the other explanations one by one. Only reasoning written in natural language, and written before the answer, makes the difference.

When you use it, first check whether the model is big enough, then whether the problem needs several steps; for a problem that takes one step, writing out the chain barely changes anything. And don't take the reasoning it writes too seriously, either. It may just be an explanation that sounds very convincing.

Back then you had to write eight exemplars by hand. Four months later, one line, "Let's think step by step," was enough. Two or three years after that, models were already thinking at length on their own.