Back to paper library
arXiv 2022ICLR 2023Language & TransformersPaper notes

Self-Consistency

Majority rules

Original paper

Self-Consistency Improves Chain of Thought Reasoning in Language Models

Paper authors

Xuezhi Wang · Jason Wei · Dale Schuurmans …

Show all 8 authors

Xuezhi Wang · Jason Wei · Dale Schuurmans · Quoc Le · Ed Chi · Sharan Narang · Aakanksha Chowdhery · Denny Zhou

Publication
ICLR 2023Conference
First submitted to arXiv
arXiv:2203.11171
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{transformersselfconsistency2022,
  title = {{Self-Consistency Improves Chain of Thought Reasoning in Language Models}},
  author = {Xuezhi Wang and Jason Wei and Dale Schuurmans and Quoc Le and Ed Chi and Sharan Narang and Aakanksha Chowdhery and Denny Zhou},
  year = {2022},
  eprint = {2203.11171},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2203.11171}
}
On this page

In January 2022, Wei et al. at Google published chain-of-thought prompting, which has the model write out its reasoning steps before giving an answer, and scores on math word problems rose sharply.

That paper's decoding, though, was conservative. Reasoning problems have a fixed answer, so people habitually used greedy decoding, picking the most probable token at every step and ending up with a single line of reasoning. If that reasoning gets one step wrong anywhere along the way, the answer is wrong, with no chance to recover.

Two months later, Wang et al. from the same team tried a different idea. Have the model work the same problem several times, taking a different route each time, and at the end see which answer comes up most often.

Defining the problem

When people solve a problem, several different lines of thought can often lead to the same answer. The authors' hypothesis is that complex reasoning problems work the same way. Correct reasoning, however differently it is written, ends up at the same answer; incorrect reasoning goes wrong in its own way each time, so its answers are more scattered.

If that hypothesis holds, there's no need to know which line of reasoning is right. You only have to see whether the answers "collide."

Before this, picking a good answer out of several candidates usually meant training a separate model. Cobbe et al. trained a verifier on GSM8K to rank solutions, and LaMDA collected extra human annotations to train a re-ranking model. This paper wants no training, no extra annotation and no additional model, just an off-the-shelf language model used as is.

Solving the problem

Original Figure 1. At the top is chain-of-thought prompting with greedy decoding: the model writes a single line of reasoning and gets the wrong answer $14. At the bottom is self-consistency, where the same prompt is used to sample several lines of reasoning instead: the first gets $18, the second makes a mistake and gets $26, and the third also gets $18. The reasoning is marginalized out and the final answers are put to a vote, which picks $18.
Original Figure 1, Wang et al., from PDF page 2, CC BY 4.0.

The method has just three steps:

  1. As before, use hand-written chain-of-thought exemplars as the prompt.
  2. Instead of greedy decoding, sample mm outputs from the model's decoder, each with its own reasoning and its own answer.
  3. Take the answer after "The answer is" at the end of each output, and pick the one that appears most often.

The example in the figure is a GSM8K question. Janet's ducks lay 16 eggs a day; she eats 3 for breakfast and uses 4 to bake muffins, and sells each of the rest for $2. How much does she make a day? Greedy decoding adds the eaten and used eggs to get 7, never subtracts them from the 16, and multiplies 7 by 2 to get $14. Of the three sampled lines of reasoning, two arrive at $18 by different routes, and one makes a mistake and gets $26. The vote comes out at 18.

Why it's called "marginalization"

The authors split the ii-th output into reasoning rir_i and answer aia_i. All we really care about is the answer; the reasoning is an intermediate product. So the various lines of reasoning are summed out, and we only look at how many votes each answer gets:

a^=arg⁡max⁡a∑i=1m1(ai=a)\hat a = \arg\max_a \sum_{i=1}^{m} \mathbb{1}(a_i = a)

That's a majority vote. The name sounds academic; in practice it's counting votes.

Weight by probability or not?

Since the model assigns a probability to every output, you could also give more votes to reasoning with higher probability. The authors tried several weighting schemes. "Normalized" here means dividing the log probability of the whole output by its number of tokens, so that long reasoning isn't penalized for multiplying together more probabilities:

P(ri,ai∣prompt,question)=exp⁡(1K∑k=1Klog⁡P(tk∣prompt,question,t1,…,tk−1))P(r_i, a_i \mid \text{prompt}, \text{question}) = \exp\left(\frac{1}{K}\sum_{k=1}^{K}\log P(t_k \mid \text{prompt}, \text{question}, t_1, \ldots, t_{k-1})\right)

Results on PaLM 540B:

Voting methodGSM8KMultiArithAQuASVAMP
Greedy decoding56.594.735.879.0
Weighted avg (unnormalized)56.390.535.873.0
Weighted avg (normalized)22.159.715.740.5
Weighted sum (unnormalized)59.992.238.276.2
Weighted sum (normalized)74.199.348.086.8
Unweighted (majority vote)74.499.348.386.6

Compiled from Table 1 of the paper, four arithmetic datasets only; values are in %.

The normalized weighted sum comes out almost the same as plain vote counting. When the authors looked closer, they found that after normalization the probabilities of the different lines of reasoning are all very close; the model regards them as "similarly likely." In other words, the model can't really tell which of its own lines of reasoning is right, which is also why earlier work had to train a separate ranking model. Since weighting doesn't help, just count the votes.

The "weighted average" divides each answer's weighted sum by its number of votes, which amounts to ignoring how many votes it got, and it does much worse.

The method has one precondition. The answers must come from a fixed set, so that you can tell whether two answers are the same. Taking the number from a math problem or the option from a multiple-choice question is no problem; open-ended generation won't work unless you separately define what it means for two pieces of text to "agree."

Experimental setup

  • Tasks: arithmetic reasoning on AddSub, MultiArith, ASDiv, AQuA, GSM8K and SVAMP; commonsense reasoning on CommonsenseQA, StrategyQA and ARC; symbolic reasoning on letter concatenation and coin flipping, where the exemplars only show two letters or two flips and the test asks for four.
  • Models: UL2 20B, GPT-3 175B (through code-davinci-001 and 002 from the Codex series), LaMDA 137B and PaLM 540B. Everything is prompting only, with no fine-tuning at all.
  • Prompts: the same as in the chain-of-thought paper; the arithmetic problems share one set of 8 hand-written exemplars.
  • Sampling: UL2 and LaMDA use temperature 0.5 with top-k 40; PaLM uses temperature 0.7 with top-k 40; GPT-3 uses temperature 0.7 with no truncation. Each run samples 40 outputs, and results are averaged over 10 runs.

Discussion

Every model improves

ModelGSM8KSVAMPAQuA
UL2 20B4.1 → 7.312.6 → 19.423.6 → 26.9
LaMDA 137B17.1 → 27.738.9 → 53.317.7 → 26.8
PaLM 540B56.5 → 74.479.0 → 86.635.8 → 48.3
code-davinci-00114.6 → 23.439.8 → 54.518.9 → 25.6
code-davinci-00260.1 → 78.075.8 → 86.839.8 → 52.0

Compiled from Table 2 of the paper. Left of the arrow is chain of thought with greedy decoding, right is self-consistency; values are in %.

All four models improve on every arithmetic task, and the bigger the model, the bigger the gain tends to be. UL2 20B gains 3 to 7 percentage points, while LaMDA 137B and code-davinci-001 mostly gain between 7 and 24. Even PaLM 540B and code-davinci-002, which were already strong, gain 17.9 percentage points on GSM8K. The 78.0% from code-davinci-002 beats the 55% that fine-tuning GPT-3 plus a verifier had reached at the time, and self-consistency involves no training at all.

Commonsense reasoning gains less, with code-davinci-002 up 6.4 percentage points on StrategyQA and 3.9 on ARC-challenge. Symbolic reasoning is tested on problems longer than the exemplars, and self-consistency still helps once the model is big enough. But UL2 20B was at 0.0% on letter concatenation, and after voting it's still 0.0%. When the model simply can't do a problem, more samples just collect more wrong answers.

More samples help, but the gain saturates quickly

Original Figure 2. Accuracy of LaMDA 137B on MultiArith, SVAMP, CommonsenseQA and ARC-challenge against the number of sampled lines of reasoning (1, 5, 10, 20, 40). The blue self-consistency line rises with the number of samples, fastest over the first 10, then levels off; the orange line is greedy decoding and stays flat. With only 1 sample, the blue line is below the orange one.
Original Figure 2, Wang et al., from PDF page 6, CC BY 4.0.

Look at the leftmost point in each plot. With only 1 sample, accuracy is worse than greedy decoding, because random sampling is more error-prone than picking the most likely token. A single sampled line of reasoning is of lower quality, and it takes numbers plus a vote to win that back.

By 5 to 10 samples the curves have already captured most of the gain, and in their conclusion the authors also suggest starting with 5 or 10 in practice.

Why voting works

The authors' hypothesis is that correct reasoning, even when each one is written differently, lands on the same answer, while incorrect reasoning goes wrong in its own way each time, with answers scattered over many different values. So if the correct answer gets just 40% of the votes and the other 60% is spread over several wrong values, it is still the largest group, and the more you sample, the steadier the majority vote.

Looked at the other way, this hypothesis also marks where voting stops working. If the model makes a systematic error, say it habitually leaves out one subtraction, the wrong answers pile up on the same value. With 40% correct and 80% of the remaining 60% making the same mistake, that wrong answer gets 48% of the votes, more than the correct answer, and the more you sample, the more confidently it picks the wrong one. Self-consistency wins because "each wrong answer goes wrong in its own way."

Compared with other approaches

Sample-and-rank: likewise sample several outputs, rank them by log probability, and take the answer from the top one. On code-davinci-001 this also improves as the number of samples grows, but by much less than voting, which matches the earlier observation that "the model can't tell which line of reasoning is better."

Beam search: with UL2 20B on AQuA, taking the top beam gets worse as the beam widens, dropping from 23.6% to 10.2% at 40 beams. Generating 40 lines of reasoning with beam search and voting gives 24.2%; sampling and voting gives 26.9%. Beam search produces candidates that are too similar, and voting needs diversity.

Ensembles: with LaMDA 137B on GSM8K, compared against two kinds of prompt ensemble:

MethodGSM8K
Chain of thought (greedy decoding)17.1
Three different hand-written prompts, each greedy-decoded, then a vote18.6
Exemplar order shuffled 40 times, each greedy-decoded, then a vote19.2
Self-consistency (40 samples)27.7

Compiled from Table 7 of the paper; values are in %.

Changing the prompt or the order makes far less difference than letting the model take a few more routes on its own. The appendix also tries having several models vote together. PaLM 540B plus LaMDA 137B gets only 36.9%, far worse than the 74.4% PaLM gets from self-consistency on its own; the weaker model drags the stronger one down.

Tasks where chain of thought makes things worse

Ye and Durrett found that on some ordinary NLP tasks, adding chain of thought actually hurts. The authors tested a few on PaLM 540B:

MethodANLI R1e-SNLIRTEBoolQ
Standard prompting (no reasoning)69.185.884.871.3
Chain-of-thought prompting68.881.079.174.2
Self-consistency78.588.486.378.4

Compiled from Table 5 of the paper, showing selected columns; values are in %.

On e-SNLI and RTE, adding chain of thought costs about 5 percentage points each. Self-consistency pulls them back up, and even beats standard prompting without reasoning.

Not fussy about settings

Original Figure 4. GSM8K accuracy. The left plot shows PaLM 540B under different sampling settings, with accuracy against the number of samples: temperature 0.3, 0.5 or 0.7, top-k 20, 40 or no truncation, and nucleus sampling with p of 0.9 or 0.95, all above the horizontal line for greedy decoding. The right plot shows accuracy against parameter count for LaMDA models of different sizes, with self-consistency and greedy decoding; the bigger the model, the clearer the gap.
Original Figure 4, Wang et al., from PDF page 8, CC BY 4.0.

In the left plot, every temperature, top-k and nucleus sampling setting ends up well above greedy decoding by 40 samples. The right plot shows that the gain on small models is limited. The authors' explanation is that abilities like arithmetic only emerge at a certain scale, and while the model still can't do the math, voting can't help either.

The authors also ran a few stress tests, all on GSM8K:

  • Flawed exemplars: randomly replace the numbers in the reasoning of the prompt exemplars (keeping only the final answer). LaMDA 137B with greedy decoding drops from 17.1% to 14.9%, and adding self-consistency brings it back to 23.4%.
  • Equations only: the reasoning only writes equations like "3 + 2 = 5". Self-consistency goes from 5.0% to 6.5%, a small gain. The equations are too short to vary much, so there isn't enough diversity.
  • Zero-shot chain of thought: no exemplars, just the sentence "Let's think step by step". PaLM 540B rises from 43.0% to 69.2%, a gain of 26.2 percentage points.
  • A different set of prompts: the appendix writes two more sets of exemplars, and PaLM 540B gains 17.5 and 16.4 percentage points with them, about the same as the 17.9 in the main experiment.

Agreement as confidence

Original Figure 5. Scatter plot of agreement against accuracy on GSM8K. The horizontal axis is the share of sampled lines of reasoning that agree with the final answer, and the vertical axis is accuracy. The higher the agreement, the higher the accuracy; for points with agreement of 80% or higher, accuracy is mostly 100%.
Original Figure 5, Wang et al., from PDF page 8, CC BY 4.0.

A vote gives you more than an answer; it also gives you the vote count. When 35 of 40 lines of reasoning agree, that's clearly more trustworthy than when only 12 do. The authors found that the share of agreement correlates strongly with accuracy, so a question with low agreement can be read as a sign that the model "isn't sure either."

Limitations

The authors themselves point out three limitations:

  • Cost: sampling 40 outputs means 40 times the inference, so they suggest trying 5 to 10 first.
  • Fixed answers only: you have to be able to tell whether two answers are the same.
  • The reasoning isn't guaranteed to be right: the vote can land on the right answer while the reasoning still talks nonsense. In the StrategyQA example from the paper's table, the model compares the populations of two Albanys to answer, and neither population figure is quite accurate.

The authors also propose a future direction, using self-consistency to produce better data for fine-tuning the model so that it gets the answer right in a single inference. That same month, Zelikman et al.'s STaR was doing something similar, training the model on the reasoning from questions it had answered correctly.

What came after

Majority voting quickly became standard equipment in reasoning evaluations. The maj@k and cons@64 you often see in paper tables mean sampling k or 64 outputs and then voting. DeepSeek-R1-Zero averages 71.0% on AIME 2024 with a single attempt, and 86.7% with a vote over 64.

The idea of "agreement as confidence" got used too. In the Gemini technical report of December 2023, Gemini Ultra reached 90.04% on MMLU using what they call uncertainty-routed chain-of-thought. It first samples 32 chains of thought; if agreement is above a threshold, it takes the majority-vote answer, and otherwise it falls back to greedy decoding without reasoning. Greedy decoding alone gives 84.0%, a vote over 32 alone gives 85.0%, and only the combination reaches 90.0%.

People have also tried to remove the fixed-answer restriction. In November 2023, Chen et al. from the same team proposed Universal Self-Consistency. Instead of matching answers itself, it hands all the candidate responses to a language model and asks it to pick the one most consistent with the others, so it can be used for generation without a standard answer, such as summaries and code.

Later still, the question became "how do you spend extra inference-time compute most effectively?" Voting is the simplest option; others include using a verifier to pick the answer and searching over each step of the reasoning.

Conclusion

Self-consistency needs no training and no additional model. Swap greedy decoding for 40 samples and a vote count, and PaLM 540B and code-davinci-002 each gain 17.9 percentage points on GSM8K, beating the fine-tuning-plus-verifier approach of the time.

It rests on the premise that incorrect reasoning goes wrong in its own way each time. When the model has a systematic blind spot, or simply can't do the problem, voting only picks the wrong answer more firmly, and the cost scales directly with the number of samples. Fortunately, most problems get most of the gain by 5 to 10 samples.

What really lasted from this paper may be that scatter plot of agreement against accuracy. The vote count is itself a confidence score. Less than two years later, Gemini Ultra pushed MMLU to 90% by first checking whether the votes were concentrated enough, then deciding whether to trust the result of the vote.