Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
- Publication
- ICLR 2025Conference
- First submitted to arXiv
- arXiv:2408.03314
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{transformerstesttimecompute2024,
title = {{Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters}},
author = {Charlie Snell and Jaehoon Lee and Kelvin Xu and Aviral Kumar},
year = {2024},
eprint = {2408.03314},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2408.03314}
}
When people run into a hard problem, they think a bit longer. Can a language model do the same, spending a little more computation before it answers to make the answer better?
In mid-2024, the evidence on this was a mess. Approaches like majority vote and best-of-N clearly worked; but other studies found that on complex tasks such as mathematical reasoning, having the model reflect on and revise its own answer helped very little, or even made things worse. Whether the extra computation spent at inference time was worth it, and how it should be spent, no one had compared systematically.
Snell of UC Berkeley, during an internship at Google DeepMind, took this on with Lee, Xu and Kumar. They wanted to answer two questions: given a compute budget at inference time, what's the most cost-effective way to spend it? And rather than making the model bigger, would it be better to spend the same computation at inference time?
Defining the problem
Proposer and verifier
The authors sort the various inference-time approaches into two directions that can be tuned:
- Changing the proposal distribution: get the model to produce better answers. One way is to train the model with methods like STaR and ReSTEM; the other is to have the model look at its own earlier answers at inference time and revise them again and again.
- Selecting with a verifier: produce many answers and pick among them with a verifier. The most basic form is best-of-N, which samples N complete answers and picks the highest-scoring one; going further, a process reward model (PRM) scores every step, and a tree search runs during the reasoning itself.
What "most cost-effective" means
The authors define the "compute-optimal strategy for test-time compute" as choosing, for a question and a compute budget , the settings that maximize the probability of getting the answer right:
Here can be whether to use search, which search to use, how many revisions to make, and how many samples to draw in parallel.
Optimizing every question separately isn't practical, so the authors approximate it with question difficulty. Difficulty is defined relative to the model itself: sample each question 2,048 times, compute the single-attempt success rate, and split all the questions into five equal bins from easy to hard. The best strategy for each difficulty level is found on a validation set and then applied to the test questions.
Defining difficulty this way requires knowing the correct answer, so the authors call it oracle difficulty. In real deployment there's no answer, so there is also a "model-predicted difficulty," which instead bins questions by the verifier's average score over the same 2,048 answers. The authors also admit that estimating difficulty costs a fair amount of computation in itself, and this cost isn't counted in the experiments.
Experimental setup
- Data: MATH, with the split of 12,000 training and 500 test questions made by Lightman et al.
- Model: PaLM 2-S* (Codey). It does reasonably on MATH without saturating it, which makes it a good subject for study.
- Fine-tuning: directly prompting an off-the-shelf model to revise answers or verify steps doesn't work well, so both abilities are fine-tuned separately. The authors expect future models to come out of pretraining with these abilities.
Solving the problem
1. Searching with a PRM
Training the PRM: the authors started with PRM800K, released by Lightman et al., but found that a PRM trained on it was easy to exploit; even the simplest best-of-N could find answers that fooled it. They suspect this is because the solutions in PRM800K were all written by GPT-4 and differ too much from PaLM 2's output.
So they switched to the Math-Shepherd approach, which needs no human labels: from each step of a solution, have the model finish it several times, and use the fraction of rollouts that reach the correct answer as the step's score. Each step prediction of a PRM trained this way is really "the probability of getting the answer right from here on." Compared with an ORM, which looks only at the final answer, the PRM is consistently better.
Aggregating scores: for a solution's overall score, Lightman et al. used the product of the step scores; here the PRM's prediction at the last step is used instead, which the authors found works best. Across multiple solutions, "weighted best-of-N" is used: the scores of solutions with the same final answer are summed, and the answer with the highest total is chosen.
Looking only at the single highest-scoring solution can get you fooled by one wrong answer with an inflated score; weighting considers both the vote count and the scores.

Three search methods:
- Weighted best-of-N: sample N solutions independently and pick with the weighting described above.
- Beam search: keep N candidates with beam width M. At each step, first generate N candidate steps, score them with the PRM, keep the best N/M, expand each of those into M more, and repeat until the solution ends, for at most 40 rounds.
- Lookahead search: like beam search, but when evaluating each step, first roll out up to k steps ahead at temperature 0 and use the PRM score at the end of the rollout to represent the step. This makes the value estimate more accurate at the cost of more computation, counted as samples. The authors point out that this can be seen as MCTS with the random exploration removed; the PRM is already trained, so at inference time you want to exploit rather than explore.
2. Having the model revise its own answers

Simply asking an off-the-shelf model to correct its own mistakes mostly doesn't work on reasoning problems, so the authors fine-tuned a model specifically for revising answers.
Training data: the ideal training data is a sequence of wrong answers followed by a correct one, with the correct answer ideally revised from those mistakes rather than rewritten from scratch. Generating multi-turn conversations turn by turn is too expensive, so the authors instead sample 64 answers in parallel at once and stitch them into multi-turn data afterward: each correct answer is preceded by 0 to 4 (chosen at random) wrong answers. When choosing wrong answers, they use character edit distance and prefer those closer to the correct answer, so that what the model learns is revising rather than rewriting.
Inference: the model was only trained with up to 4 previous answers in front of it, but at inference time it can keep revising, keeping only the most recent 4 as context. The more revisions it makes, the higher its single-attempt success rate climbs, and it is still improving beyond 4 revisions.
There's a side effect, though. In training, the previous answers were all wrong; at inference time, there may already be a correct answer among them, and the model "revises" it anyway. About 38% of correct answers turn wrong after the next revision. So the final answer can't simply be the last one in the chain; majority vote or a verifier has to pick one out of the whole revision chain.
Discussion
Search: useful on small budgets, harmful when questions are too easy

At small budgets, beam search clearly beats best-of-N; once the budget is large, the advantage disappears, and beam search often loses to best-of-N. Lookahead search is mostly worse at the same budget.
The authors attribute this to the PRM being exploited. The stronger the search, the easier it is to find solutions the PRM scores highly but that are actually useless, such as ones that repeat some uninformative steps at the end, or ones only a step or two long. The strongest search method ends up performing worst.
Broken down by difficulty, the trends are completely different:
- Easy questions: the model readily writes correct answers on its own, and the verifier mostly judges them correctly. Optimizing hard on top of that with beam search only amplifies the spurious features the verifier has learned, and more budget makes things worse.
- Medium questions: the model has a harder time getting it right in one go, so search can guide it to the correct answer, and beam search consistently beats best-of-N.
- The hardest questions: no method makes clear progress.
Choosing the best search method by difficulty, at smaller budgets, gets close to best-of-N's 64-sample results with only 16 samples, a nearly 4-fold saving in compute.
Revision: keep revising on easy questions, open more paths on hard ones
At the same number of samples, sequential revision slightly beats parallel sampling. But the two are good at different things: parallel sampling is more like a global search that can try completely different approaches, while sequential revision is more like a local fix, suited to answers already roughly heading in the right direction.
So the authors give part of the budget to parallel sampling and part to sequential revision, sweeping across different ratios. At every budget there is a best ratio, and it depends on difficulty: for easy questions, putting everything into sequential revision is best; for hard questions, parallel and sequential need to be balanced.
Choosing the ratio by difficulty beats best-of-N's 256-sample results with 64 samples, again a 4-fold saving.
Can inference compute replace a bigger model?
That leaves the last question. With the same FLOPs, should you make the model bigger, or let a small model compute more at inference time?
The authors use the common approximations: pretraining compute is and inference is , where is the parameter count. Scale the model's parameters up by a factor of , and both pretraining and inference compute become times larger. If that computation goes instead to the small model's inference, its inference compute can be multiplied by:
How many times more it can compute depends on the ratio of inference tokens to pretraining tokens, . Comparing against a model about 14 times larger (), the authors try three cases, of 0.16, 0.79 and 22. When there's little inference (in a self-improvement training pipeline, for example), a lot of pretraining compute is saved, and the same FLOPs let the small model use about 258 times as much inference; when inference and pretraining are about equal, about 63 times; when inference far exceeds pretraining (for example, a large-scale production service), only about 16 times.

The conclusion is that the two can't be exchanged one for one. On easy and medium questions, or where inference demand is small, spending the compute on inference is often better than making the model 14 times bigger; on very hard questions, or where inference demand is large, making the model bigger is still more effective.
Limitations
- Estimating difficulty is expensive: each question has to be sampled 2,048 times before it can be binned, and this computation isn't counted in the experiments. The authors suggest having models predict question difficulty directly in the future, or switching dynamically between estimating difficulty and solving.
- No progress on the hardest questions: neither search nor revision helps at the hardest level.
- Methods aren't combined: revision is paired with a verifier, but not used together with PRM tree search.
- Tested in only one setting: only MATH and PaLM 2-S*, and both the revision and verification abilities had to be fine-tuned separately.
The authors also mention that answers obtained by spending more computation at inference time could later be distilled back into the model itself, forming a continual self-improvement loop.
What came next
One month after this paper went up on arXiv, on September 12, 2024, OpenAI released o1. o1 is trained with reinforcement learning to write a very long internal chain of thought before answering; OpenAI's post says that o1's performance improves steadily with more train-time compute (reinforcement learning) and more test-time compute (time spent thinking). The "sequential revision" this paper studied became, in o1, self-checking and correction within a single chain of thought, with no need to fine-tune a separate revision model.
DeepSeek-R1 took a similar road, and explicitly recorded that PRMs and MCTS didn't work out in their hands.
The idea of allocating inference compute by difficulty was pushed to more extreme lengths. Can 1B LLM Surpass 405B LLM?, from February 2025, reports that with the best inference-time strategy, a 1B model can beat a 405B model on MATH-500, and a 0.5B model can beat GPT-4o on MATH-500 and AIME24; they also found that the best strategy depends on the model used, the PRM and the difficulty of the questions.
As for getting a model to extend its own thinking within a single chain of thought with as little training as possible, see s1.
Conclusion
There is no one-size-fits-all way to spend test-time compute. On easy questions, let the model revise its own answers, since heavy search only amplifies the verifier's mistakes; on medium questions, search is most effective; on the hardest questions, none of the current methods help. Allocating by difficulty cuts compute by up to a factor of 4.
Compared with making the model 14 times bigger, spending the same computation on inference is often better for easy and medium questions, while hard questions still call for a bigger model. Inference compute and pretraining compute can't be exchanged one for one.
In their introduction, the authors write that this hints at a future with less spent on pretraining and more on inference. A month later o1 came out, and test-time compute became the thing everyone competed on, only along a more direct road than this paper imagined, with no bolted-on search or verifiers, just a model that has learned on its own to think a while longer.