DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Publication
- Publication venue not yet verified
- First submitted to arXiv
- arXiv:2402.03300
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{deepseekdeepseekmath2024,
title = {{DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models}},
author = {Zhihong Shao and Peiyi Wang and Qihao Zhu and Runxin Xu and Junxiao Song and Xiao Bi and Haowei Zhang and Mingchuan Zhang and Y. K. Li and Y. Wu and Daya Guo},
year = {2024},
eprint = {2402.03300},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2402.03300}
}
In early 2024, GPT-4 and Gemini Ultra had already passed 50% on MATH, the competition-math benchmark, while open-source models were still far behind. What data these closed models used and how they were trained, no one outside knew.
Shao, Wang, Zhu and colleagues at DeepSeek wanted to build an open-source math model with just 7B parameters. They did two things: they pulled 120 billion tokens of math data out of public Common Crawl web pages, and they proposed a reinforcement learning algorithm that uses fewer resources than PPO, called GRPO.
In the end, DeepSeekMath 7B, with no external tools and no voting, reaches 51.7% on MATH, close to Gemini Ultra and GPT-4; with 64 samples and a vote, it reaches 60.9%.

Defining the problem
This paper sets out to answer two questions.
First, where does good math data come from? Existing math corpora mostly come from arXiv papers or from modest collections of web pages. Is there really enough math content, of good enough quality, on the public web?
Second, how much further can reinforcement learning push after supervised fine-tuning? PPO was the most common approach at the time, but it has to train an extra value model as large as the model itself, which is expensive.
Solving the problem
1. Pulling math pages out of Common Crawl

The pipeline is a loop:
- Train a classifier: take OpenWebMath, a collection of high-quality math web pages, as the seed, randomly pick 500,000 entries from it as positives and 500,000 random Common Crawl pages as negatives, and train a fastText classifier.
- Recall pages: first deduplicate Common Crawl, which leaves 40 billion HTML pages. Score them with the classifier and keep only the highest-scoring portion. The first round keeps the top 40 billion tokens.
- Identify math domains: the first-round classifier has only seen OpenWebMath, so its positives aren't diverse enough, and it misses many math pages. The authors group pages by domain; if more than 10% of a domain's pages have been collected, the domain is treated as math-related, mathoverflow.net for example.
- Annotate URLs by hand: within these domains, people mark which URL paths hold math content, such as mathoverflow.net/questions. Pages under these paths that haven't been collected yet are added back into the seed data, and the classifier is retrained.
After four rounds, this yields 35.5 million web pages and 120 billion tokens. Nearly 98% of the data collected in the fourth round had already been collected in the third, so they stopped there.
To keep benchmark problems out of the training data, any text segment containing 10 consecutive words that exactly match a substring of a benchmark such as GSM8K, MATH, CMATH or AGIEval is removed entirely; for benchmark texts shorter than 10 words but at least 3 words long, exact matching is used.
Testing data quality with a small model
The authors trained the 1.3B DeepSeek-LLM for 150 billion tokens on each math corpus separately and compared the results:
| Math corpus | Size | GSM8K | MATH | CMATH |
|---|---|---|---|---|
| No math training | - | 2.9 | 3.0 | 12.3 |
| MathPile | 8.9B | 2.7 | 3.3 | 1.2 |
| OpenWebMath | 13.6B | 11.5 | 8.9 | 16.8 |
| Proof-Pile-2 | 51.9B | 14.3 | 11.2 | 19.9 |
| DeepSeekMath Corpus | 120.2B | 23.8 | 13.6 | 41.5 |
Compiled from Table 1 of the paper, selected columns; few-shot chain of thought, in %.
The DeepSeekMath Corpus leads on every measure. It also includes Chinese math pages, so it gains the most on CMATH, the Chinese benchmark; English-only corpora barely help on the Chinese benchmark, and MathPile even drops CMATH from 12.3% to 1.2%.
Its other advantage is size. The other corpora are smaller, so training on 150 billion tokens means going over them several times, and the model's performance saturates quickly.
DeepSeekMath-Base 7B
The base model starts from DeepSeek-Coder-Base-v1.5 7B and continues training for 500 billion tokens. The data mix is 56% DeepSeekMath Corpus, 4% AlgebraicStack (math code), 10% arXiv, 20% GitHub code, and 10% general web pages in Chinese and English.
| Model | Size | GSM8K | MATH |
|---|---|---|---|
| Minerva | 540B | 58.8 | 33.6 |
| Mistral | 7B | 40.3 | 14.3 |
| Llemma | 34B | 54.0 | 25.3 |
| DeepSeekMath-Base | 7B | 64.2 | 36.2 |
Compiled from Table 2 of the paper, selected rows and columns; few-shot chain of thought, in %.
On MATH, DeepSeekMath-Base is more than 10 percentage points above other open-source base models, and it also beats Google's Minerva 540B. Minerva is built on PaLM with extra training on math data, and has 77 times as many parameters as DeepSeekMath.
Math training also improves general abilities. Compared with its starting point, DeepSeek-Coder-Base-v1.5, MMLU rises from 42.9% to 54.9% and BBH from 42.9% to 59.5%; because the training data still includes code, coding ability is largely preserved.
Code first, then math
"Learning to code improves reasoning" is a claim you hear often but that had never been properly tested. The authors ran a controlled experiment with the 1.3B model:
| Training | GSM8K | MATH | GSM8K (with Python) | MATH (with Python) |
|---|---|---|---|---|
| General text 400B → math 150B | 19.1 | 14.4 | 14.3 | 6.7 |
| Code 400B → math 150B | 21.9 | 15.3 | 17.4 | 9.4 |
| Math only 150B | 20.5 | 13.1 | 11.4 | 6.5 |
| Code 400B and math 150B mixed together | 17.6 | 12.1 | 19.7 | 13.5 |
Compiled from Table 6 of the paper, selected rows and columns; in %.
Learning code before math beats learning general text first, whether or not the problems are solved with Python. This is also why the authors chose to start training from a code model.
Training on code and math mixed together gives the best Python problem solving, and the model is less likely to forget how to code; but math reasoning without tools actually gets worse. The authors guess that the 1.3B model is too small to absorb both kinds of data at once.
arXiv papers don't help: math corpora often include arXiv papers, but the authors found that training on arXiv alone, whether with the 1.3B or the 7B model, brings no clear improvement on any math benchmark, and sometimes makes things worse. The authors also point out the limits of this conclusion: these benchmarks may simply fail to measure the benefit, and the outcome might differ when arXiv is mixed with other data or when the model is larger.
Supervised fine-tuning
The math instruction data has 776,000 examples, covering Chinese and English problems from grade school to university level. Solutions come in three formats: chain of thought, program of thought (writing a Python program to solve the problem), and a format that alternates natural-language reasoning with a code tool. Training for 500 steps with batch size 256 and learning rate gives DeepSeekMath-Instruct 7B, which scores 46.8% on MATH without tools and 57.4% with tools.
2. GRPO: removing the value model

PPO's objective is:
Here the advantage is computed with GAE, which requires a learned value function. This value model is usually as large as the policy model, doubling both memory and compute. On top of that, a language model's reward usually arrives only at the last token, and training a value function that is accurate at every token is hard in itself.
GRPO's idea is straightforward. For the same question, have the model write answers and use the group's average reward as the baseline, encouraging answers better than average and penalizing those worse than average, with no value model needed:
Here is the ratio of the new policy's probability to the old policy's at token of answer .
Under outcome supervision, the reward model gives each answer one score, which is then normalized by the group's mean and standard deviation, and every token in that answer uses the same value as its advantage:
Let's work it out with a reward of 1 for a correct answer and 0 for a wrong one:
import statistics
def group_advantages(rewards, eps=1e-8):
"""GRPO outcome supervision: normalize the G rewards for one question by the group's mean and std"""
mean = statistics.mean(rewards)
std = statistics.pstdev(rewards) # population standard deviation here
return [round((r - mean) / (std + eps), 2) for r in rewards]
# Sample 8 answers to the same question: 1 point if correct, 0 if wrong
for rewards in ([1, 0, 0, 1, 0, 0, 0, 0], [1, 1, 1, 1, 1, 1, 1, 0], [1] * 8):
print(rewards, "→", group_advantages(rewards))
# [1, 0, 0, 1, 0, 0, 0, 0] → [1.73, -0.58, -0.58, 1.73, -0.58, -0.58, -0.58, -0.58]
# [1, 1, 1, 1, 1, 1, 1, 0] → [0.38, 0.38, 0.38, 0.38, 0.38, 0.38, 0.38, -2.65]
# [1, 1, 1, 1, 1, 1, 1, 1] → [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
On a hard question, the few correct answers get a large positive advantage; on an easy question, the only wrong answer is penalized heavily. On a question where every answer is right (or every answer is wrong), every answer's advantage is 0, so the question contributes nothing to training, and the compute spent sampling it is wasted.
Under process supervision, a process reward model scores every step of every answer, and the scores are likewise normalized by the mean and standard deviation of all the step scores. A token's advantage is the sum of the normalized scores of all the steps from there on.
KL is handled differently, too. PPO adds the KL penalty into each token's reward; GRPO adds the KL directly to the loss, so it doesn't interfere with the advantage computation. For the KL, it uses the estimator below, which is unbiased and never goes below 0:
Compared with simply taking the log ratio of the two probabilities:
import math
def kl_estimate(p_theta, p_ref):
"""Eq. (4) in the paper: π_ref/π_θ − log(π_ref/π_θ) − 1, never below 0"""
ratio = p_ref / p_theta
return ratio - math.log(ratio) - 1
for p_theta, p_ref in [(0.5, 0.5), (0.6, 0.5), (0.4, 0.5), (0.9, 0.3)]:
print(f"π_θ={p_theta} π_ref={p_ref} log(π_θ/π_ref)={math.log(p_theta / p_ref):+.3f} Eq.(4)={kl_estimate(p_theta, p_ref):.3f}")
# π_θ=0.5 π_ref=0.5 log(π_θ/π_ref)=+0.000 Eq.(4)=0.000
# π_θ=0.6 π_ref=0.5 log(π_θ/π_ref)=+0.182 Eq.(4)=0.016
# π_θ=0.4 π_ref=0.5 log(π_θ/π_ref)=-0.223 Eq.(4)=0.027
# π_θ=0.9 π_ref=0.3 log(π_θ/π_ref)=+1.099 Eq.(4)=0.432
The log ratio can be positive or negative, so the estimate for a single token can come out negative; Eq. (4) is 0 when the two distributions are the same and positive otherwise, growing as the gap widens.
Iterative GRPO: as training goes on, the old reward model may fall behind the policy model. The authors use fresh samples from the policy model to produce new training data for the reward model, continue training the reward model on it together with 10% of the old data, then replace the reference model with the current policy model and keep training.
Training setup
- Data: only the roughly 144,000 chain-of-thought-format questions in the SFT data that relate to GSM8K and MATH. Other questions are deliberately left out, to see how reinforcement learning affects benchmarks it wasn't trained on.
- Reward model: training data built the Math-Shepherd way, trained from DeepSeekMath-Base 7B with learning rate .
- Policy model: learning rate , KL coefficient 0.04, 64 answers sampled per question, maximum length 1,024 tokens, batch size 1,024; only one update after each round of sampling.
Discussion
Results
| Model | GSM8K | MATH | MGSM-zh | CMATH |
|---|---|---|---|---|
| Gemini Ultra | 94.4 | 53.2 | - | - |
| GPT-4 | 92.0 | 52.9 | - | 86.0 |
| DeepSeek-LLM-Chat 67B | 84.1 | 32.6 | 74.0 | 80.3 |
| Instruct 7B | 82.9 | 46.8 | 73.2 | 84.6 |
| RL 7B | 88.2 | 51.7 | 79.6 | 88.8 |
Compiled from Table 5 of the paper, the chain-of-thought, no-tool part, selected rows; Instruct 7B and RL 7B are DeepSeekMath's two 7B models; in %.
GRPO lifts GSM8K from 82.9% to 88.2% and MATH from 46.8% to 51.7%, beating every open-source model from 7B to 70B and most closed models too, though still slightly below GPT-4 and Gemini Ultra.
Reinforcement learning used only English GSM8K and MATH questions, yet the Chinese MGSM-zh and CMATH improved along with them. With tools allowed, MATH rises from 57.4% to 58.8%, also an improvement, even though the reinforcement learning data was all in chain-of-thought format.
A unified view
The authors write the various fine-tuning methods in one form. Their gradients all look like this:
They differ in only three places: the data source (where the questions and answers come from), the reward function (where the signal comes from), and the gradient coefficient (how much each token should be encouraged or penalized).
- SFT: data prepared by people, with a coefficient of 1 for every token.
- Rejection sampling fine-tuning (RFT): sample from the SFT model itself and keep only the correct answers; the coefficient is 1 for correct and 0 for wrong. This is the same thing as STaR's filtering.
- Online RFT: the same rule, but sampling from the model being trained instead.
- DPO: sample pairs of answers from the SFT model and train with a pairwise loss.
- PPO and GRPO: sample from the model being trained, with coefficients set by the reward, which can be positive or negative and vary in size.
Comparing these methods on the 1.3B Instruct model, the authors make a few observations:
- Online sampling beats offline: online RFT starts out about the same as RFT, but pulls clearly ahead later in training, because as the model drifts further from the SFT model, only its own samples keep up with it.
- Penalizing wrong answers beats only encouraging right ones: GRPO beats online RFT. Online RFT doesn't penalize wrong answers and treats all correct answers the same; GRPO adjusts how hard it encourages or penalizes according to the size of the reward.
- Process supervision beats outcome supervision: GRPO with process supervision beats GRPO with outcome supervision.
- Iteration helps: this one was run on the 7B Instruct model instead; iterative reinforcement learning clearly improves performance, especially in the first round.
What reinforcement learning actually changes

This is the figure in the paper most worth a close look. Maj@K asks whether the answer chosen by voting over samples is right; Pass@K counts a question as right if any one of samples is right.
Reinforcement learning raises Maj@K but not Pass@K. With enough samples, the original Instruct model gets just as many questions right. The authors' reading is that reinforcement learning makes the output distribution more robust, pushing correct answers that were already among the top few to the very top; it doesn't teach the model to solve questions it couldn't solve at all before.
At the end, the authors also note that they used only questions from the SFT stage and the simplest nucleus sampling, which may be why only Maj@K improved. In the future they want to try out-of-distribution questions and more advanced sampling methods such as tree search.
Limitations
- Weak on geometry and proofs: the model can't handle problems involving triangles and ellipses, which the authors think may reflect bias in how the pretraining and fine-tuning data were selected.
- Few-shot ability falls short of GPT-4: GPT-4 improves when given examples, while DeepSeekMath does about the same with or without them, which the authors attribute to model scale.
- Unreliable reward signals: existing methods trust the reward model's signal completely, yet even PRM800K, carefully labeled by people, turned out to have about 20% wrong labels. The authors see a need for reinforcement learning algorithms that can withstand noisy rewards.
What came next
Less than a year later, DeepSeek-R1 used GRPO, but replaced the reward with simple rules: is the answer right, is the format right. With no neural reward model, there is no reward model to exploit either. DeepSeek-R1-Zero even skipped supervised fine-tuning and started reinforcement learning straight from the base model. GRPO then became one of the most widely used reinforcement learning algorithms for open-source reasoning models.
GRPO itself has also been examined closely. In March 2025, Dr. GRPO pointed out that GRPO's objective has an optimization bias that artificially lengthens answers during training, most noticeably wrong ones, and proposed an unbiased fix.
The observation in Figure 7 was also put under the magnifying glass a year later. In April 2025, Yue et al., in Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?, compared models before and after reinforcement learning at much larger : at small , the model after reinforcement learning is better; at large , the base model's pass@k is actually higher. Their conclusion is that the reasoning ability gained from this kind of reinforcement learning comes from the base model and is also bounded by it. It's the same point this paper's own figure made back in 2024.
Conclusion
With 7B parameters, DeepSeekMath reaches 51.7% on MATH without tools or voting, close to GPT-4 at the time. The credit splits in two: half goes to the 120 billion tokens of math web pages pulled iteratively from Common Crawl, and half to GRPO, which removes the value model and has a group of answers to the same question compete against each other.
The analysis in this paper is often overlooked, but it carries real weight. Code first is better, arXiv doesn't help much, online sampling beats offline, and penalizing wrong answers works better than only rewarding right ones; later work kept coming back to all of these lessons.
The most honest part is Figure 7. The authors' own plot shows reinforcement learning raising only the voting accuracy, not the fraction of questions answered correctly at least once, and they wrote plainly that the progress may come from pushing correct answers to the front rather than from stronger ability. A year later, the whole field spent quite a bit of effort confirming this all over again.