Back to paper library
arXiv 2025Language & TransformersPaper notes

Kimi k1.5

Long story short

Original paper

Kimi k1.5: Scaling Reinforcement Learning with LLMs

Paper authors

Kimi Team · Angang Du · Bofei Gao …

Show all 96 authors

Kimi Team · Angang Du · Bofei Gao · Bowei Xing · Changjiu Jiang · Cheng Chen · Cheng Li · Chenjun Xiao · Chenzhuang Du · Chonghua Liao · Chuning Tang · Congcong Wang · Dehao Zhang · Enming Yuan · Enzhe Lu · Fengxiang Tang · Flood Sung · Guangda Wei · Guokun Lai · Haiqing Guo · Han Zhu · Hao Ding · Hao Hu · Hao Yang · Hao Zhang · Haotian Yao · Haotian Zhao · Haoyu Lu · Haoze Li · Haozhen Yu · Hongcheng Gao · Huabin Zheng · Huan Yuan · Jia Chen · Jianhang Guo · Jianlin Su · Jianzhou Wang · Jie Zhao · Jin Zhang · Jingyuan Liu · Junjie Yan · Junyan Wu · Lidong Shi · Ling Ye · Longhui Yu · Mengnan Dong · Neo Zhang · Ningchen Ma · Qiwei Pan · Qucheng Gong · Shaowei Liu · Shengling Ma · Shupeng Wei · Sihan Cao · Siying Huang · Tao Jiang · Weihao Gao · Weimin Xiong · Weiran He · Weixiao Huang · Weixin Xu · Wenhao Wu · Wenyang He · Xianghui Wei · Xianqing Jia · Xingzhe Wu · Xinran Xu · Xinxing Zu · Xinyu Zhou · Xuehai Pan · Y. Charles · Yang Li · Yangyang Hu · Yangyang Liu · Yanru Chen · Yejie Wang · Yibo Liu · Yidao Qin · Yifeng Liu · Ying Yang · Yiping Bao · Yulun Du · Yuxin Wu · Yuzhi Wang · Zaida Zhou · Zhaoji Wang · Zhaowei Li · Zhen Zhu · Zheng Zhang · Zhexu Wang · Zhilin Yang · Zhiqi Huang · Zihao Huang · Ziyao Xu · Zonghan Yang · Zongyu Lin

Publication
Publication venue not yet verified
First submitted to arXiv
arXiv:2501.12599
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{transformerskimik152025,
  title = {{Kimi k1.5: Scaling Reinforcement Learning with LLMs}},
  author = {Kimi Team and Angang Du and Bofei Gao and Bowei Xing and Changjiu Jiang and Cheng Chen and Cheng Li and Chenjun Xiao and Chenzhuang Du and Chonghua Liao and Chuning Tang and Congcong Wang and Dehao Zhang and Enming Yuan and Enzhe Lu and Fengxiang Tang and Flood Sung and Guangda Wei and Guokun Lai and Haiqing Guo and Han Zhu and Hao Ding and Hao Hu and Hao Yang and Hao Zhang and Haotian Yao and Haotian Zhao and Haoyu Lu and Haoze Li and Haozhen Yu and Hongcheng Gao and Huabin Zheng and Huan Yuan and Jia Chen and Jianhang Guo and Jianlin Su and Jianzhou Wang and Jie Zhao and Jin Zhang and Jingyuan Liu and Junjie Yan and Junyan Wu and Lidong Shi and Ling Ye and Longhui Yu and Mengnan Dong and Neo Zhang and Ningchen Ma and Qiwei Pan and Qucheng Gong and Shaowei Liu and Shengling Ma and Shupeng Wei and Sihan Cao and Siying Huang and Tao Jiang and Weihao Gao and Weimin Xiong and Weiran He and Weixiao Huang and Weixin Xu and Wenhao Wu and Wenyang He and Xianghui Wei and Xianqing Jia and Xingzhe Wu and Xinran Xu and Xinxing Zu and Xinyu Zhou and Xuehai Pan and Y. Charles and Yang Li and Yangyang Hu and Yangyang Liu and Yanru Chen and Yejie Wang and Yibo Liu and Yidao Qin and Yifeng Liu and Ying Yang and Yiping Bao and Yulun Du and Yuxin Wu and Yuzhi Wang and Zaida Zhou and Zhaoji Wang and Zhaowei Li and Zhen Zhu and Zheng Zhang and Zhexu Wang and Zhilin Yang and Zhiqi Huang and Zihao Huang and Ziyao Xu and Zonghan Yang and Zongyu Lin},
  year = {2025},
  eprint = {2501.12599},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.12599}
}
On this page

On January 20, 2025, DeepSeek released R1. About an hour later, Moonshot AI released Kimi k1.5.

The two models set out to do very similar things. After OpenAI announced o1 in September 2024, everyone knew that "let the model think for a long time before answering" could greatly improve math and coding performance, but OpenAI did not say how. Both reports gave an answer, and the answers were close. Both train the model with reinforcement learning to write long chains of thought, with no Monte Carlo tree search, no value function, and no process reward model scoring each step.

The difference is that R1 released its weights and k1.5 did not; it is available only through Kimi's products and API. The report does not state the model's parameter count either. So what matters in k1.5 is not the model itself but how it was trained. Two parts of it have no counterpart in the R1 report: scaling the RL context length to 128K, and a whole section on carrying long-thinking ability back into short-answer models. One of the methods there is model merging.

Defining the problem

Scaling pretraining is limited by the total amount of high-quality data. The authors wanted another path that can keep scaling, letting the model explore on its own through rewards instead of only learning from a fixed dataset.

The problem setup is simple. Given a problem xx and a reference answer y∗y^*, the model first writes a reasoning process zz and then gives an answer yy. The reward r(x,y,y∗)r(x,y,y^*) is only 0 or 1. Coding problems are judged by whether the tests pass, math problems by whether the answer is right:

max⁡θ E(x,y∗)∼D, (y,z)∼πθ[r(x,y,y∗)]\max_\theta\ \mathbb{E}_{(x,y^*)\sim D,\ (y,z)\sim\pi_\theta}\big[r(x,y,y^*)\big]

The authors have an interesting view of "long thinking." Traditional inference-time search, such as tree search, needs an evaluator to judge which path looks more promising before deciding which node to expand. But if every attempt, evaluation, and backtrack in the search is written out as one long piece of text, then a model that can write at great length can search implicitly within its own context.

This view turns the problem into two questions: how long can the context get, and can reinforcement learning teach the model to make good use of that length?

Solving the problem

Pick good problems first

The authors found that the success of reinforcement learning depends to a large extent on the problem set. A good problem set covers many domains, has an even spread of difficulty, and has answers that can be verified accurately.

Difficulty is estimated by the model itself. The SFT model answers each problem 10 times at a relatively high sampling temperature; the lower the pass rate, the harder the problem. Along the way, this also filters out problems that are too easy.

Verification has to guard against lucky guesses. Some reasoning problems have answers that are easy to guess, and the model may reach the right answer through wrong reasoning, fooling the reward. The authors excluded multiple-choice, true/false, and proof problems. For general QA problems, the model first guesses directly 8 times without thinking; if any guess is right, the problem is judged too easy to guess and removed.

Warming up long thinking

Before the real reinforcement learning, the authors use prompt engineering to generate a small batch of high-quality long-thinking examples. These deliberately include behaviors such as planning, evaluating intermediate steps, reflecting and correcting, and trying other solutions. A round of lightweight SFT on them follows. The goal is for the model to pick up this way of writing first, so that reinforcement learning can then amplify it.

Policy optimization: online mirror descent

The authors use a variant of online mirror descent. In round ii, with the current model πθi\pi_{\theta_i} as the reference, they solve a KL-regularized problem:

max⁡θ E[r(x,y,y∗)]−τ KL(πθ(x) ∥ πθi(x))\max_\theta\ \mathbb{E}\Big[r(x,y,y^*)\Big]-\tau\,\mathrm{KL}\big(\pi_\theta(x)\,\|\,\pi_{\theta_i}(x)\big)

This problem has a closed-form solution, π∗(y,z∣x)=πθi(y,z∣x)exp⁡(r/τ)/Z\pi^*(y,z\mid x)=\pi_{\theta_i}(y,z\mid x)\exp(r/\tau)/Z. Taking logarithms of both sides gives an equality that must hold for every sample, which the authors then turn into a squared-error surrogate loss:

L(θ)=E[(r(x,y,y∗)−τlog⁡Z−τlog⁡πθ(y,z∣x)πθi(y,z∣x))2]L(\theta)=\mathbb{E}\left[\left(r(x,y,y^*)-\tau\log Z-\tau\log\frac{\pi_\theta(y,z\mid x)}{\pi_{\theta_i}(y,z\mid x)}\right)^2\right]

τlog⁡Z\tau\log Z is approximated by the mean reward rˉ\bar r of kk responses sampled for the same problem. In the gradient of this loss, the first term is a policy gradient with the mean reward as its baseline, and the second is an ℓ2\ell_2 regularizer.

Put plainly, we sample kk responses to the same problem, raise the probability of the correct ones (reward above the mean), lower that of the wrong ones, and at the same time don't drift too far from the previous round's model. At the end of each round, the reference model is replaced by the new model and the optimizer state is reset.

This method has no value network, and the authors' reasoning is illuminating. Suppose the model is partway through and can take a next step that leads straight to the right answer, or one that will lead to a mistake. A traditional value function penalizes the latter. But as long as the model can notice and fix the mistake and still get the right answer in the end, this experience of "making a mistake and then fixing it" is exactly what the model most needs in order to learn long thinking. So the authors look only at the final answer and do no credit assignment over intermediate steps.

Length penalty

As reinforcement learning proceeds, responses keep getting longer. Performance does improve, but inference cost keeps rising too. The authors added a length reward.

Among the kk responses to the same problem, let the shortest and longest lengths be min_len and max_len. The length reward for the ii-th response is

λi=0.5−len(i)−min_lenmax_len−min_len,length reward={λiif correctmin⁡(0,λi)if wrong\lambda_i = 0.5-\frac{\text{len}(i)-\text{min\_len}}{\text{max\_len}-\text{min\_len}},\qquad \text{length reward}=\begin{cases}\lambda_i & \text{if correct}\\ \min(0,\lambda_i) & \text{if wrong}\end{cases}
Teaching figure drawn for this note. The horizontal axis is the relative length of each response to the same problem, from 0 (shortest) to 1 (longest); the vertical axis is the length reward. The solid blue line is the reward for correct responses, falling linearly from 0.5 to −0.5; the dashed red line is the reward for wrong responses, 0 up to a relative length of one half and then coinciding with the blue line down to −0.5. The plot marks the shortest correct response at +0.5, the longest correct response at −0.5, and a wrong response at a relative length of about 0.67 with −0.17.
Drawn for this note from the definition in Section 2.3.3 of the paper; the four points correspond to the code example below.

Let's work through an example. Sample four responses to the same problem, with lengths of 1,000, 2,000, 3,000, and 4,000 tokens; the third one is wrong:

python
def length_reward(lengths, correct):
    lo, hi = min(lengths), max(lengths)
    if hi == lo:
        return [0.0] * len(lengths)
    out = []
    for n, ok in zip(lengths, correct):
        lam = 0.5 - (n - lo) / (hi - lo)
        out.append(lam if ok else min(0.0, lam))
    return out

lengths = [1000, 2000, 3000, 4000]
correct = [True, True, False, True]
for n, ok, r in zip(lengths, correct, length_reward(lengths, correct)):
    print(f"length {n:>5}  {'right' if ok else 'wrong'}  length reward {r:+.3f}")
text
length  1000  right  length reward +0.500
length  2000  right  length reward +0.167
length  3000  wrong  length reward -0.167
length  4000  right  length reward -0.500

Among correct responses, shorter is better. Wrong responses get no reward for being short, but are still penalized for being too long. Notice that the longest correct response has a length reward of −0.5, lower than the wrong one. That is fine, because the length reward is added with some weight to the original correctness reward of 0 or 1, and the 1 point for a correct answer still dominates.

The authors also found that the length penalty slows progress early in training, so they leave it out at first and apply it as a constant penalty once training has run for a while.

Sampling strategies and reward models

There are two strategies for sampling problems. Curriculum sampling trains on easy problems before hard ones, because early on the model is too weak, and compute spent on hard problems yields almost no correct samples. Prioritized sampling tracks each problem's success rate sis_i and draws problems in proportion to 1−si1-s_i, drilling the model where it is weakest.

Coding problems collected from the web often have no test cases, so the authors generate them automatically with the model and the test-generation library CYaRon. For each problem, 50 test cases are generated first and validated against 10 ground-truth solutions; a test case is kept only if at least 7 solutions agree on it. A problem enters the training set only if 9 or more solutions pass all the retained test cases. Of 1,000 online contest problems, only 323 made it in.

The difficulty with math problems is that the same answer can be written in many ways, such as a2−4a^2-4 and (a+2)(a−2)(a+2)(a-2). The authors compared two reward models: a classic reward model that outputs a score, with about 84.4% accuracy on manual spot checks, and a CoT reward model that writes out its reasoning before judging right or wrong, with about 98.5% accuracy. Reinforcement learning uses the latter.

128K long context and partial rollouts

When the model writes up to 128K tokens during reinforcement learning, one especially long response can hog sampling resources and slow down the whole training batch. This was the biggest engineering problem.

The authors' solution is called partial rollouts. Each sampling pass has a fixed cap on output length; responses that are not finished are stored in a replay buffer and continued in the next round. Only the current segment needs to be computed with the current policy, and earlier segments are taken straight from the buffer. Sampling workers run asynchronously, with some handling long responses while others handle short ones at the same time, so resources do not sit idle. The system can also detect repetitive content, cutting it off early and assigning a penalty.

Discussion

Long-thinking results

The long-thinking version of k1.5 performs on par with OpenAI o1. It scores 77.5 on AIME 2024 (o1: 74.4), 96.2 on MATH-500 (o1: 94.8), the 94th percentile on Codeforces (same as o1), and 62.5 on LiveCodeBench (o1: 67.2). On vision, it scores 74.9 on MathVista and 70.0 on MMMU.

The authors also used a mid-sized model to observe scaling. As training proceeds, response length and accuracy rise together, and the harder the benchmark, the more steeply length grows.

Another ablation weighs model size against long thinking. When a large and a small model are trained on the same data, the large model starts out better, but the small model can catch up by thinking longer; the large model, however, is more token-efficient. If you are after the best performance, scaling up the context of the larger model has a higher ceiling; if the inference budget is limited, training a smaller model to write longer is also a viable choice.

There are two more ablations. Compared with ReST, which learns only from the best responses and does not penalize wrong ones, the authors' method is clearly more sample-efficient, showing that negative gradients matter in training long thinking. Curriculum sampling is also clearly better than uniform sampling.

Long story short

Long thinking is powerful, but expensive. In Section 2.4, the authors compare four ways to carry long-thinking ability over to short-answer models:

  1. Model merging: directly average the weights of the long-thinking model and the short-answer model, with no training at all.
  2. Shortest rejection sampling: sample 8 times per problem and use the shortest correct response for SFT.
  3. DPO: the shortest correct response is the positive example; wrong long responses, and correct responses more than 1.5 times as long as the positive, are negative examples.
  4. Long-to-short RL: pick the model with the best balance of performance and length, then add a length penalty, sharply cut the cap on sampling length, and run another round of reinforcement learning.

The fourth turned out to be the most token-efficient. The short-answer k1.5 scores 60.8 on AIME 2024 using only 3,272 tokens on average; GPT-4o scores 9.3 and Claude 3.5 Sonnet 16.0. On MATH-500, the shortest version scores 88.2 with about as many tokens as other short-answer models.

Here, model merging is effective but not the best method. Its advantage is that it needs no training at all; by comparison, methods that do train can control length more precisely. This conclusion was later cited by Assembly of Experts to illustrate the limits of merging for shortening reasoning.

Things to watch for when reading this report

The model is not open-sourced, and neither its parameter count nor its pretraining scale is disclosed, so third parties cannot verify the report's numbers. The long-to-short comparison shows token efficiency only in figures, and most of the methods' specific numbers are not listed. Most ablations were run on smaller models.

Conclusion

Kimi k1.5 and DeepSeek-R1 gave similar answers on the same day. With nothing but a reward for whether the outcome is correct, plus a long enough context, a model can learn long thinking, with no need for tree search, value functions, or process rewards.

k1.5's own contribution is to spell out the engineering details of this path: how to choose problems, how to guard against guessing, how to sample efficiently at 128K length, how to curb overthinking with a length reward, and how to compress long thinking back into short answers.

If you want to build a product on a long-thinking model, and it comes from the same base as your existing short-answer model, try simply averaging the two first. No training is involved, just one weight operation; if the result isn't good enough, then consider running a round of long-to-short reinforcement learning.