Kimi k1.5: Scaling Reinforcement Learning with LLMs
- Publication
- Publication venue not yet verified
- First submitted to arXiv
- arXiv:2501.12599
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{transformerskimik152025,
title = {{Kimi k1.5: Scaling Reinforcement Learning with LLMs}},
author = {Kimi Team and Angang Du and Bofei Gao and Bowei Xing and Changjiu Jiang and Cheng Chen and Cheng Li and Chenjun Xiao and Chenzhuang Du and Chonghua Liao and Chuning Tang and Congcong Wang and Dehao Zhang and Enming Yuan and Enzhe Lu and Fengxiang Tang and Flood Sung and Guangda Wei and Guokun Lai and Haiqing Guo and Han Zhu and Hao Ding and Hao Hu and Hao Yang and Hao Zhang and Haotian Yao and Haotian Zhao and Haoyu Lu and Haoze Li and Haozhen Yu and Hongcheng Gao and Huabin Zheng and Huan Yuan and Jia Chen and Jianhang Guo and Jianlin Su and Jianzhou Wang and Jie Zhao and Jin Zhang and Jingyuan Liu and Junjie Yan and Junyan Wu and Lidong Shi and Ling Ye and Longhui Yu and Mengnan Dong and Neo Zhang and Ningchen Ma and Qiwei Pan and Qucheng Gong and Shaowei Liu and Shengling Ma and Shupeng Wei and Sihan Cao and Siying Huang and Tao Jiang and Weihao Gao and Weimin Xiong and Weiran He and Weixiao Huang and Weixin Xu and Wenhao Wu and Wenyang He and Xianghui Wei and Xianqing Jia and Xingzhe Wu and Xinran Xu and Xinxing Zu and Xinyu Zhou and Xuehai Pan and Y. Charles and Yang Li and Yangyang Hu and Yangyang Liu and Yanru Chen and Yejie Wang and Yibo Liu and Yidao Qin and Yifeng Liu and Ying Yang and Yiping Bao and Yulun Du and Yuxin Wu and Yuzhi Wang and Zaida Zhou and Zhaoji Wang and Zhaowei Li and Zhen Zhu and Zheng Zhang and Zhexu Wang and Zhilin Yang and Zhiqi Huang and Zihao Huang and Ziyao Xu and Zonghan Yang and Zongyu Lin},
year = {2025},
eprint = {2501.12599},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2501.12599}
}
On January 20, 2025, DeepSeek released R1. About an hour later, Moonshot AI released Kimi k1.5.
The two models set out to do very similar things. After OpenAI announced o1 in September 2024, everyone knew that "let the model think for a long time before answering" could greatly improve math and coding performance, but OpenAI did not say how. Both reports gave an answer, and the answers were close. Both train the model with reinforcement learning to write long chains of thought, with no Monte Carlo tree search, no value function, and no process reward model scoring each step.
The difference is that R1 released its weights and k1.5 did not; it is available only through Kimi's products and API. The report does not state the model's parameter count either. So what matters in k1.5 is not the model itself but how it was trained. Two parts of it have no counterpart in the R1 report: scaling the RL context length to 128K, and a whole section on carrying long-thinking ability back into short-answer models. One of the methods there is model merging.
Defining the problem
Scaling pretraining is limited by the total amount of high-quality data. The authors wanted another path that can keep scaling, letting the model explore on its own through rewards instead of only learning from a fixed dataset.
The problem setup is simple. Given a problem and a reference answer , the model first writes a reasoning process and then gives an answer . The reward is only 0 or 1. Coding problems are judged by whether the tests pass, math problems by whether the answer is right:
The authors have an interesting view of "long thinking." Traditional inference-time search, such as tree search, needs an evaluator to judge which path looks more promising before deciding which node to expand. But if every attempt, evaluation, and backtrack in the search is written out as one long piece of text, then a model that can write at great length can search implicitly within its own context.
This view turns the problem into two questions: how long can the context get, and can reinforcement learning teach the model to make good use of that length?
Solving the problem
Pick good problems first
The authors found that the success of reinforcement learning depends to a large extent on the problem set. A good problem set covers many domains, has an even spread of difficulty, and has answers that can be verified accurately.
Difficulty is estimated by the model itself. The SFT model answers each problem 10 times at a relatively high sampling temperature; the lower the pass rate, the harder the problem. Along the way, this also filters out problems that are too easy.
Verification has to guard against lucky guesses. Some reasoning problems have answers that are easy to guess, and the model may reach the right answer through wrong reasoning, fooling the reward. The authors excluded multiple-choice, true/false, and proof problems. For general QA problems, the model first guesses directly 8 times without thinking; if any guess is right, the problem is judged too easy to guess and removed.
Warming up long thinking
Before the real reinforcement learning, the authors use prompt engineering to generate a small batch of high-quality long-thinking examples. These deliberately include behaviors such as planning, evaluating intermediate steps, reflecting and correcting, and trying other solutions. A round of lightweight SFT on them follows. The goal is for the model to pick up this way of writing first, so that reinforcement learning can then amplify it.
Policy optimization: online mirror descent
The authors use a variant of online mirror descent. In round , with the current model as the reference, they solve a KL-regularized problem:
This problem has a closed-form solution, . Taking logarithms of both sides gives an equality that must hold for every sample, which the authors then turn into a squared-error surrogate loss:
is approximated by the mean reward of responses sampled for the same problem. In the gradient of this loss, the first term is a policy gradient with the mean reward as its baseline, and the second is an regularizer.
Put plainly, we sample responses to the same problem, raise the probability of the correct ones (reward above the mean), lower that of the wrong ones, and at the same time don't drift too far from the previous round's model. At the end of each round, the reference model is replaced by the new model and the optimizer state is reset.
This method has no value network, and the authors' reasoning is illuminating. Suppose the model is partway through and can take a next step that leads straight to the right answer, or one that will lead to a mistake. A traditional value function penalizes the latter. But as long as the model can notice and fix the mistake and still get the right answer in the end, this experience of "making a mistake and then fixing it" is exactly what the model most needs in order to learn long thinking. So the authors look only at the final answer and do no credit assignment over intermediate steps.
Length penalty
As reinforcement learning proceeds, responses keep getting longer. Performance does improve, but inference cost keeps rising too. The authors added a length reward.
Among the responses to the same problem, let the shortest and longest lengths be min_len and max_len. The length reward for the -th response is

Let's work through an example. Sample four responses to the same problem, with lengths of 1,000, 2,000, 3,000, and 4,000 tokens; the third one is wrong:
def length_reward(lengths, correct):
lo, hi = min(lengths), max(lengths)
if hi == lo:
return [0.0] * len(lengths)
out = []
for n, ok in zip(lengths, correct):
lam = 0.5 - (n - lo) / (hi - lo)
out.append(lam if ok else min(0.0, lam))
return out
lengths = [1000, 2000, 3000, 4000]
correct = [True, True, False, True]
for n, ok, r in zip(lengths, correct, length_reward(lengths, correct)):
print(f"length {n:>5} {'right' if ok else 'wrong'} length reward {r:+.3f}")
length 1000 right length reward +0.500
length 2000 right length reward +0.167
length 3000 wrong length reward -0.167
length 4000 right length reward -0.500
Among correct responses, shorter is better. Wrong responses get no reward for being short, but are still penalized for being too long. Notice that the longest correct response has a length reward of −0.5, lower than the wrong one. That is fine, because the length reward is added with some weight to the original correctness reward of 0 or 1, and the 1 point for a correct answer still dominates.
The authors also found that the length penalty slows progress early in training, so they leave it out at first and apply it as a constant penalty once training has run for a while.
Sampling strategies and reward models
There are two strategies for sampling problems. Curriculum sampling trains on easy problems before hard ones, because early on the model is too weak, and compute spent on hard problems yields almost no correct samples. Prioritized sampling tracks each problem's success rate and draws problems in proportion to , drilling the model where it is weakest.
Coding problems collected from the web often have no test cases, so the authors generate them automatically with the model and the test-generation library CYaRon. For each problem, 50 test cases are generated first and validated against 10 ground-truth solutions; a test case is kept only if at least 7 solutions agree on it. A problem enters the training set only if 9 or more solutions pass all the retained test cases. Of 1,000 online contest problems, only 323 made it in.
The difficulty with math problems is that the same answer can be written in many ways, such as and . The authors compared two reward models: a classic reward model that outputs a score, with about 84.4% accuracy on manual spot checks, and a CoT reward model that writes out its reasoning before judging right or wrong, with about 98.5% accuracy. Reinforcement learning uses the latter.
128K long context and partial rollouts
When the model writes up to 128K tokens during reinforcement learning, one especially long response can hog sampling resources and slow down the whole training batch. This was the biggest engineering problem.
The authors' solution is called partial rollouts. Each sampling pass has a fixed cap on output length; responses that are not finished are stored in a replay buffer and continued in the next round. Only the current segment needs to be computed with the current policy, and earlier segments are taken straight from the buffer. Sampling workers run asynchronously, with some handling long responses while others handle short ones at the same time, so resources do not sit idle. The system can also detect repetitive content, cutting it off early and assigning a penalty.
Discussion
Long-thinking results
The long-thinking version of k1.5 performs on par with OpenAI o1. It scores 77.5 on AIME 2024 (o1: 74.4), 96.2 on MATH-500 (o1: 94.8), the 94th percentile on Codeforces (same as o1), and 62.5 on LiveCodeBench (o1: 67.2). On vision, it scores 74.9 on MathVista and 70.0 on MMMU.
The authors also used a mid-sized model to observe scaling. As training proceeds, response length and accuracy rise together, and the harder the benchmark, the more steeply length grows.
Another ablation weighs model size against long thinking. When a large and a small model are trained on the same data, the large model starts out better, but the small model can catch up by thinking longer; the large model, however, is more token-efficient. If you are after the best performance, scaling up the context of the larger model has a higher ceiling; if the inference budget is limited, training a smaller model to write longer is also a viable choice.
There are two more ablations. Compared with ReST, which learns only from the best responses and does not penalize wrong ones, the authors' method is clearly more sample-efficient, showing that negative gradients matter in training long thinking. Curriculum sampling is also clearly better than uniform sampling.
Long story short
Long thinking is powerful, but expensive. In Section 2.4, the authors compare four ways to carry long-thinking ability over to short-answer models:
- Model merging: directly average the weights of the long-thinking model and the short-answer model, with no training at all.
- Shortest rejection sampling: sample 8 times per problem and use the shortest correct response for SFT.
- DPO: the shortest correct response is the positive example; wrong long responses, and correct responses more than 1.5 times as long as the positive, are negative examples.
- Long-to-short RL: pick the model with the best balance of performance and length, then add a length penalty, sharply cut the cap on sampling length, and run another round of reinforcement learning.
The fourth turned out to be the most token-efficient. The short-answer k1.5 scores 60.8 on AIME 2024 using only 3,272 tokens on average; GPT-4o scores 9.3 and Claude 3.5 Sonnet 16.0. On MATH-500, the shortest version scores 88.2 with about as many tokens as other short-answer models.
Here, model merging is effective but not the best method. Its advantage is that it needs no training at all; by comparison, methods that do train can control length more precisely. This conclusion was later cited by Assembly of Experts to illustrate the limits of merging for shortening reasoning.
Things to watch for when reading this report
The model is not open-sourced, and neither its parameter count nor its pretraining scale is disclosed, so third parties cannot verify the report's numbers. The long-to-short comparison shows token efficiency only in figures, and most of the methods' specific numbers are not listed. Most ablations were run on smaller models.
Conclusion
Kimi k1.5 and DeepSeek-R1 gave similar answers on the same day. With nothing but a reward for whether the outcome is correct, plus a long enough context, a model can learn long thinking, with no need for tree search, value functions, or process rewards.
k1.5's own contribution is to spell out the engineering details of this path: how to choose problems, how to guard against guessing, how to sample efficiently at 128K length, how to curb overthinking with a length reward, and how to compress long thinking back into short answers.
If you want to build a product on a long-thinking model, and it comes from the same base as your existing short-answer model, try simply averaging the two first. No training is involved, just one weight operation; if the result isn't good enough, then consider running a round of long-to-short reinforcement learning.