Assembly of Experts: Linear-time construction of the Chimera LLM variants with emergent and adaptable behaviors
- Publication
- Publication venue not yet verified
- First submitted to arXiv
- arXiv:2506.14794
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{modelmergingassemblyofexperts2025,
title = {{Assembly of Experts: Linear-time construction of the Chimera LLM variants with emergent and adaptable behaviors}},
author = {Henrik Klagges and Robert Dahlke and Fabian Klemm and Benjamin Merkel and Daniel Klingmann and David A. Reiss and Dan Zecha},
year = {2025},
eprint = {2506.14794},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2506.14794}
}
At one point DeepSeek had two models that were very much alike, yet had very different personalities.
One was DeepSeek-R1, a reasoning model. Before answering, it writes a long stretch of reasoning between <think> and </think>; it is very good at math problems, but it talks a lot. The other was DeepSeek-V3-0324, updated in March 2025, a general chat model. It writes no reasoning process, answers directly, and is fast.
The two have exactly the same architecture, both starting from the DeepSeek-V3 base, with 671 billion parameters in total. If R1's reasoning ability could be combined with V3-0324's concision without any retraining, that would be a great deal.
The German software consultancy TNG Technology Consulting actually did this. In April 2025 they put the merged DeepSeek-R1T-Chimera on Hugging Face, and only a month or so later followed up with this paper, explaining how they did it and what they saw along the way.
Defining the problem
First, a quick review of DeepSeek-V3's architecture, which is the key to understanding the whole paper.
It is a mixture-of-experts (MoE) Transformer with 61 decoder layers. The first 3 are ordinary dense layers. In the remaining 58, the feedforward network is replaced by experts: each layer has a shared expert that every token passes through, plus 256 routed experts, of which a router picks 8 for each token. So there are 671B parameters in total, but each token actually uses only about 37B.
Take a model apart and it is just a pile of tensor files. The safetensors weight files of the DeepSeek-V3 family contain 91,991 tensors in all, which fall into several categories: the first 3 dense layers, each layer's attention, shared experts, routed experts, and the router gates.
When merging two models, each category of tensors can be handled separately. This is what the authors mean by "assembly": deciding not only how much weight each model gets, but also which parts are merged and which are taken directly from one side.
Others had used merging to shorten the responses of reasoning models before. Wu et al., for example, got good results on 7B to 14B models but ran into difficulties at 32B and above, where it was hard to shorten responses substantially while preserving reasoning quality. This paper asks whether a 671B model can be assembled this way, and how the behavior of the assembled model changes.
Solving the problem
The merging rule
Let be the base model (here V3-0324) and the other models (here only R1). For each tensor , the merged tensor is
There are three things to tune here:
- : the weight of each model, usually a convex combination with summing to 1. With two models, is R1's share.
- : the set of tensors to merge. The authors compare two: all tensors (full merging), and only the routed experts, excluding the gates (expert merging).
- : a threshold. Only tensors that differ enough from the base are merged, with the difference measured by the normalized Frobenius norm, that is, the norm divided by the square root of the number of elements in the tensor.
The threshold borrows the trimming idea of TIES-Merging, but TIES trims parameter by parameter, whereas here the decision to merge is made for a whole tensor at once.
In practice, you read the 91,991 tensors one by one, compute the merged result, and write it back to files in the same format, which vLLM can load directly. The computation scales with model size, hence "linear time" in the title, and no gradient computation is needed at all.
First, where do the two models differ?
The authors first measured the difference between R1 and V3-0324 for every tensor. The results are quite regular. The routed experts differ fairly evenly throughout the model, with a clear pattern only in the first and last few layers. The shared experts' gate projections are almost identical in the early layers and differ a great deal in the last few layers, and the attention layers show a similar trend.
This distribution of differences later became the basis for choosing the threshold .
How to evaluate
Every merged model runs inference with vLLM on 8 H100s or 8 MI325Xs and is tested mainly on two benchmarks: MT-Bench (multi-turn conversation quality) and AIME-2024 (math competition problems). The authors call the average of the two the "intelligence score."
Cost is measured by the average number of output tokens, with R1's output taken as 100%.
There is also a behavioral metric. The authors force every response to begin with <think>, then check whether the closing </think> appears in the response. R1 always produces it and V3-0324 never does, so this proportion shows how much a model "behaves like a reasoning model."
Discussion
Almost every combination works

The ideal position on this plot is the upper left: high score, low cost.
R1 is on the right, with a score of about 0.85 at 100% cost; V3-0324 is at the lower left, with a score of about 0.69 at about 31% cost. The various merged models are scattered between the two, and none of them drops to the bottom. The authors stress that across the whole parameter space they searched, no broken model appeared.
Several points sit at around 50% cost with scores of 0.82 or more, getting close to R1's score with half the tokens. There is also a full-tensor merge at about 80% cost with a score of nearly 0.88, higher than R1 itself.
The authors interpret this as V3, V3-0324, and R1 all lying in the same low-loss valley, so convex combinations do not leave it. This agrees with the observations in Model Soups, only at a scale several orders of magnitude larger.
Reasoning behavior appears abruptly

The horizontal axis is R1's share . Output length jumps sharply around 0.5 and is relatively flat on either side, forming an S-shaped curve.
The </think> tag shows this even more clearly. Merged models with an R1 share of 0.504 or more usually write out a complete reasoning process and close it with </think>; those with a share just slightly lower usually do not. This transition happens only within a very narrow range around 0.5.
So different properties of a merged model change in different ways. General reasoning ability changes smoothly with the share, while a behavioral trait such as "whether to write out a reasoning process" acts like a switch, appearing only once a threshold is crossed.
Another key point in the figure is the gap between the two symbols. The cost of merging only the routed experts (circles) is much lower overall than that of full-tensor merging (squares). Even with every routed expert replaced by R1's, the cost is only about 60% of R1's.
Threshold: which parts matter
With full-tensor merging fixed at , the authors gradually raise the threshold . At low thresholds, most attention and routed-expert tensors get merged. Once the threshold rises to a certain level, most shared-expert tensors are excluded and keep V3-0324's values; raise it further and the attention tensors are excluded too, leaving only the routed experts with the largest differences still being merged.
As a result, cost falls steadily as the threshold rises while the intelligence score stays stable, until the threshold gets high enough to start excluding large numbers of R1's routed experts; only then does the score start to drop.
R1T-Chimera
Putting these observations together, the authors propose two hypotheses. First, R1's reasoning ability resides mainly in the routed experts. Second, the rest of V3-0324, including attention, shared experts, and dense layers, is good enough to work with these experts without noticeably lowering performance.
So they tried the most extreme setting: merge only the routed experts, with , replacing the routed experts entirely with R1's and keeping everything else from V3-0324. This is DeepSeek-R1T-Chimera.

On MT-Bench the three are nearly identical. On AIME 2024, Chimera scores about 73, far above V3-0324's roughly 59 but still behind R1's roughly 80. On the four BigCodeBench subtasks, Chimera is roughly level with R1. Output tokens drop by about 40%.
The authors say Chimera's reasoning is more compact and better organized than that of either parent; this is a qualitative observation, and the paper does not quantify it.
TNG first replaced R1 with Chimera on its internal GPU cluster, then released the weights. According to the paper, by the end of May 2025 it was processing nearly 5 billion tokens a day through Chutes, a serverless inference platform.
Scope of the evidence
The paper's experimental design is fairly bare-bones, and a few things should be kept in mind.
- Too few evaluations: the entire parameter search uses only two benchmarks, MT-Bench and AIME-2024. AIME-2024 has only 30 problems, so a difference of one or two problems amounts to several percentage points, and the paper reports neither the number of repeats nor the variance.
- The "intelligence score" is a custom metric: directly averaging a model-judged conversation score with a math-competition score works for ranking, but is too coarse to serve as an absolute measure of capability.
- Only one pair of parents: all observations come from V3-0324 and R1, and the hypothesis that "reasoning resides in the routed experts" has not been verified on other model families.
In the related-work section, the authors also mention that the Kimi k1.5 team found fine-tuning with a length-penalized loss produces more token-efficient models than merging a long-reasoning and a short-reasoning model. Merging's advantage lies in being cheap, not necessarily in being best.
In July 2025, TNG released R1T2 Chimera, switching to three parents (R1-0528, R1, V3-0324) and still keeping V3-0324's attention and shared layers.
Conclusion
This paper takes model merging to the 671B scale and, by exploiting the structure of MoE, breaks "merging" down into "choosing parts." Swapping R1's routed experts into V3-0324's body yields a model whose reasoning is close to R1's and whose output is 40% shorter, all without a single gradient computation.
The most interesting part is the phase transition observed along the way. Reasoning ability changes smoothly, yet the habit of writing out a reasoning process appears suddenly, the moment R1's share passes one half.
The parts of MoE models can be taken apart and swapped like this, provided both models come from the same base. Stitch together two MoEs from different families, and you really do end up with just a patchwork monster.