What Matters for Model Merging at Scale?
- Publication
- Transactions on Machine Learning Research 2025Journal
- First submitted to arXiv
- arXiv:2410.03617
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{modelmergingmergingatscale2024,
title = {{What Matters for Model Merging at Scale?}},
author = {Prateek Yadav and Tu Vu and Jonathan Lai and Alexandra Chronopoulou and Manaal Faruqui and Mohit Bansal and Tsendsuren Munkhdalai},
year = {2024},
eprint = {2410.03617},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2410.03617}
}
Early model-merging research often centered on CLIP, ViT, T5, and BERT, with individual experiments using larger vision or language models. DARE, for example, had already tested merges of 7B and 13B experts. But base quality, parameter scale, expert count, and merging method were rarely compared systematically within one experiment. Do techniques that work best in small settings still matter at scale?
By 2024, merging was already appearing in large-model training pipelines. The paper notes that the Llama 3 and Gemma 2 technical reports both describe model merging. Would findings from smaller models hold when scaled up?
During an internship at Google, UNC's Prateek Yadav, first author of TIES-Merging, joined Tu Vu, Jonathan Lai, Alexandra Chronopoulou, Manaal Faruqui, and Tsendsuren Munkhdalai from Google and Virginia Tech, and UNC's Mohit Bansal, to test this systematically with PaLM-2, up to 64 billion parameters.
Defining the problem
The authors ask four questions:
- Are experts easier to merge when derived from a purely pretrained base or an instruction-tuned base?
- Does merging become easier or harder as models grow?
- How does merging affect zero-shot ability on unseen tasks?
- How many experts can be merged without substantial degradation, and how does that depend on model size?
Solving the problem
Experimental design
This paper introduces no new method; its effort goes into experimental design.
Base models: PaLM-2 and its instruction-tuned counterpart PaLM-2-IT, at 1B, 8B, 24B, and 64B.
Experts: eight task categories from the T0 data collection are “held-in,” with one fully fine-tuned expert per category. A separate collection of “held-out” tasks, excluded from expert fine-tuning, measures zero-shot generalization.
Merging: merge 2, 4, 6, or 8 experts. Three random seeds select expert subsets for each setting, and results are averaged.
Methods: four methods requiring no additional information and scalable to tens of billions of parameters: simple averaging, task arithmetic, TIES-Merging, and DARE plus TIES, called Dare-TIES in the paper. Methods requiring Fisher matrices, backpropagation, or activations are excluded.
Baselines: task-specific experts and a multitask model trained on a mixture of all eight task categories.
Expanding this matrix gives experts and merge configurations, totaling about nine thousand model evaluations including baselines. The three seeds select which experts enter a merge; each expert is not retrained three times. The same subset seeds are shared across sizes, bases, and methods, avoiding an accidental advantage from drawing easier tasks.
What do the experts know?
The eight held-in categories contain 16 datasets, two per category: multiple-choice QA, extractive QA, closed-book QA, sentiment analysis, topic classification, structure-to-text, summarization, and paraphrase identification. The summarization expert, for example, uses CNN/DailyMail and XSum, while the paraphrase expert uses MRPC and QQP. A “QA plus summarization” merge is evaluated held-in only on those two categories' four datasets.
Held-out evaluation reserves seven datasets in four categories: sentence completion with COPA and HellaSwag, inference with ANLI and RTE, coreference with WSC and Winogrande, and word sense with WiC. Every merged model uses this same held-out set to test transfer after learning expert tasks.
PaLM-2-IT also has a specific provenance. The authors instruction-tune with FLAN-v2 for up to 60,000 steps after removing T0-mixture tasks, then fully fine-tune experts from this base. Experts default to 2,000 steps, learning rate , and dropout 0.05, adjusting steps for convergence on some tasks. This “IT” version is built for the study's comparison, and expert training data are relatively limited, an important condition when interpreting scale effects later.
Reading the scores
All scores are normalized. Held-in scores divide by the corresponding expert score, so 1 matches the expert. Held-out scores divide by the base-model score, so 1 matches the base before expert fine-tuning. Values above 1 indicate that merging improves the base on unseen tasks.
For example, if an expert scores 80 and the merge 72, held-in retention is 0.9. If the base scores 40 on a held-out task and the merge 44, held-out performance is 1.1: a 10% relative improvement, or 4 raw points. Because datasets include summarization and other metrics, a normalized increase of 0.1 cannot uniformly be converted to ten percentage points of accuracy.
Averaging also has an order: first average normalized dataset scores within each category, then average across categories and the three subset seeds. This gives each task category equal weight. Averaging all datasets directly would underweight a category such as word sense, represented only by WiC.
Discussion
Starting from instruction-tuned models
Across all settings, experts from PaLM-2-IT merge to higher held-in performance than those from PaLM-2, regardless of method, size, or expert count. The gap grows as more experts are merged.
Following Ortiz-Jimenez and colleagues, the authors suggest that large-scale instruction tuning may make weights more disentangled, reducing interference between task updates. They recommend deriving experts from the strongest available base, with zero-shot ability as an indicator of base quality.
Larger models merge better

Start with the two held-in plots on the left. Colors progress from blue at 1B to red at 64B, with larger models generally higher. Large instruction-tuned models merge best: merged 64B PaLM-2-IT nearly matches the task-specific experts.
The same figure answers the fourth question. In the appendix's task-arithmetic results, increasing experts from 2 to 8 lowers 8B PaLM-2 held-in performance from 0.68 to 0.42, versus 0.95 to 0.88 for 8B PaLM-2-IT. With the 64B IT base, the corresponding values are 1.00 and 0.93. Both scale and base quality reduce the cost of merging more experts, though merging eight still loses about 7% of mean retained performance.
Merging can improve transfer
The two right-hand plots show held-out tasks. In most cases, merged models outperform the original base on unseen tasks, giving values above 1.
The bases behave differently. Expert count has little effect for PaLM-2, but its 64B version improves over the base by roughly 30%. The authors attribute this mainly to PaLM-2's weak initial zero-shot ability and substantial room for improvement. PaLM-2-IT benefits from more experts, especially at 24B and 64B. They hypothesize that merging more experts cancels more of their individual noise.
Finally, compare with multitask training. Merging 8 experts from 64B PaLM-2-IT gives task arithmetic a held-out score of 1.09 versus 1.05 for the multitask model, alongside held-in 0.93 versus 0.99. Better mean generalization to new tasks comes with a tradeoff on learned tasks. At 24B, task arithmetic with 8 experts reaches held-out 1.18, matching the multitask model's 1.18. These cases should be distinguished.
The choice of method matters less
On 64B PaLM-2-IT, all four merging methods have almost identical held-in and held-out performance, regardless of expert count.

The horizontal axis groups 2, 4, 6, and 8 experts, with four method bars per group. The left reference line represents experts; the right represents the IT base. A value of 1 therefore has different references on the two sides. Left-hand bars sit slightly below experts; right-hand bars rise above the base as sources increase. Differences between methods remain small.
The authors hypothesize that very large experts trained on relatively limited data are heavily overparameterized. Techniques useful in smaller models, such as TIES's interference handling and DARE's random sparsification, then matter less. They conclude that simple averaging may suffice for large instruction-tuned experts.
Comparing all four methods
Take the same setting of eight IT experts and compare two scales side by side. Every cell follows the normalization defined above:
| Method | 24B held-in | 24B held-out | 64B held-in | 64B held-out |
|---|---|---|---|---|
| Weight averaging | 0.77 | 1.16 | 0.93 | 1.09 |
| Task arithmetic | 0.92 | 1.18 | 0.93 | 1.09 |
| DARE-TIES | 0.88 | 1.17 | 0.93 | 1.09 |
| TIES | 0.86 | 1.16 | 0.93 | 1.10 |
| Multitask training | 0.98 | 1.18 | 0.99 | 1.05 |
Compiled from the PaLM-2-IT tables in Appendix C, all with eight experts.
At 64B, the four methods nearly coincide. At 24B, held-in performance still differs by 0.15. “Bigger is easier to merge” depends on the combination of base, tasks, and scale; there is no universal parameter-count threshold at which all methods suddenly become equivalent. Even at 64B, returning to non-instruction-tuned PaLM-2 with eight experts gives task arithmetic 0.67 held-in and TIES 0.56, restoring a gap.
The methods also use different scales. Weight averaging is , while task arithmetic uses , permitting adjustment of the total update. TIES's coordinatewise average and DARE's random sparsification further alter updates. The table shows that final performance can converge, while the operations remain distinct.
What can we take from these experiments?
First, normalized held-out improvement depends on the base score. The non-IT 64B base has weaker zero-shot scores, and task arithmetic with eight experts reaches 1.35, still below multitask training at 1.72. To judge usefulness, place the base, multitask model, and experts back into the comparison rather than considering only the “over 30% improvement.”
Second, scale trends have exceptions. With the non-IT 24B base and eight experts, simple averaging scores 0.83 held-out and TIES 0.90, both below the base. The study supports the overall trend that stronger bases and greater scale ease merging, while degradation still calls for task-level inspection. Instruction tuning disentangling weights and merging canceling expert noise are explanatory hypotheses; the study does not directly establish a causal chain.
Third, this study covers one model family, at most eight experts, specified T0 tasks, and a defined expert fine-tuning budget. It excludes differing tokenizers, independently pretrained sources, and dozens of extensively trained domain experts, and does not measure complete-product latency, safety, or serving cost. For needs outside this scope, its experimental matrix is the most useful guide: control base quality and expert count before comparing recipes.
Conclusion
Scaling PaLM-2 merging to 64 billion parameters gives a consistent message. Experts should derive from a strong, preferably instruction-tuned base. Larger models are easier to merge and can accommodate more experts. Merging usually improves zero-shot generalization and can exceed multitask training in large models. Differences among merging methods also shrink with scale.
The authors remind us that merged models usually still trail experts on their own tasks. They suggest that, compared with multitask models, this loss may be less severe.
The final observation is worth remembering. TIES-Merging's own first author runs this experiment and finds that, at 64B, TIES performs about the same as simple averaging.