Arcee's MergeKit: A Toolkit for Merging Large Language Models
- Publication
- EMNLP 2024 Industry Track 2024Conference
- First submitted to arXiv
- arXiv:2403.13257
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{modelmergingmergekit2024,
title = {{Arcee's MergeKit: A Toolkit for Merging Large Language Models}},
author = {Charles Goddard and Shamane Siriwardhana and Malikeh Ehghaghi and Luke Meyers and Vlad Karpukhin and Brian Benedict and Mark McQuade and Jacob Solawetz},
year = {2024},
eprint = {2403.13257},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2403.13257}
}
From roughly late 2023 to early 2024, a whole crop of models with very long names sprang up on Hugging Face.
Their names often carried strings like slerp, ties or dare, or simply chained together the names of two or three models. None of these models had gone through any new training. Their creators had just mixed the weights of a few existing fine-tuned models and uploaded the result, and quite a few of them climbed into the upper ranks of the Open LLM Leaderboard for a while.
Most of them were made with the same tool, MergeKit, an open-source library developed by Charles Goddard and colleagues at Arcee. This paper is its formal introduction. It is short, mostly covering how the tool is designed and which methods it supports, with a set of merging experiments in the medical domain at the end.
The value of reading it lies in seeing clearly what “model merging” actually looked like once it moved from papers into everyday community practice.
Defining the problem
By then, research on model merging had been accumulating for several years, from weight averaging, Model Soups and task arithmetic to TIES-Merging and DARE. But each paper had its own code, mostly written for particular models and datasets. Anyone trying to use it to merge two 70B language models ran into very practical problems:
- Memory: two 70B models stored in 16 bits take two to three hundred GB, more than an ordinary machine can load, let alone several models at once.
- Architecture differences: Llama, Mistral and Qwen differ in weight naming and layer structure, so each needs separate handling.
- Recipes are hard to reproduce: which model gets how much weight, and which method applies to which layers, lives in ad hoc scripts that others find hard to follow.
What the authors wanted was a shared tool that would let people who don't write code merge models, and let researchers add new methods easily.
Solving the problem
How the paper classifies merging methods
The authors first lay out a taxonomy of merging methods and mark how far MergeKit's support extends.

The taxonomy turns on two questions, whether the models share an architecture and whether they come from the same initialization.
The left branch is what MergeKit actually supports, and all of it requires “the same architecture, starting from the same base model.” This is exactly the condition under which linear mode connectivity and task arithmetic hold. The right branch deals with independently trained models, which need neuron alignment first, and was not yet supported when the paper was written. The authors also say frankly that the methods on the right have not yet seen the kind of wide use the ones on the left have.
Models with entirely different architectures, such as CALM, which connects two models with cross-attention, or FuseLLM, which aligns the output distributions of two models, all require extra training. The authors class them as “fusion” rather than merging.
A merge recipe is just a YAML file
MergeKit's main interface is a YAML configuration file. The file specifies the merge method, the input models and the parameters the method needs, and running it takes a single command:
mergekit-yaml path/to/your/config.yml ./output-model-directory
A TIES merge can be configured like this (taken from the official examples/ties.yml):
models:
- model: psmathur/orca_mini_v3_13b
parameters:
density: [1, 0.7, 0.1] # density gradient
weight: 1.0
- model: garage-bAInd/Platypus2-13B
parameters:
density: 0.5
weight: [0, 0.3, 0.7, 1] # weight gradient
- model: WizardLM/WizardMath-13B-V1.0
parameters:
density: 0.33
weight:
- filter: mlp
value: 0.5
- value: 0
merge_method: ties
base_model: TheBloke/Llama-2-13B-fp16
parameters:
normalize: true
int8_mask: true
dtype: float16
All three 13B models are fine-tuned from the same Llama-2-13B. base_model names this common base, and each task vector is the difference between a model and that base. density is the fraction TIES keeps after pruning, and weight is the weight given to each task vector.
This configuration also uses two other features.
First, parameters can be written as lists. density: [1, 0.7, 0.1] is interpolated along the depth of the layers into a “gradient,” with shallow layers keeping everything and deeper layers keeping less and less.
Second, parameters can use filter to specify where they apply. The WizardMath setting gives the MLP layers a weight of 0.5 and every other tensor 0, which means only its changes to the MLP layers are taken.
The benefit of a configuration file is that a merge recipe becomes text that can be shared, reproduced and rewritten. The model cards of many community merges simply include a YAML file like this one.
Why SLERP is so common
Among the methods MergeKit supports, one of the most widely used in the community is SLERP (spherical linear interpolation). It comes from computer animation research in 1985, where it was used to interpolate smoothly between rotations.
Between two tensors and , standard linear interpolation is . SLERP instead first computes the angle between the two normalized tensors and then moves along the sphere:
What is the difference? Take two unit-length vectors at 90 degrees. The midpoint of linear interpolation has a length of only , while SLERP's midpoint still has length 1. The larger the angle, the more linear interpolation “shortens” the weights.
For language models fine-tuned from the same base, however, the angle between the weights is usually tiny. At an angle of only 2 degrees, the midpoint of linear interpolation has length , and the two methods are almost indistinguishable. MergeKit's implementation allows for this, falling back to standard linear interpolation whenever the normalized inner product of two tensors exceeds 0.9995.
The paper says SLERP's success shows that “there is often a spherical path with a lower loss barrier than a direct linear interpolation.” That is the authors' claim; the paper provides no corresponding barrier measurement.
Also, SLERP can merge only two models at a time, which is why the example above uses TIES instead. The official gradient-slerp.yml example shows another way to use SLERP, giving attention and MLP different interpolation coefficients that also vary with layer depth; for example, attention ramps from to while MLP does the reverse.
Two kinds of merging without arithmetic
MergeKit also has two methods that do no arithmetic in parameter space at all.
Passthrough, which the community calls “FrankenMerging,” stitches layers from different models directly together into a new model with an unusual number of layers. The popular Goliath-120B interleaves layer segments from Xwin and Euryale, two Llama-2 70B fine-tunes, about 16 layers per segment, with neighboring segments partly overlapping. The first step of SOLAR-10.7B's Depth Up-Scaling works the same way. It makes two copies of one 32-layer model, removes the last 8 layers from one copy and the first 8 from the other, and joins them into 48 layers before continuing pretraining.
mergekit-moe, which the community calls “FrankenMoE,” takes several dense models as experts and assembles them into a mixture-of-experts (MoE) model. The router can route semantically using the hidden states of prompts, or be randomly initialized and trained afterward.
Neither method involves any theory of alignment or interpolation. Whether the stitched-together result works can only be settled by evaluation.
Merging 70B models on a laptop
MergeKit's efficiency comes from an “out-of-core” design. Instead of loading a whole model into memory at once, it loads only the tensors the current operation needs.
Internally, a merge is represented as a directed acyclic graph (DAG) in which each node is an operation. A scheduler uses this graph to decide the execution order, keeping as little data in memory at any one time as possible, and intermediate results that are no longer needed are released automatically. The authors say this lets MergeKit scale from research clusters all the way down to laptops with no GPU and limited memory.

This architecture diagram is most useful to anyone who wants to add a new method. The authors list several modules to get to know first. merge_methods/base.py defines the interface a new method must implement, graph.py handles scheduling and data movement, plan.py builds the computation graph, and architecture.py describes the structure of each model type, with most architectures defined by a single simple JSON file.
Discussion
One set of medical experiments
The paper's only experiment merges the medical model Meditron-7B with Llama2-7B-Chat. Both start from the Llama2-7B base, and Meditron was further pretrained on a large medical corpus.
| Model | USMLE | MedMCQA | PubMedQA | ARC-C | HellaSwag | MMLU |
|---|---|---|---|---|---|---|
| Llama2-7B-Chat | 35.90 | 35.45 | 73.40 | 44.20 | 55.40 | 46.37 |
| Meditron-7B | 38.40 | 24.07 | 71.40 | 40.20 | 54.50 | 33.06 |
| LERP | 39.10 | 36.65 | 75.60 | 46.76 | 58.66 | 48.44 |
| SLERP | 39.20 | 36.91 | 75.60 | 46.84 | 58.67 | 47.97 |
| DARE-TIES | 36.37 | 27.56 | 72.20 | 42.92 | 54.79 | 41.17 |
| TIES | 38.73 | 32.27 | 75.60 | 45.05 | 58.23 | 45.03 |
The first three columns are medical exams and question answering; the last three measure general ability.
Except for DARE-TIES, which falls below Meditron-7B on USMLE (36.37 vs. 38.40), the merged models beat Meditron-7B on all three medical benchmarks, and LERP and SLERP also surpass Llama2-7B-Chat on general ability. On this basis, the authors judge SLERP the best choice in the medical domain.
Read this table with a few cautions in mind, though. First, most differences between LERP and SLERP are within 0.5 percentage points, and on MMLU LERP actually comes out ahead, consistent with the earlier calculation that “when the angle is small, the two are nearly the same.” Second, the table does not state the merge coefficients, the prompt format used for evaluation or the number of runs, and it reports no variance, so there is no way to tell whether these differences are stable. Third, there is only one pair of models and one domain. The experiment works more as a usage example than as grounds for ranking methods.
The merging craze on the leaderboard
The paper says MergeKit “has facilitated the merging of thousands of models.” That craze brought problems too.
In March 2024, a user complained on the Open LLM Leaderboard discussion board that the leaderboard hid models tagged “merge” by default, costing their merged models visibility. The Hugging Face staff who maintain the leaderboard replied that the models were hidden because the lineage of merged models is almost impossible to trace. After merge upon merge, it is hard to know what data has been mixed in, which makes accidental contamination of evaluation sets easy, and the scores of such models often fail to translate into real-world capability (discussion thread).
This is not MergeKit's own fault, but it does make one thing clear. Merging pushes the cost of producing a new model toward zero, while the trustworthiness of evaluation has not risen to match. Merge a pile of models with high leaderboard scores and the score may climb even higher, but that may just be adding up the test questions each model happened to pick up along the way.
The license reversal
MergeKit was first released under LGPL v3. In February 2025, Arcee switched to the Business Source License with the v0.1 release, citing the need to fund long-term development. On October 31 of the same year, it announced a return to LGPL v3, with the earlier BSL releases kept in the version history. If your project is pinned to an older version, check which license applies to that version first.
Conclusion
The MergeKit paper proposes no new theory of merging; its contribution is engineering. It gathers methods from many papers into one framework, makes merge recipes shareable through YAML, and uses lazy loading so that large models can be merged on small machines.
Precisely because it was so easy to use, model merging went from papers to everyday community practice in 2024. The technical reports of later models such as Llama 3 and Tülu 3 also show merging written into their official training pipelines.
Before you try it yourself, keep one thing in mind. MergeKit reduces merging to the effort of writing one YAML file; evaluation gets no easier along the way.