Back to paper library
arXiv 2024Nat Mach Intell 2025Model mergingPaper notes

Evolutionary Model Merge

Let evolution choose the recipe

Original paper

Evolutionary Optimization of Model Merging Recipes

Paper authors

Takuya Akiba · Makoto Shing · Yujin Tang · Qi Sun · David Ha

First submitted to arXiv
arXiv:2403.13187
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{modelmergingevolutionarymodelmerge2024,
  title = {{Evolutionary Optimization of Model Merging Recipes}},
  author = {Takuya Akiba and Makoto Shing and Yujin Tang and Qi Sun and David Ha},
  year = {2024},
  eprint = {2403.13187},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.13187}
}
On this page

In early 2024, model merging was booming in the open-source community. Much of the front of Hugging Face's Open LLM Leaderboard consisted of combinations of fine-tuned models, and tools such as mergekit let people create new models on CPUs.

The Stable Diffusion community had followed a similar path earlier. As this paper notes, enthusiasts' merges of several style models had at one point become some of the community's most popular choices.

At Tokyo's Sakana AI, Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha pointed to a problem. Many people treated merging as “black magic or alchemy”: which models to pick, how much to mix at each layer, and which method to use all depended on intuition. Most recipes were also tuned for the same familiar leaderboard benchmarks.

They wanted an evolutionary algorithm to choose the recipe.

Defining the problem

The authors separate merging into two independent spaces.

Parameter space, or PS: how to mix weights at each layer. This is conventional model merging, requiring identical architectures across source models.

Data-flow space, or DFS: keep each layer's weights unchanged, but alter which layers the data pass through. After model A's layer ii, for example, tokens might enter model B's layer jj. This resembles community “frankenmerge” layer stacking, but searches the entire inference path.

Both spaces are large. The authors ask whether evolutionary search can find better combinations than hand-crafted recipes, especially across domains, such as merging a Japanese model with an English mathematics model, where suitable settings are hard to choose intuitively.

Solving the problem

Original Figure 1, an overview of evolutionary model merging. Source models on the left include a blue model, such as a Japanese model, answering the first question correctly and the second incorrectly with accuracy 0.18, and a red model, such as a mathematics model, showing the opposite pattern with accuracy 0.31. On the right, parameter-space merging blends layers into purple and reaches 0.52. Data-flow merging preserves layer colors but connects blue and red layers into a new path, reaching 0.36. Combining both reaches 0.56.
Original Figure 1, Akiba et al., from PDF page 5.

The colors offer a useful analogy. Parameter-space merging mixes paint: blue and red become purple. Data-flow merging rearranges building blocks: each retains its color but changes position.

Parameter space: evolving coefficients

In parameter space, the authors use TIES-Merging with DARE. The framework can assign separate sparsification rates and mixture weights to each layer, including input and output embeddings and each Transformer block. CMA-ES evolves these hyperparameters using the target-task score, such as mathematics accuracy, as fitness.

That is the fine-grained setting permitted by the framework. The recipe chosen for the main results is simpler: one density and one weight per source model, giving 6 search variables across three models. The authors try finer layer groupings but obtain no clear improvement with the limited search data, so the most complex setting is not the main result.

Density controls how many task deltas survive DARE; weight controls their contribution to the merge. The three evolved weights are similar, yet sum to nearly 2, showing that useful solutions include amplifying task updates. Coefficients need not sum to 1, so the search extends beyond ordinary convex combinations.

Final density and weight settings for the three source models. Japanese Shisa has higher density than the two mathematics models, and all three weights make substantial contributions.
Original Figure 3, Akiba et al., from PDF page 10.

The horizontal axis lists sources; blue bars show retention density, and orange bars show merge weights. The Japanese model's higher blue bar means that more of its deltas need to survive. None of the orange bars suppresses a source close to zero. Both original specialties contribute.

CMA-ES maintains a search distribution. Each round samples recipes, ranks them by task score, and updates the distribution's center, scale, and orientation. A candidate is first merged, then generates answers on designated data for scoring. No gradients update the merged model. This allows direct search even with discrete fitness criteria such as “the final number is correct and the explanation is in Japanese.”

Data-flow space: evolving paths

Data-flow search is more involved. Suppose two models each have 32 layers, giving M=64M=64 source layers, and an inference path has length T=60T=60. Choosing any layer or a skip at each step gives (M+1)T(M+1)^T possibilities, too many to search freely.

Preliminary experiments narrow the space. Repeating or reordering early layers often hurts performance. The authors instead preserve source-layer order, repeat the full sequence rr times to obtain length T=M×rT = M\times r, and use an indicator array II to decide whether to execute each position. The space becomes 2T2^T: still large, but manageable for evolutionary search.

A second problem is input mismatch. A layer may receive representations from another model, with a distribution different from its training inputs. Preliminary experiments find that even swapping adjacent layers hurts performance. To compensate, whenever execution jumps from layer ii to layer jj, the authors multiply the input by a scaling coefficient WijW_{ij}, evolving this matrix too.

For example, concatenate A's 32 layers with B's 32, then repeat three times to obtain 192 candidate positions. Retain positions with positive indicators and skip the others. An original layer can be reused in another repetition. The chosen path is fixed: all tokens follow the same route, with no content-dependent expert router.

The search also fixes model A's tokenizer and input/output embeddings. Paths begin at A's first Transformer layer and end at its last to preserve the embedding interface. Scaling between originally adjacent layers of the same model stays at 1; search adjusts scales only for cross-model or skipped-layer transitions. Interfaces such as hidden size must remain compatible. DFS relaxes layer count and path structure while retaining those compatibility requirements.

The figure below compares paths at the beginning, middle, and end of search. The horizontal axis gives execution steps, the vertical axis identifies source layers, blue and red denote the two models, and color intensity represents scaling.

Source layers and execution orders selected early, midway, and at the end of DFS search. The final path preserves a continuous early segment, then interleaves layers from both sources.
Original Figure 4, Akiba et al., from PDF page 10.

The right plot initially follows nearly consecutive blue layers, adding red layers and skips later. This path uses the PS-merged model and the Japanese model as sources, producing a longer 10B network and thus changing the deployment architecture.

Combining the two

The spaces can be used in sequence. First merge a model in parameter space, return it to the source-model pool, then merge in data-flow space.

Discussion

A Japanese mathematics model

The main experiment builds a model that solves mathematics in Japanese. Three sources are fine-tuned from Mistral-7B-v0.1: the Japanese model Shisa Gamma 7B v1 and the English mathematics models WizardMath 7B v1.1 and Abel 7B 002.

Evaluation uses the 250-question Japanese MGSM test set, translated from the first 250 GSM8K test questions. Correct answers require both the right final number and Japanese reasoning. Evolution searches on 1,069 separately translated GSM8K test questions with zero-based IDs 250 through 1318, fully disjoint from final evaluation. The authors avoid translating GSM8K's training set because the open-source mathematics models already trained on it, making it unsuitable for this evaluation.

ModelSizeMGSM-JAJP-LMEH
Shisa Gamma 7B v1, Japanese7B9.666.1
WizardMath 7B v1.1, English mathematics7B18.460.1
Abel 7B 002, English mathematics7B30.056.5
Merge: parameter space7B52.070.5
Merge: data-flow space10B36.453.2
Merge: both10B55.266.2
Llama 2 70B70B18.064.5
Japanese StableLM 70B70B17.268.3
Swallow 70B70B13.671.5
GPT-3.5–50.4–
GPT-4–78.8–

Compiled from the original Table 1. MGSM-JA measures accuracy on Japanese mathematics questions; JP-LMEH is the average score on Japanese language evaluations.

All three sources struggle. The Japanese model lacks mathematics, and the mathematics models lack sufficient Japanese. All score 30 or below on MGSM-JA.

Parameter-space merging jumps to 52.0 while staying at 7B. Data-flow merging also improves performance, but reaches only 36.4 while growing to 10B. Combining both works best at 55.2, slightly above GPT-3.5's 50.4 and still far below GPT-4's 78.8.

On general Japanese ability, JP-LMEH, parameter-space merging reaches 70.5, above Japanese StableLM 70B at 68.3 and Llama 2 70B at 64.5, but below Swallow 70B at 71.5. This column supports the paper's claim of outperforming some prior 70B Japanese models.

The authors also plot which questions each model answers correctly. The merge retains questions its sources can solve and answers some that all three sources miss. Appendix examples require both Japanese cultural knowledge and mathematics. The authors argue that translating such questions into English for an English mathematics model may still fail because it lacks the background knowledge.

Compared with manual recipes and fine-tuning

How much does evolution contribute? Using the same source models, the authors build three unoptimized mergekit baselines with settings from Maxime Labonne's widely circulated Hugging Face blog tutorial:

MethodMGSM-JAJP-LMEH
TIES-Merging, tutorial defaults4.463.7
DARE-TIES, tutorial defaults35.266.3
Frankenmerge layer stacking, tutorial defaults0.016.1
Evolutionary merge: parameter space52.070.5
Evolutionary merge: both55.266.2
Best LoRA or full fine-tuning run on the same 1,069 questions43.255.9

Compiled from the original Table 3.

Default TIES settings almost erase mathematical capability, and layer stacking answers none of the questions correctly. DARE-TIES fares better, only slightly above the source models. With the same models and methods, recipe selection can create differences of tens of points.

Fine-tuning also trails in this comparison. The authors apply LoRA and full fine-tuning separately to all three sources using the same 1,069 questions as evolutionary search. The best reaches only 43.2. General Japanese ability also often falls substantially, which they interpret as evidence that fine-tuning needs more data and is vulnerable to forgetting.

They also add unrelated models to the candidate pool. With 8 unrelated models, parameter-space merging falls from 50.0 to 40.8 on MGSM-JA, still above every source model.

This unrelated-model experiment additionally uses weight normalization, so its 50.0 starting point should be compared separately from the main 52.0 result. Expanding the pool to eleven models increases the selection burden. The experiment shows that some unsuitable sources can be tolerated while preserving the importance of manual source selection.

The LoRA and full fine-tuning comparisons fix data size, search three learning rates, and train for 3 epochs. They answer which use of these 1,069 questions works better, rather than comparing against retraining on all sources' complete training data, most of which are unavailable. Evolutionary search also performs many evaluations, so its total computation must be distinguished from a single fine-tuning run.

Why can these models be merged?

An easily missed prerequisite is that all three sources derive from Mistral-7B-v0.1. Parameter-space merging still uses task-vector arithmetic; evolution searches the coefficients in the recipe.

The authors also discuss a difficulty. Shisa Gamma 7B v1's base underwent continued pretraining on 100 billion Japanese tokens, moving much farther from original Mistral weights than ordinary fine-tuning and making sparsification harder. This agrees with DARE's observation: larger deltas are harder to drop without damage when reducing interference.

A Japanese vision-language model

The same method also applies to vision-language models. A VLM passes an image through a vision encoder, then a projection network converts its output into representations accepted by a language model, which answers. The authors freeze the first two components and merge only LLaVA-1.6-Mistral-7B's language model with Shisa Gamma to create EvoVLM-JP. Both share the Mistral base; the vision encoder itself is not averaged.

Search uses a separate Japanese Visual Genome subset. Tests include 500 general VQA questions and a manually filtered Japanese cultural set with 42 images and 50 questions. The metric is Japanese ROUGE-L. Answers detected as non-Japanese generally score 0, with exceptions when the reference itself uses common foreign-language abbreviations such as UFO.

ModelSizeJA-VG-VQA-500Japanese cultural scenes
LLaVA-1.6-Mistral-7B8B14.341.1
Japanese Stable VLM8B–40.5
EvoVLM, PS8B19.751.2
EvoVLM, DFS12B16.846.5
EvoVLM, PS+DFS11B20.447.6

ROUGE-L, higher is better, compiled from the original Table 5. Japanese Stable VLM trained on Japanese Visual Genome, so that column is not evaluated.

PS+DFS performs best on general VQA, while PS wins on cultural scenes. Deeper paths and more complex combinations do not win every test. ROUGE-L measures textual overlap with references, together with the language constraint here, and covers only part of cultural understanding. Appendix examples involving carp streamers and Japanese terms for traffic-light colors help show which answers actually improve.

Contributions of paths and scaling

The authors also apply the method to Japanese and mathematics experts based on Llama-2-13B. Their MGSM-JA scores are 13.2% and 8.0%; PS reaches 31.2%, and PS+DFS reaches 34.0% with 21B parameters. These beat the two sources but trail the Mistral family's 7B/10B merges, illustrating the importance of source-model capabilities.

DFS source order also affects search. Reversing the two sources in the 13B experiment changes the test score from 23.2% to 12.8%. Initialization favors retaining A's original path, reducing candidates that generate gibberish at the outset but introducing an order preference. That is a limitation of narrowing the search space.

Removing cross-layer scaling from PS+DFS gives 26.4% on MGSM-JA, versus 34.0% for the full method. The resulting models also differ in size, at 19B and 21B. This supports a benefit from scale alignment, while the architecture change prevents attributing the whole gap to scaling alone in a fixed network. The authors also observe that a single scale becomes insufficient when extensive source fine-tuning makes representations too different to connect.

Costs and limitations

Evolutionary merging reuses trained models, avoiding gradient training to learn their capabilities again. For comparison, japanese-stablelm-base-gamma-7b underwent Japanese continued pretraining on 100 billion tokens, estimated at over ten thousand GPU hours. That estimate and measured evolutionary-search time are not presented as a same-hardware cost comparison.

PS performs 1,000 recipe trials, each scored on all 1,069 questions. DFS uses 128 candidates for 100 generations, reserves the final 200 questions of the training subset for validation, and evaluates batches of 200 questions during search. PS mixes weights before evaluation. DFS must also retain source layers and execute longer paths, requiring more GPU memory. That requirement prevents an unrelated-model DFS experiment. Generating and scoring many candidates is the expensive part.

The authors also state limitations: merges inherit source-model weaknesses and sometimes produce logically inconsistent answers. This study does not perform instruction fine-tuning or alignment, and outputs may contain factual errors.

Conclusion

This paper gives evolutionary algorithms the merging recipes that the community usually tunes by intuition. Parameter-space merging combines TIES and DARE; the main results search only each source model's density and weight. Data-flow merging rearranges the inference path. In the Japanese mathematics example, three underperforming 7B sources combine to exceed GPT-3.5.

The cost needs to be counted carefully. Merging itself requires no training, but evolution repeatedly evaluates candidates. Parameter-space search runs 1,000 trials, each scored on more than a thousand questions. A new task requires a new search.

To try this approach, start with a small source pool and model-wide density and weight search, retaining a test set entirely separate from search. Add layer-level freedom or DFS paths after establishing a benefit. That makes it easier to see what the additional evaluation buys.