Back to paper library
arXiv 2025TMLR 2026Model mergingPaper notes

In-the-Wild Merging

Survival in the wild

Original paper

A Systematic Study of In-the-Wild Model Merging for Large Language Models

Paper authors

Oğuz Kağan Hitit · Leander Girrbach · Zeynep Akata

First submitted to arXiv
arXiv:2511.21437
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{modelmerginginthewildmerging2025,
  title = {{A Systematic Study of In-the-Wild Model Merging for Large Language Models}},
  author = {Oğuz Kağan Hitit and Leander Girrbach and Zeynep Akata},
  year = {2025},
  eprint = {2511.21437},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.21437}
}
On this page

Model-merging papers usually design tidy experiments. Each expert is fine-tuned on a distinct task, such as eight image-classification datasets or seven NLP tasks, and the merge is tested on those tasks individually.

The real world is messier. Open Hugging Face and a single Llama or Qwen base may have hundreds or thousands of community fine-tunes. Some do mathematics, some role-play, and some simply use another instruction dataset. Their objectives can overlap or conflict. The fine-tuned models accumulated within a company may look much the same.

Oğuz Kağan Hitit, Leander Girrbach, and Zeynep Akata from Koç University, the Technical University of Munich, Helmholtz Munich, and MCML ask a practical question: can merging these “wild” fine-tunes produce a model better than its base, or even better than every individual fine-tune?

Defining the problem

The paper defines its objective precisely. It merges heterogeneous, potentially conflicting fine-tunes and evaluates general language-model ability, only partly covered by the sources' individual training tasks.

The authors ask three questions:

  1. Can simply merging multiple fine-tunes outperform the base model?
  2. Which weight-interpolation-based merging methods can achieve this?
  3. Do “subspace” merging methods that recently performed well on vision models also work for in-the-wild LLM merging?

A merge beating both the base and individual fine-tunes offers evidence of what the paper calls constructive interference. Evaluation has two comparisons: against the base, and against the fine-tune with the highest average score in the entire pool. This strongest fine-tune is a fixed reference and may be absent from a particular sampled subset.

Solving the problem

Original Figure 1, the evaluation protocol. On the left, four base models, Llama-3.2 3B, Llama-3.1 8B, Qwen3 4B, and Qwen3 8B, each have 12 public fine-tunes. Subsets of 2, 4, up to 12 fine-tunes are sampled, then merged with mergekit using three task-arithmetic-family methods, task arithmetic, TIES-Merging, and Model Stock, and three subspace methods, TSV-Merge, Iso-C, and Subspace Boosting. On the right, lm-eval-harness scores benchmarks including TruthfulQA, MMLU, MedMCQA, and HellaSwag.
Original Figure 1, Hitit et al., from PDF page 3.

This paper also introduces no new method. Its contribution is a repeatable evaluation protocol.

Bases and fine-tunes: four open-weight models, Llama-3.2-3B-Instruct, Llama-3.1-8B-Instruct, Qwen3 4B, and Qwen3 8B, each paired with 12 public Hugging Face fine-tunes covering different objectives and domains. The Llama sources start from Instruct models. Each family shares its own base; the four families are not mixed with one another.

Merging methods: six methods in two groups. The task-arithmetic family includes task arithmetic with λ=1.0\lambda=1.0, TIES-Merging with λ=0.1\lambda=0.1 and the largest 10% of values retained per layer, and Model Stock. The subspace family includes TSV-Merge, Iso-C, and Subspace Boosting. All use mergekit, with hyperparameters selected beforehand in the appendix and then fixed.

Sampling: the number of fine-tunes nn is 2, 4, 6, 8, 10, or 12. Because there are too many combinations, the authors uniformly sample 15 subsets per nn, except n=12n=12, which has only one. Every method uses the same subsets, isolating algorithmic differences.

Evaluation: lm-evaluation-harness runs 16 standard Open LLM Leaderboard benchmarks, including ARC, HellaSwag, Winogrande, MMLU, GPQA, BBH, MMLU-Pro, and TruthfulQA, without chat templates, reporting accuracy.

Most tasks use 0-shot prompting; BBH uses 3-shot and MMLU-Pro 5-shot. Evaluation uses float16, with batch size selected automatically by the harness. This fixes prompting for multiple-choice and knowledge tests, while leaving complete conversations, tool use, and specific role-playing requirements unmeasured. A fine-tune weaker on the average score may still serve its original purpose better.

Match task-arithmetic scales first

One easily missed setting is that mergekit normalization is enabled. For source ii, let its difference be δi\delta_i and recipe weight αi\alpha_i. The actual merged update is

Δnorm=∑iαiδi∑iαi,θm=θ0+λΔnorm.\Delta_{\text{norm}}=\frac{\sum_i\alpha_i\delta_i}{\sum_i\alpha_i}, \qquad \theta_m=\theta_0+\lambda\Delta_{\text{norm}}.

With αi=1\alpha_i=1 and λ=1\lambda=1, this averages task deltas. Directly summing twelve fine-tunes' deltas would produce an update twelve times larger. Reproduction requires the normalization setting as well as “TA, λ=1\lambda=1.”

The appendix measures this too. With normalization enabled, performance is fairly flat as λ\lambda ranges from 0.1 to 1.9, degrading clearly only at 10. Without it, increasing λ\lambda quickly hurts performance even within the ordinary search range. This gives “simple task arithmetic is most reliable” a more precise operational scope: the result uses averaging and scaling.

Both original plots use λ\lambda horizontally and mean accuracy vertically, with colors for the four bases. The upper plot enables normalization and adds λ=10\lambda=10 at the far right; the lower disables it. Horizontal positions represent listed coefficient settings, whose actual spacing differs, especially between the two rightmost points.

With normalization enabled, task arithmetic remains fairly stable across the usual coefficient range for all four bases; a coefficient of ten severely degrades some models.
Original Figure 11, Hitit et al., from PDF page 26.
Without normalization, mean merged accuracy for the four bases drops substantially as the coefficient increases from 0.1.
Original Figure 12, Hitit et al., from PDF page 26.

The lower decline results from summing and then amplifying multiple updates, very different from the averaged recipe above. Omitting one Boolean setting during reproduction can reverse the curve.

Discussion

Only task arithmetic consistently improves

Methodn=1n=1n=2n=2n=4n=4n=6n=6n=8n=8n=10n=10n=12n=12
Task arithmetic25 / −1.1149 / −0.0288 / +0.50100 / +0.74100 / +0.87100 / +0.93100 / +1.02
Model Stock25 / −1.1153 / +0.0277 / +0.0682 / +0.0982 / +0.1080 / +0.11100 / +0.13
TIES-Merging25 / −1.1178 / +0.1185 / +0.1487 / +0.1885 / +0.1780 / +0.1775 / +0.13

Averaged across the four bases, compiled from the original Table 1. Each cell gives the percentage of combinations beating the base, followed by the paper's reported mean relative improvement (%). n=1n=1 is an individual fine-tune.

Start with n=1n=1. Only about a quarter of individual fine-tunes beat their own base, and the mean drops by 1.11%. Later improvements therefore cannot simply be attributed to drawing one particularly strong fine-tune.

Task arithmetic has the clearest trend. With two fine-tunes, about half of combinations beat the base. From n=6n=6 onward, every sampled combination across all four models beats its base. Mean improvement grows with nn, reaching about 1% at n=12n=12.

Here, 100% refers to all combinations sampled in this study. Each family tests only 15 subsets at n=6n=6 and one at n=12n=12, rather than every possible Hugging Face fine-tune combination. Both reported quantities matter: a high share of base-beating combinations can coexist with only a tiny average gain. They answer separate questions about reliability and improvement magnitude.

The gains are small. The authors summarize that, averaged across tasks, most improvements remain below 1%.

TIES-Merging also often beats the base, but gains quickly plateau around 0.1 to 0.2%. On Qwen3 8B, more fine-tunes make it worse: +0.16 at n=4n=4 becomes −0.08 at n=12n=12. The authors argue that TIES is sensitive to task-vector magnitudes among heterogeneous fine-tunes. Model Stock stays very close to the base, with weights that barely move away from it.

Does it beat the best individual model?

Beating the base is only one comparison. The authors also ask whether merges beat the best fine-tune in the full pool. Task arithmetic does so for three of the four models, with generally more favorable outcomes for both Qwen families and Llama 3.2 3B when n≥4n\ge4.

The exception is Llama 3.1 8B. Its pool contains one particularly strong fine-tune. Merging improves over the base but still trails that fixed reference. In practice, keep “select an existing model” as a baseline; otherwise, merging and evaluation may only bring you closer to an option already available.

Subspace methods: more merging, worse results

Over the preceding year or two, subspace methods such as TSV-Merge and Iso-C performed well on vision-model merging benchmarks. Rather than simply adding in full weight space, they decompose each layer's task updates with singular value decomposition and handle interference in low-rank subspaces. TSV-Merge retains each task's most important singular directions and orthogonalizes directions across tasks; Iso-C flattens the singular-value distribution.

Their transfer to in-the-wild LLM merging is disappointing. TSV-Merge and Iso-C accuracy declines as more fine-tunes are merged. The authors argue that truncation and orthogonalization progressively discard useful information during multi-model merging.

They also measure how far merged weights move from the base. Task-vector norms, ∥θmerged−θbase∥2\lVert\theta_{\text{merged}} - \theta_{\text{base}}\rVert_2, generally stay within 50 for task arithmetic, TIES, and Model Stock. Iso-C and TSV-Merge often reach 100 to 300, exceeding 300 in the two 8B models. Greater distance correlates with greater degradation. The authors interpret this as moving weights too far, outside the base model's basin, and causing catastrophic forgetting.

For a layer of shape m×dm\times d with update ΔW\Delta W, SVD gives UΣV⊤U\Sigma V^\top. TSV-Merge retains roughly 1/n1/n of each source's singular directions, then combines and orthogonalizes them; as nn grows, fewer directions survive per source. Iso-C decomposes the summed update and replaces every singular value with their mean, strengthening weaker directions relatively. Both alter direction combinations or their strength distribution substantially compared with averaging original deltas.

Subspace Boosting does not fail in the same way

The third subspace method deserves separate treatment. Subspace Boosting starts from a merged update, selects a cutoff based on cumulative singular values, and raises weaker singular values beyond it to the cutoff value. It can follow TA or TIES, so results are reported as TA+SB and TIES+SB, rather than as an independent model-selection procedure.

The paper fixes its threshold at 0.2. TA+SB varies greatly with few fine-tunes: at n=2n=2, only 23% of combinations beat the base on average. At n=12n=12, that becomes 100%, with reported mean improvement +1.07. TIES+SB is steadier and reaches +0.36 with twelve fine-tunes, versus −2.36 for TSV-Merge and −5.36 for Iso-C in the same setting. Subspace methods therefore have different outcomes and cannot all be described as worsening with more models. TA+SB largely follows TA's trend, with no clear, consistent additional advantage from the singular-value-spectrum operation.

A cleaner source pool

The appendix separately selects three mathematics and three medical experts for Qwen3 4B and 8B, comparing same-domain merges of three against cross-domain merges of six. Evaluation switches to corresponding mathematics and medical questions. At 8B, mixed-domain merging performs similarly to separate domain merges. TA and TIES also preserve performance at 4B, while some subspace variants show larger gaps.

This makes the main experiment more informative. A random pool may combine overlapping updates pointing in different directions. Establishing each source's strengths and selecting a purposeful combination can give different results. The main experiment deliberately omits alignment, clustering, and candidate filtering, so its findings do not settle the value of every carefully selected recipe.

Distance analysis also belongs within this scope. Within a family, larger update norms correlating with poorer scores provide a useful anomaly signal. Across parameter counts or different network normalization, absolute L2L_2 values cannot share a universal cutoff of 50 or 300. The authors use distance to support their hypothesis of leaving a favorable region; they do not directly measure loss barriers along every candidate's path.

Conclusion

The study's conclusion has two parts.

The encouraging result is that community fine-tunes with varied, potentially conflicting objectives can still merge into a model better than its base. The oldest, simplest task arithmetic does it, becoming more reliable as more fine-tunes are included.

The gain remains limited. TIES and Model Stock stay close to the base, some subspace recipes degrade, and TA+SB largely follows TA's trend. Source-pool selection, normalization, and fixed evaluation prompts all matter when interpreting these scores.

If you have a collection of fine-tunes from the same base with varied origins, try task arithmetic first. Before moving to a more elaborate method, measure how far the merged weights have moved from the base.