Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch
- Publication
- ICML 2024Conference
- First submitted to arXiv
- arXiv:2311.03099
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{modelmergingdare2023,
title = {{Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch}},
author = {Le Yu and Bowen Yu and Haiyang Yu and Fei Huang and Yongbin Li},
year = {2023},
eprint = {2311.03099},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2311.03099}
}
By the second half of 2023, Hugging Face hosted many models fine-tuned from the same Llama 2. WizardLM specialized in instruction following, WizardMath in mathematics, and various others in coding.
The most direct way to combine those abilities is to pool the data and fine-tune again. But many models release only weights, leaving their training data unavailable. Even with the data, retraining 13B or 70B models is expensive.
Alibaba's Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li liken this to Super Mario collecting power-ups. Could a model directly “absorb” abilities from models of the same lineage, without retraining or even a GPU?
The paper first answers a more basic question: how much do weights actually change during fine-tuning?
Defining the problem
Let be pretrained weights and the weights after supervised fine-tuning, or SFT, on task . Their difference is called the delta parameters:
This is the task vector in task arithmetic. TIES-Merging has already identified many delta parameters as redundant: they dilute important updates from other models during merging. Its approach trims by magnitude, retaining only the largest 20%.
The observation here goes further. Measuring delta parameters in several language models, the authors find that most SFT changes are within 0.002, with extensive redundancy. Could a model still work if most were simply discarded?
Solving the problem
Drop, then rescale
DARE's name states the procedure: Drop And REscale.
Each delta parameter is zeroed with probability . Survivors are multiplied by . The final model is .
Dropping is random and ignores magnitude, distinguishing it from TIES and conventional pruning.
Why rescale?
Consider a linear layer with input , pretrained weights , bias , and deltas and . Output is
DARE changes only the delta component. Each survives with probability and is multiplied by , or becomes 0 with probability . The expected delta contribution is therefore
With , the expectation returns to the original . This identity applies to one linear layer with fixed inputs. A full network includes nonlinearities, and random changes in earlier layers alter later inputs, so it requires separate measurement.
The authors call this a “rough proof.” An unchanged expectation gives no bound on the error of a single draw; experiments must establish how much can be dropped.
Two delta values illustrate this. The following enumerates all four equally likely masks at , without training a network:
import numpy as np
from itertools import product
delta = np.array([0.001, 0.002])
x = np.array([1., 2.])
p = 0.5
outputs = []
for dropped in product([0, 1], repeat=2):
kept_delta = (1 - np.array(dropped)) * delta / (1 - p)
outputs.append(float(kept_delta @ x))
print("各遮罩的 delta 輸出:", np.round(outputs, 6).tolist())
print(f"原本: {delta @ x:.6f}, 遮罩平均: {np.mean(outputs):.6f}")
各遮罩的 delta 輸出: [0.01, 0.002, 0.008, 0.0]
原本: 0.005000, 遮罩平均: 0.005000
The average of the four results is correct, but a single draw may give 0 or twice the original output. DARE samples its mask once when creating the model. Deployment neither resamples on every inference nor runs all four models. Preserving capabilities therefore also depends on the size of the random perturbation.
Take the example one step further. For a fixed delta value , the variance after DARE is
This follows directly from the Bernoulli mask. The multiplier is 9 at and 99 at . Larger deltas and higher drop rates make a single draw harder to keep close to the original. For fixed inputs and independent masks in a linear layer, output-perturbation variance accumulates terms . This calculation helps explain the failures below; the relationship between model size and tolerable still requires experiments.
A preprocessing step for merging
DARE itself does not merge models. It is a plug-in preprocessing step: apply DARE separately to each source model, then use an existing merging method. With task arithmetic:
The idea is that each model retains only a small fraction of nonzero deltas, reducing overlap and thus interference during merging.
With independent masks for two models, the probability that both retain a coordinate is . At , only about 1% of coordinates that both deltas could originally modify survive in both. This explains how sparsification reduces direct overlap. Surviving values are also scaled up tenfold, so merge quality depends on overlap, scale, and source-model capabilities together.
Discussion
How much can be dropped?

The left plot shows how dropping affects WizardMath. All three sizes change little with 90% dropped. At 99%, 7B and 13B fail to hold up, while 70B remains strong.
The authors draw two observations: SFT deltas can be redundant enough to discard 90% or even 99%, and larger models tolerate higher drop rates. Encoder models BERT and RoBERTa show a similar trend on GLUE, with substantial declines only beyond 0.9.
The sweep uses . Decoder instruction following is measured by AlpacaEval win rate, mathematics by GSM8K and MATH accuracy, and coding by HumanEval and MBPP pass@1, with greedy decoding. Encoder experiments use BERT-base and RoBERTa-base on eight GLUE tasks, with accuracy, F1, correlation, or other task-specific metrics. “Preserved performance” must be read against each plot's metric. WizardMath-70B's high tolerance on GSM8K cannot simply be transferred to every expert.
The authors reserve another 10% of GLUE's original training sets for validation and use the official validation sets as tests. This permits selecting drop rates on held-out data. The extreme 99% result demonstrates an achievable case; practical sweeps can begin at lower rates.
Merging three 13B models
The authors merge three models fine-tuned from Llama-2-13b: WizardLM-13B for instruction following (LM), WizardMath-13B for mathematics (Math), and llama-2-13b-code-alpaca for coding (Code). They exclude WizardCoder-Python-13B because it was fine-tuned from CodeLlama-13b-Python, a different base.
| Model | DARE | AlpacaEval | GSM8K | HumanEval | MBPP |
|---|---|---|---|---|---|
| LM | – | 67.20 | 2.20 | 36.59 | 34.00 |
| Math | – | – | 64.22 | – | – |
| Task arithmetic: LM + Math | No | 67.04 | 66.34 | 28.66 | 30.60 |
| Task arithmetic: LM + Math | Yes | 67.45 | 66.26 | 26.83 | 32.40 |
| TIES: LM + Math | No | 68.63 | 15.77 | 37.80 | 35.60 |
| TIES: LM + Math | Yes | 68.70 | 36.16 | 36.59 | 37.00 |
| Task arithmetic: LM + Math + Code | No | 69.03 | 58.45 | 18.29 | 29.80 |
| Task arithmetic: LM + Math + Code | Yes | 69.28 | 56.48 | 23.17 | 31.60 |
Selected results from the original Table 1. “–” marks an unevaluated model–metric pair.
The striking comparison is between the first and third rows. WizardLM scores only 2.20 on GSM8K. Merging with WizardMath through task arithmetic gives 66.34, above WizardMath's own 64.22. This works without DARE; adding it gives a similar 66.26.
DARE helps TIES most clearly. TIES's magnitude-based trimming damages mathematical capability in these models, leaving only 15.77 on GSM8K. Applying DARE first restores it to 36.16. The authors explain that most deltas are already zero after DARE, so TIES trims those zeros rather than useful values.
This TIES setting also differs from its original paper's fixed recipe. The DARE paper's decoder experiments search retention fractions 0.5, 0.7, and 0.9 and scaling coefficients 0.5 and 1.0, comparing results with and without DARE. The improvement is established for these source models and search settings; the 15.77 score alone cannot characterize TIES on other tasks.
DARE does not improve every result. With three models, GSM8K falls from 58.45 to 56.48. Across five merging methods for encoder models, average gains range only from −0.03% to 0.84%, and merged models often trail individual experts.
The authors also notice that llama-2-13b-code-alpaca is weaker at coding than WizardLM-13B. They conclude that each source model must itself be well trained for merging to work well.
Rescaling is necessary
To isolate rescaling's role, the authors introduce DropOnly, which drops values without rescaling, and compare layerwise hidden representations of the original and modified models under identical inputs.
The left plot below uses GSM8K inputs for WizardMath-7B; the right uses CoLA inputs for BERT. The horizontal axis progresses through layers, and the vertical axis measures mean cosine similarity to the original model's representations. Compare DARE and DropOnly at the same drop rate.

With 90% dropped, DARE keeps cosine similarity above 0.95 at every layer. DropOnly holds up only at . On WizardMath-7B, similarity falls to roughly 0.85 at and 0.68 at . Performance follows the same pattern: higher drop rates widen the gap between DropOnly and DARE.
Another baseline applies magnitude pruning, or MP, then rescales. Dropping 70% in 7B models reduces AlpacaEval from 43.85 to 10.61, GSM8K from 46.70 to 0.37, and HumanEval from 21.34 to 3.05. The authors argue that retaining only large values and then amplifying them destabilizes outputs.
When it fails
DARE relies on small deltas. The authors deliberately create a failure by computing WizardCoder-Python-13B's deltas against Llama-2-13b instead of its actual base, CodeLlama-13b-Python. CodeLlama underwent another 500 billion code-related tokens of training beyond Llama 2, so its parameters have moved substantially. Deltas against Llama 2 often exceed 0.01 in absolute value, while those against CodeLlama-13b-Python stay within 0.0002.
Dropping just 10% then sends HumanEval and MBPP pass@1 from 63.41 and 55.4 to 0. With continued pretraining, deltas can reach roughly 0.03, where DARE becomes unsuitable.
The observed 0.002 range belongs to this model collection, rather than a universal safety threshold across architectures. Parameter scales vary by model and layer, and inputs also amplify output perturbations. A useful check computes the delta distribution against the correct base, then measures retention curves across . Matching “13B” labels or tensor shapes alone leaves shared-base compatibility unverified.
The authors also note that dropping full fine-tuned weights rather than deltas makes performance approach zero even at 10%. Their interpretation is that SFT mainly elicits capabilities already present in the pretrained model rather than adding entirely new ones.
Engineering also needs to distinguish sparse deltas from a smaller model. Reconstructed weights are , and the base remains a complete dense model. Dropping nine-tenths of the differences does not automatically remove nine-tenths of inference parameters or matrix multiplication. Storing “a shared base plus sparse deltas” may save storage across versions, but requires appropriate formats and loading support. The paper primarily demonstrates capability retention and merging; it does not compress a 70B network into a 0.7B network.
Onto the leaderboard
Finally, the authors merge several Mistral-7B-based models with DARE to create supermario v1 and v2. As of January 28, 2024, supermario v2 ranks first in the Open LLM Leaderboard's 7B category with an average of 75.49.
The neighboring Beagle14-7B and NeuralBeagle14-7B also identify mergekit in their model cards. Merged models crowded the front of the 7B leaderboard at the time, and their scores began attracting concerns about data contamination.
Conclusion
DARE's finding is more interesting than the method itself. SFT weight changes are tiny; randomly dropping most and rescaling the survivors can preserve the model's usefulness. Larger models can tolerate more dropping.
As merging preprocessing, it often reduces interference, especially with TIES. Sometimes the benefit is only a fraction of a percentage point, and occasionally performance declines.
Before using it to merge models, check that every delta is computed against the actual base. With the wrong base, dropping just a tenth can wipe out performance.