Back to paper library
arXiv 2023NeurIPS 2023Model mergingPaper notes

Tangent Task Arithmetic

Stay in your lane

Original paper

Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models

Paper authors

Guillermo Ortiz-Jimenez · Alessandro Favero · Pascal Frossard

Publication
NeurIPS 2023Conference
First submitted to arXiv
arXiv:2305.12827
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{modelmergingtangenttaskarithmetic2023,
  title = {{Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models}},
  author = {Guillermo Ortiz-Jimenez and Alessandro Favero and Pascal Frossard},
  year = {2023},
  eprint = {2305.12827},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.12827}
}
On this page

Task arithmetic showed us that subtracting the weights before fine-tuning from the weights after gives a “task vector,” and that these vectors can be added and subtracted. Add the vectors of two tasks and the model can do both; subtract one and the model forgets that task.

It works well, but why it works was never quite clear.

The popular explanation at the time went like this. When a large pretrained model is fine-tuned, its weights move only a little, and within that small neighborhood the network is roughly linear. If the network is linear, then adding weights naturally adds outputs.

Ortiz-Jimenez, Favero and Frossard at EPFL decided to check this explanation carefully. They found that it was wrong, but by following the wrong lead further, they arrived at a better method.

Defining the problem

What task arithmetic needs

First, let's spell out what “task arithmetic works” means. Let θ0\theta_0 be the pretrained weights, θt⋆\theta_t^\star the weights after fine-tuning on task tt, and τt=θt⋆−θ0\tau_t = \theta_t^\star - \theta_0 the task vector. Each task has its own input domain Dt\mathcal{D}_t, such as EuroSAT's satellite images or SVHN's house numbers, and the input domains of different tasks do not overlap.

The authors write task arithmetic as a property:

f(x; θ0+∑t=1Tαtτt)={f(x; θ0+αtτt)x∈Dtf(x; θ0)x∉⋃tDtf\Big(x;\ \theta_0+\sum_{t=1}^{T}\alpha_t\tau_t\Big)= \begin{cases} f(x;\ \theta_0+\alpha_t\tau_t) & x\in\mathcal{D}_t\\[2pt] f(x;\ \theta_0) & x\notin\bigcup_t\mathcal{D}_t \end{cases}

In plain terms, after several task vectors are added at once, the model should behave on satellite images exactly as if “only the satellite-image vector had been added,” and on inputs that belong to no task, exactly like the pretrained model. Each vector changes only its own patch and leaves the others alone.

The linearity hypothesis

Around θ0\theta_0, a neural network can be expanded as a first-order Taylor series:

f(x;θ)≈f(x;θ0)+(θ−θ0)⊤∇θf(x;θ0)f(x;\theta) \approx f(x;\theta_0) + (\theta-\theta_0)^\top\nabla_\theta f(x;\theta_0)

The right-hand side is linear in θ\theta and equivalent to a kernel method whose kernel is the neural tangent kernel (NTK) k(x,x′)=∇θf(x;θ0)⊤∇θf(x′;θ0)k(x,x')=\nabla_\theta f(x;\theta_0)^\top\nabla_\theta f(x';\theta_0).

If fine-tuning really happened inside this linear regime, adding τ1+τ2\tau_1+\tau_2 would change the output by the sum of the two tasks' individual changes, and task arithmetic would appear to hold automatically.

So is CLIP fine-tuning actually linear?

Solving the problem

Check one: fine-tuning is not linear

If fine-tuning were linear, plugging the fine-tuned task vector τ\tau into the linear approximation should give results close to the original fine-tuned model. The authors call this post-hoc linearization:

flin(x;θ0+τ)=f(x;θ0)+τ⊤∇θf(x;θ0)f_{\text{lin}}(x;\theta_0+\tau) = f(x;\theta_0) + \tau^\top\nabla_\theta f(x;\theta_0)

The experiments follow the setup of the original task arithmetic paper, fine-tuning CLIP's ViT-B/32, ViT-B/16 and ViT-L/14 on eight datasets, namely Cars, DTD, SUN397, EuroSAT, GTSRB, MNIST, SVHN and RESISC45.

Post-hoc linearization turns out to have clearly lower single-task accuracy than the original fine-tuned models, for ViTs of all three sizes. The authors call this gap the “non-linear advantage.” Fine-tuning does not stay inside the linear regime.

There is another possibility. The non-linear part might matter for individual tasks while task arithmetic only uses the linear part. The authors check this too. They take task vectors from non-linear fine-tuning and add all eight tasks on the post-hoc linearized model:

ModelViT-B/32ViT-B/16ViT-L/14
Pretrained (zero-shot)48.455.264.4
Non-linear fine-tuning71.475.585.1
Post-hoc linearization57.165.075.2

The table gives mean absolute accuracy (%) over the eight tasks. Post-hoc linearization loses more than 10 percentage points, so task arithmetic clearly relies on the non-linear part as well. The linearity hypothesis does not hold.

But the table also hides something odd. Switch to normalized accuracy, dividing by the accuracy of each task's own single-task fine-tuned model, and post-hoc linearization actually comes out ahead. On ViT-B/32, non-linear fine-tuning scores 76.5% and post-hoc linearization 81.9%. Task forgetting tells the same story, with post-hoc linearization forgetting more cleanly.

The linearized model does worse on every individual task, yet it does better at “how much survives the addition.”

Weight disentanglement

The authors argue that the condition task arithmetic needs is weight disentanglement:

f(x; θ0+∑tαtτt)=∑tgt(x; αtτt)+g0(x)f\Big(x;\ \theta_0+\sum_t\alpha_t\tau_t\Big)=\sum_t g_t(x;\ \alpha_t\tau_t)+g_0(x)

Here gtg_t is nonzero only on Dt\mathcal{D}_t, and g0g_0 is nonzero only outside every task domain. The model's output can be split into several non-overlapping local functions, each controlled only by its own task vector.

This condition is equivalent to the task arithmetic property above, and it does not require linearity; gtg_t can be a function of any shape in the parameters. The authors also stress that weight disentanglement is about “not interfering with each other,” which is a separate question from how well each task is performed.

To measure how disentangled two tasks are, the authors define the disentanglement error between them:

ξ(α1,α2)=∑t=12Ex∼μt[1(f(x;θ0+αtτt)≠f(x;θ0+α1τ1+α2τ2))]\xi(\alpha_1,\alpha_2)=\sum_{t=1}^{2}\mathbb{E}_{x\sim\mu_t}\Big[\mathbb{1}\big(f(x;\theta_0+\alpha_t\tau_t)\neq f(x;\theta_0+\alpha_1\tau_1+\alpha_2\tau_2)\big)\Big]

On task tt's data, this is the fraction of inputs on which “adding only its own vector” and “adding both vectors” give different predictions. The smaller the error, the less the two vectors interfere with each other.

Original Figure 3. Six heat maps, with α1 on the horizontal axis and α2 on the vertical axis, both from −3 to 3; color shows the disentanglement error ξ, light for small error and dark for large. The three columns are the task pairs EuroSAT and SVHN, DTD and SUN397, and Cars and RESISC45. The top row is non-linearly fine-tuned CLIP ViT-B/32, where the light region is confined to a small area around the origin; the bottom row is the post-hoc linearized model, where the light region is clearly larger. A red dashed box marks the range used when searching for the best coefficients.
Original Figure 3, Ortiz-Jimenez et al., from PDF page 6, arXiv license, scholarly quotation.

The center of each plot is the pretrained weights θ0\theta_0, and the two red arrows are the two task vectors. In the top row, the non-linear model is light only in a small patch near the origin, and the error climbs quickly once α\alpha goes beyond 1. This explains two empirical observations in the original task arithmetic paper, that the best coefficient is usually below 1 and that fine-tuning calls for a small learning rate and few steps.

In the bottom row, the post-hoc linearized model has a much larger light region. Linearization makes task vectors interfere less with one another, which is why its normalized accuracy is higher.

Now the problem is clear. The linearized model disentangles well but is too weak on individual tasks; the non-linear model is strong on individual tasks but disentangles less well. Can we have both?

Fine-tuning directly in the tangent space

The authors' answer is to skip linearizing after the fact and instead fine-tune the linearized model from the very beginning.

Original Figure 4. A conceptual diagram. The curved teal surface N represents all neural network functions, and the orange plane K is the tangent space touching the surface at f(θ0). Three dashed paths start from f(θ0): the blue one follows the surface to the non-linearly fine-tuned f(θ*); projecting f(θ*) perpendicularly onto the plane gives the post-hoc linearized flin(θ*); the orange-red one goes straight across the plane to the linearly fine-tuned flin(θ*lin).
Original Figure 4, Ortiz-Jimenez et al., from PDF page 6, arXiv license, scholarly quotation.

This figure puts the three approaches side by side. The curved surface N\mathcal{N} is the set of all network functions, and the plane K\mathcal{K} is the tangent space at θ0\theta_0. Standard fine-tuning travels along the surface to f(θ⋆)f(\theta^\star). Post-hoc linearization projects that endpoint onto the plane, which is why it loses accuracy. Linearized fine-tuning stays on the plane from start to finish, and what it optimizes for is the best point on the plane, θlin⋆\theta^\star_{\text{lin}}.

The two linear models share the same kernel, but the task vector τlin=θlin⋆−θ0\tau_{\text{lin}}=\theta^\star_{\text{lin}}-\theta_0 is optimized directly for the linear model, so its single-task performance is much better.

In practice, a linearized model is a network that “keeps θ0\theta_0 fixed and trains the displacement.” PyTorch's torch.func.jvp computes both f(x;θ0)f(x;\theta_0) and the Jacobian times the displacement in a single forward pass. Let's use a small MLP to see when the linear approximation is accurate:

python
import torch
from torch.func import functional_call, jvp

torch.manual_seed(0)
net = torch.nn.Sequential(torch.nn.Linear(4, 16), torch.nn.Tanh(), torch.nn.Linear(16, 2))
theta0 = {k: v.detach().clone() for k, v in net.named_parameters()}
x = torch.randn(8, 4)

def f(params):
    return functional_call(net, params, (x,))

def f_lin(params):
    delta = {k: params[k] - theta0[k] for k in params}
    out, dp = jvp(f, (theta0,), (delta,))     # f(θ0) and J·(θ − θ0), both from a single forward pass
    return out + dp

direction = {k: torch.randn_like(v) for k, v in theta0.items()}
for scale in [0.01, 0.1, 1.0]:
    theta = {k: theta0[k] + scale * direction[k] for k in theta0}
    gap = (f(theta) - f_lin(theta)).abs().max().item()
    print(f"displacement scale {scale:<4}  max gap between original and linearized model {gap:.5f}")
text
displacement scale 0.01  max gap between original and linearized model 0.00152
displacement scale 0.1   max gap between original and linearized model 0.13638
displacement scale 1.0   max gap between original and linearized model 8.19715

With a small displacement the two are nearly identical; as the displacement grows, the gap widens fast. The linear model has no say in where standard fine-tuning ends up. Linearized fine-tuning instead ties the model to the tangent space, so however far it travels, the relationship between weights and outputs stays linear.

In the paper's experiments, the authors linearize only CLIP's image encoder; the text encoder is frozen anyway. The training setup is exactly that of standard fine-tuning, with 2,000 steps, batch size 128, learning rate 10−510^{-5} and AdamW with a cosine schedule, and no extra hyperparameter tuning.

Discussion

Addition and subtraction both improve

The last row is linearized fine-tuning:

Task addition (absolute/normalized %)ViT-B/32ViT-B/16ViT-L/14
Non-linear fine-tuning71.4/76.575.5/80.085.1/88.8
Linearized fine-tuning76.5/85.481.3/86.088.5/93.5

Mean absolute accuracy after adding the eight tasks rises by 5.1, 5.8 and 3.4 percentage points for the three models.

Task forgetting is evaluated by subtracting a task vector and seeing how low the target task can be pushed while the control task, ImageNet, keeps at least 95% of its pretrained accuracy. On ViT-B/32, non-linear fine-tuning pushes the target task down to 24.0% and linearized fine-tuning to 10.9%, forgetting 13.1 percentage points more; on ViT-L/14 the figure goes from 18.0% to 7.9%. Accuracy on the control task is almost the same.

Coefficients are chosen as in the original paper, with a single α\alpha shared by all tasks, searched from 0 to 1 in steps of 0.05, and the best value picked on a small portion of the training data.

The benefit of linearization is largest for small models and shrinks as models grow. The appendix also finds that larger models are more disentangled to begin with.

The cost

Linearization does not come free. The appendix explains that at inference, every operation needs an extra derivative computation, so it takes about two to three times as long as the original model; during training, the computation and memory for gradients roughly double. The authors describe this as “only a constant-factor increase,” but for large models, two to three times the inference cost shows up directly in deployment.

Later research picked up from exactly this point. Partial Linearization, for example, linearizes only LoRA's adapters, confining the cost to a small fraction of the parameters.

Why disentanglement happens

The authors close with an explanation from the kernel-method point of view. A kernel can be decomposed into a set of eigenfunctions ϕρ\phi_\rho. If each task uses only eigenfunctions that are “nonzero only inside its own input domain,” different tasks will not interfere with one another.

They check this on ViT-B/32. After linearized fine-tuning on RESISC45, they eigendecompose its NTK and compute the total energy of the eigenfunctions at each input point. RESISC45 data points have clearly higher energy, while Cars data points, used as a control, have very low energy. The eigenfunctions that represent RESISC45 really are concentrated on RESISC45's data.

The authors also explain that this localization is not a necessary condition for task arithmetic, because eigenfunctions might cancel each other out in other regions. It becomes necessary only when the eigenfunctions are linearly independent within each region.

Disentanglement is learned in pretraining

The last experiment takes pretraining away. Starting from a randomly initialized ViT, the authors fine-tune separately on each of the eight tasks to get task vectors, then add them together.

Task arithmetic fails completely. For ViT-B/32 from random initialization, non-linear fine-tuning gives a mean single-task accuracy of 48.5%, but after the eight vectors are added only 5.5% is left, about the same as a randomly initialized model with no fine-tuning (5.3%). Linearized fine-tuning behaves the same way.

So weight disentanglement emerges during pretraining. Pretraining does more than learn features that keep different meanings apart; it also keeps apart the weight directions that control those semantic regions.

The scope of the evidence

The main experiments all use CLIP ViTs and eight image classification datasets; the appendix adds T5 and convolutional networks. “Non-overlapping input domains” across tasks is the premise of the whole definition, and it roughly holds for these eight datasets. When two tasks' inputs overlap from the outset, as with two kinds of labels on the same text, this analysis cannot be applied directly.

Conclusion

The paper first refutes the claim that “task arithmetic comes from linear fine-tuning,” then formulates the condition task arithmetic needs as weight disentanglement, meaning that each task vector affects only its own input domain.

Linearized models disentangle better but are too weak on individual tasks. Move fine-tuning itself into the tangent space and you get the benefits of both, at the cost of two to three times the inference time.

If your deployment budget can absorb that multiple and you often need to add a pile of task vectors together, linearized fine-tuning is worth a try. The official code wraps the linearized model as a standard nn.Module, so training scripts need almost no changes.

If the budget can't take it, no problem. Just keep the coefficient at 1 or below. Past 1, the task vectors start fighting one another.