Back to paper library
arXiv 2022ICLR 2023Model mergingPaper notes

REPAIR

Averaged into a daze

Original paper

REPAIR: REnormalizing Permuted Activations for Interpolation Repair

Paper authors

Keller Jordan · Hanie Sedghi · Olga Saukh · Rahim Entezari · Behnam Neyshabur

Publication
ICLR 2023Conference
First submitted to arXiv
arXiv:2211.08403
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{modelmergingrepair2022,
  title = {{REPAIR: REnormalizing Permuted Activations for Interpolation Repair}},
  author = {Keller Jordan and Hanie Sedghi and Olga Saukh and Rahim Entezari and Behnam Neyshabur},
  year = {2022},
  eprint = {2211.08403},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2211.08403}
}
On this page

By late 2022, the literature held two contradictory results on whether two networks can be averaged directly once their permutations are aligned.

On one side, Singh and Jaggi aligned neurons with optimal transport in 2020. For two ResNet18s with over 93% accuracy on CIFAR-10, linear interpolation after alignment gave a midpoint of 77% at best, still 16 percentage points short.

On the other side was Git Re-Basin from September 2022. On widened ResNets, it brought two independently trained CIFAR-10 models to nearly zero barrier. One of the alignment methods it compared, “activation matching,” is precisely a special case covered by Singh and Jaggi's method.

How could both results hold at the same time?

This paper, written by Keller Jordan of Hive AI together with Sedghi, Saukh, Entezari and Neyshabur of Google and TU Graz, supplies the answer. The last four are the same group that proposed the permutation invariance conjecture.

Defining the problem

The survey by Entezari et al. in the previous paper had already shown that the barrier after alignment falls with width but rises sharply with depth. For MLPs or CNNs more than a few layers deep, even alignment cannot save them.

The authors wanted to know what exactly depth breaks.

Instead of further refining alignment algorithms, they opened up the interpolated network and looked at the activations of each layer. The measurement is simple. For a given layer, compute each neuron's activation variance over a batch of training data and sum these into vv. Then compare the interpolated model's vαv_\alpha with the endpoint models' v1v_1 and v2v_2:

variance ratio=vα(v1+v2)/2\text{variance ratio} = \frac{v_\alpha}{(v_1+v_2)/2}

A ratio near 1 means the interpolated model's activations are as active as the endpoint models'; a ratio near 0 means the layer's output barely changes with the input.

Original Figure 2. Three scatter plots; the horizontal axis is the layer index and the vertical axis is the variance ratio. From left to right, they show an MLP on MNIST, VGG11 on CIFAR-10 and ResNet50 on ImageNet. Gray squares are direct interpolation without alignment and are near zero after the first layer. Blue triangles are after alignment but before correction; they decline steadily as the layers go on, and the MLP is already near zero by about layer 15. Orange dots are after REPAIR's correction and stay near 1 at every layer.
Original Figure 2, Jordan et al., from PDF page 4, CC0 1.0.

Look at the blue triangles first, which show the interpolated model after alignment and before any correction. The MLP's first-layer variance ratio is about 0.6. From there it falls layer by layer and is almost zero in the last few of its 35 layers. VGG11 and ResNet50 behave the same way, with the variance in their last few layers down by more than a factor of ten.

The authors call this phenomenon variance collapse. Activations in the last few layers are nearly constant, so the network can no longer tell different inputs apart, and accuracy naturally suffers.

The gray squares, direct interpolation without alignment, collapse even faster.

Solving the problem

Why the variance shrinks

Start with the first layer of the interpolated network. The input to the first layer is the raw data, so a neuron's preactivation after interpolation (its value before ReLU) is exactly a linear combination of the corresponding neurons at the two endpoints:

Xα=(1−α)X1+αX2X_\alpha = (1-\alpha)X_1 + \alpha X_2

Treat X1X_1 and X2X_2 as random variables that vary with the data. At the midpoint α=0.5\alpha=0.5,

Var(Xα)=Var(X1)+Var(X2)+2 Cov(X1,X2)4=(std(X1)+std(X2)2)2−1−corr(X1,X2)2 std(X1) std(X2)\begin{aligned} \mathrm{Var}(X_\alpha) &= \frac{\mathrm{Var}(X_1)+\mathrm{Var}(X_2)+2\,\mathrm{Cov}(X_1,X_2)}{4} \\ &= \left(\frac{\mathrm{std}(X_1)+\mathrm{std}(X_2)}{2}\right)^2 - \frac{1-\mathrm{corr}(X_1,X_2)}{2}\,\mathrm{std}(X_1)\,\mathrm{std}(X_2) \end{aligned}

The first term is the variance we would hope for, namely the square of the averaged endpoint standard deviations. The second term is the part that gets subtracted, and it is subtracted whenever the correlation between the two neurons is below 1.

If the two endpoints have similar standard deviations, the expression simplifies to

Var(Xα)=(0.5+0.5 ρ) Var(X1),ρ=corr(X1,X2)\mathrm{Var}(X_\alpha) = (0.5+0.5\,\rho)\,\mathrm{Var}(X_1),\qquad \rho=\mathrm{corr}(X_1,X_2)

However good the alignment, two independently trained networks will not learn exactly the same neurons. The authors observe that neurons paired up by alignment usually have a correlation of only about 0.4. Plugging that in,

0.5+0.5×0.4=0.70.5+0.5\times0.4 = 0.7

the first layer loses 30% of its variance.

Deeper layers cannot be derived this precisely, because their inputs are already the shrunken activations of the previous layer. The authors' intuition is that the loss compounds from layer to layer. As a rough assumption, if each layer keeps only 70%, then after 10 layers only 0.710≈0.030.7^{10}\approx0.03 remains.

Left panel of Original Figure 2. Per-layer variance ratio of the MLP on MNIST; the horizontal axis is the layer index (0 to 34) and the vertical axis is the variance ratio. Gray squares, direct interpolation without alignment, fall from about 0.16 to zero within two or three layers. Blue triangles, after alignment and before correction, start at about 0.62 at layer 0 and decline layer by layer, to about 0.12 at layer 10 and close to zero after layer 20. Orange dots, after REPAIR's correction, sit near 1 at every layer.
Original Figure 2, left panel, Jordan et al., from PDF page 4, CC0 1.0.

Compare this rough estimate with the MLP measurements. The blue triangles start at about 0.62 at layer 0, are still at about 0.12 at layer 10, and only fall to around 0.03 near layer 20. The actual decay is slower than 70% per layer, but the shape of a layer-after-layer decline is the same.

What REPAIR aims for

Since the problem lies in the statistics, fix the statistics. For each channel to be corrected, REPAIR requires the interpolated model to satisfy two conditions:

E[Xα]=(1−α) E[X1]+α E[X2]\mathbb{E}[X_\alpha] = (1-\alpha)\,\mathbb{E}[X_1] + \alpha\,\mathbb{E}[X_2] std(Xα)=(1−α) std(X1)+α std(X2)\mathrm{std}(X_\alpha) = (1-\alpha)\,\mathrm{std}(X_1) + \alpha\,\mathrm{std}(X_2)

The condition on the mean already holds on its own; what actually breaks is the second one. Before correction, typically std(Xα)≪min⁡(std(X1),std(X2))\mathrm{std}(X_\alpha) \ll \min(\mathrm{std}(X_1),\mathrm{std}(X_2)).

As for which channels to correct, the authors choose the output of every convolutional layer in VGG. For ResNet, they add the output of every residual block to the convolutional outputs.

The approximate version: one scaling factor

For the first layer alone, the correction factor can be computed directly. After measuring Var(X1)\mathrm{Var}(X_1), Var(X2)\mathrm{Var}(X_2) and Cov(X1,X2)\mathrm{Cov}(X_1,X_2) on the endpoint models, the variance of the interpolated neuron is

Var(Xα)=(1−α)2Var(X1)+α2Var(X2)+2α(1−α) Cov(X1,X2)\mathrm{Var}(X_\alpha) = (1-\alpha)^2\mathrm{Var}(X_1) + \alpha^2\mathrm{Var}(X_2) + 2\alpha(1-\alpha)\,\mathrm{Cov}(X_1,X_2)

The required scaling factor is simply “the target standard deviation divided by the actual standard deviation”:

β=(1−α) std(X1)+α std(X2)Var(Xα)\beta = \frac{(1-\alpha)\,\mathrm{std}(X_1)+\alpha\,\mathrm{std}(X_2)}{\sqrt{\mathrm{Var}(X_\alpha)}}

With the earlier numbers, ρ=0.4\rho=0.4 and equal endpoint standard deviations, β=1/0.7≈1.20\beta = 1/\sqrt{0.7}\approx1.20, so each neuron's output only needs to be scaled up by 20%.

This factor is exact for the first layer and only approximate for deeper ones. The authors test it on MLPs with 2 to 50 layers trained on MNIST. Without correction, the midpoint model drops below 90% once there are more than 6 layers; with this approximate correction, it stays above 90% all the way to 27 layers.

The advantage of the approximate version is that it needs forward passes only on the endpoint models. Once the statistics are computed, the correction factor for any α\alpha follows directly.

The exact version: let BatchNorm do the measuring

The approximate version falls short on hard cases such as ResNet50 on ImageNet. The authors therefore propose an exact version that runs forward passes directly on the interpolated model.

The approach is to temporarily attach a BatchNorm layer after every module to be corrected:

python
from torch import nn

class RepairWrap(nn.Module):
    def __init__(self, module, channels):
        super().__init__()
        self.module = module
        self.bn = nn.BatchNorm2d(channels)      # newly added layer, starts in train mode

    def set_goal(self, goal_mean, goal_std):
        self.bn.bias.data = goal_mean            # (1-α)·E[X1] + α·E[X2]
        self.bn.weight.data = goal_std           # (1-α)·std(X1) + α·std(X2)

    def forward(self, x):
        return self.bn(self.module(x))

In training mode, BatchNorm first normalizes its input to mean 0 and variance 1, then multiplies by weight and adds bias. Set these two parameters to the target standard deviation and mean, and the output statistics come out exactly as the conditions require.

Next, about 5,000 training examples are fed through the network so that the new BatchNorm layers can estimate their running mean and running variance, while all of the network's original BatchNorm layers stay frozen. After estimation, the model switches to evaluation mode, and each new layer becomes a fixed scale and shift that can finally be fused into the weights of the preceding convolution, leaving the model architecture exactly as it was.

There is an interesting parallel here. If a network already has BatchNorm after every convolution, correcting only the convolution outputs is mathematically equivalent to re-estimating the BatchNorm statistics. SWA already treated averaged models this way in 2018, but no one had applied it between two independently trained networks. REPAIR extends the approach to residual block outputs and to networks without BatchNorm.

REPAIR complements alignment algorithms. Throughout the paper, the authors use the activation-correlation matching of Li et al. (2015), solved with the Hungarian algorithm. In networks with residual connections, all layers on the same residual stream must share the same permutation.

Discussion

ResNet18 on CIFAR-10

For two aligned ResNet18s, the endpoints score 95.5% on the CIFAR-10 test set. The midpoint of direct interpolation reaches only 66.0%; after REPAIR the midpoint is 94.1%, leaving a barrier of just 1.5 percentage points. For comparison, Singh and Jaggi's optimal-transport alignment has a midpoint of 77.0%.

Normalization layers resolve the contradiction

Original Figure 4. The left panel shows the loss barrier of ResNet20 on CIFAR-10, with width multipliers from 1× to 16× on the horizontal axis, comparing BatchNorm, LayerNorm and normalization-free FixUp; hollow markers are before correction and filled markers after. BatchNorm has the highest barrier before correction, roughly 2.6 to 2.1. The middle panel shows the accuracy barrier of the LayerNorm ResNet, which falls from about 42 to 4 before correction. The right panel shows the test accuracy of a standard-width ResNet20 along the interpolation coefficient; before correction, BatchNorm drops to about 35% at the midpoint, while the other three curves all stay above 88%.
Original Figure 4, Jordan et al., from PDF page 8, CC0 1.0.

The authors train ResNet20 on CIFAR-10 and compare three normalization choices, BatchNorm, LayerNorm and Fixup without normalization, at width multipliers from 1× to 16× (the last block going from 64 to 1024 channels).

The right panel is the most convincing. Before correction, the standard-width BatchNorm version drops to 34.7% at the midpoint, while the LayerNorm version already has 88.4%. After correction, both reach 92.0%.

The same pattern holds at 8× width. The LayerNorm midpoint is already at 90.3% before correction and reaches 93.2% after; BatchNorm has only 32.3% before and 94.4% after. The left panel shows that at 16× width, the corrected loss barriers of both BatchNorm and LayerNorm are below 0.05.

Why is LayerNorm special? The authors' explanation is that LayerNorm renormalizes according to the current input even at inference, so the variance each layer loses gets pulled back rather than accumulating. At inference, BatchNorm is just a fixed affine transformation, so the shrinkage stacks up layer after layer.

That resolves the contradiction raised at the outset. The authors checked Git Re-Basin's code and found that its ResNets replace BatchNorm with LayerNorm. Singh and Jaggi used a standard BatchNorm ResNet18, so however good their alignment was, variance collapse dragged it down.

LayerNorm has a cost too. Under the same training setup, the LayerNorm ResNet20 reaches 92.4% on CIFAR-10, against 93.6% for the BatchNorm version.

ImageNet

On ImageNet, direct interpolation after alignment gives midpoint accuracy below 1% for ResNet18, ResNet50 and a double-width ResNet50. After REPAIR, they reach 41.1%, 56.5% and 64.2% respectively.

Whether to correct the residual block outputs makes a sizable difference. For ResNet50, correcting only the convolution outputs gives a midpoint of 53.2%; adding the residual block outputs raises it to 56.5%, and the barrier falls from 23.4 percentage points to 20.1.

That is still a large barrier, though. After correction, the standard ResNet50 is still left with 20.1 percentage points, and the double-width version with 12.9. The authors speculate that hard datasets need wider networks before the barrier gets very low.

Width and depth

For 10-layer MLPs (width 32 to 2048) and ResNet20 (width 1× to 16×), across the four datasets MNIST, FashionMNIST, SVHN and CIFAR-10, REPAIR lowers the barrier every time, and the wider the network, the lower it goes. On MNIST even the narrower networks get close to zero.

As for depth, for VGG on CIFAR-10 from 11 to 19 layers, the barrier rises with depth before correction and becomes almost independent of depth after correction. The “depth raises the barrier” pattern seen in Entezari et al.'s survey comes at least partly from variance collapse.

Merging models trained on different data

The last experiment reproduces Git Re-Basin's setup. The 50,000 CIFAR-100 training images are split into two sets of 25,000. The first set takes 80% of the first 50 classes and 20% of the last 50, and the second set the reverse. Each network is good at half of the classes.

After alignment and REPAIR, many models along the interpolation path have lower loss on the full test set than both endpoints. The best is at α=0.3\alpha=0.3, with a test loss of 1.30; the best value Git Re-Basin reported was 1.73, also at α≈0.3\alpha\approx0.3. The authors candidly say that this gap comes partly from REPAIR and partly from the standard ResNet simply being stronger than the LayerNorm version.

Is this still a permutation?

In their discussion, the authors raise an honest question. REPAIR achieves low barriers in more cases, which seems to support Entezari's conjecture; but it rescales the preactivations of the interpolated model, so it is no longer strictly “linear interpolation after permutation.”

In other words, at practically usable widths and without switching to LayerNorm, permutation alone seems insufficient, and this kind of statistical correction is needed as well. The conjecture itself remains unproven.

In practice, remember also that REPAIR needs training data. The endpoint models' statistics are measured on training data, and the exact version feeds several thousand more examples through the interpolated model. If you have only the weights and no data, this step is out of reach.

Conclusion

This paper traced what looked like an alignment problem to the statistics inside the interpolated network. Unless two neurons are perfectly correlated, averaging shrinks the variance, and BatchNorm networks let that shrinkage compound layer after layer until the whole network stops responding to its input.

The fix simply pulls each channel's mean and standard deviation back to the interpolation between the two endpoints. ResNet18's barrier on CIFAR-10 falls from the 16 percentage points left by Singh and Jaggi's optimal-transport alignment to 1.5, and ResNet50 on ImageNet is rescued from below 1% to 56.5%.

If weight averaging is part of your routine, remember to re-estimate the BatchNorm statistics after averaging. The official code is at KellerJordan/REPAIR, and the correction takes only one forward pass over a few thousand training examples.

SWA was already doing this in 2018. Back then, though, the models being averaged all came from the same training trajectory.