Back to paper library
arXiv 2018NeurIPS 2018Model mergingPaper notes

Mode Connectivity

Just Go Around

Original paper

Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs

Paper authors

Timur Garipov · Pavel Izmailov · Dmitrii Podoprikhin · Dmitry Vetrov · Andrew Gordon Wilson

Publication
NeurIPS 2018Conference
First submitted to arXiv
arXiv:1802.10026
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{modelmergingmodeconnectivity2018,
  title = {{Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs}},
  author = {Timur Garipov and Pavel Izmailov and Dmitrii Podoprikhin and Dmitry Vetrov and Andrew Gordon Wilson},
  year = {2018},
  eprint = {1802.10026},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1802.10026}
}
On this page

Train the same network with two random seeds and you get very different weights, often with similar accuracy.

Those weight sets are two points in parameter space. Join them with a straight line and evaluate a few points along it: the middle often has alarmingly high error. The paper's VGG-16 example on CIFAR-10 reaches 90% test error at the midpoint—chance performance across ten classes.

It is natural to picture each solution as an isolated valley, separated from others by high mountains.

Garipov, Izmailov, Podoprikhin, Vetrov and Wilson, from Samsung's Moscow AI Center, Skoltech, Cornell and HSE, ask a different question: if a straight road fails, what about a curved one?

Defining the problem

Let w^1\hat w_1 and w^2\hat w_2 be two independently trained networks' weights, and L(w)L(w) the training loss. The straight connection is

w(t)=(1−t) w^1+t w^2,t∈[0,1]w(t) = (1-t)\,\hat w_1 + t\,\hat w_2,\qquad t\in[0,1]

The goal is a curve ϕθ(t)\phi_\theta(t) with endpoints fixed at w^1\hat w_1 and w^2\hat w_2. Parameters θ\theta control its shape, and loss should stay low along the whole curve.

Finding one would change the picture of isolated solution valleys. Failure would require distinguishing a restrictive curve family, unsuccessful optimization and genuinely hard-to-connect endpoints. One failed search cannot rule out other paths.

Solving the problem

Let the curve learn

Every point on the curve supplies network weights whose loss can be evaluated. The objective averages loss along the curve:

ℓ(θ)=∫01L(ϕθ(t)) dt=Et∼U(0,1)[L(ϕθ(t))]\ell(\theta) = \int_0^1 L\big(\phi_\theta(t)\big)\,dt = \mathbb{E}_{t\sim U(0,1)}\Big[L\big(\phi_\theta(t)\big)\Big]

Sampling estimates the gradient of this expectation. At each step, use [0,1][0,1] as the sampling range and draw tt uniformly. Construct that point's weights, evaluate loss on a minibatch, and take a gradient step on θ\theta. The endpoints w^1\hat w_1, w^2\hat w_2 stay fixed; only the curve-shaping parameters θ\theta learn.

Two curve families

The authors use two simple parameterizations. First is a polygonal chain with one bend, located at θ\theta:

ϕθ(t)={2(t θ+(0.5−t) w^1),0≤t≤0.52((t−0.5) w^2+(1−t) θ),0.5≤t≤1\phi_\theta(t) = \begin{cases} 2\big(t\,\theta + (0.5-t)\,\hat w_1\big), & 0\le t\le 0.5\\ 2\big((t-0.5)\,\hat w_2 + (1-t)\,\theta\big), & 0.5\le t\le 1 \end{cases}

Second is a quadratic Bézier curve, where θ\theta is the control point:

ϕθ(t)=(1−t)2 w^1+2t(1−t) θ+t2 w^2\phi_\theta(t) = (1-t)^2\,\hat w_1 + 2t(1-t)\,\theta + t^2\,\hat w_2

Both curves reach the fixed endpoints at t=0t=0, t=1t=1. θ\theta has exactly the same shape as the network weights, so the number of learned parameters equals one network's parameter count.

Consider the Bézier midpoint. At t=0.5t=0.5, each endpoint contributes 1/41/4 and the control point contributes 1/21/2. Moving the control point one unit therefore moves the midpoint weights half a unit. During backpropagation, the control-point gradient is the current weight gradient multiplied by 2t(1−t)2t(1-t). Endpoint weights stay outside the optimizer, preserving the original models.

The polygonal chain works similarly, with the multiplier changing between its two halves. Curve training reuses ordinary forward and backward passes; the main extra operation is a linear weight combination. It still requires training data, additional optimization and storage for one control point. Finding a low-loss path comes with these costs.

Try a small network

A small network makes the effect visible. Use scikit-learn's 1,797 handwritten 8×8 digits and an MLP with two 200-unit hidden layers. Train two endpoints on the same 1,200 images with different initial weights, then initialize a Bézier control point at the straight-line midpoint:

python
import torch
import torch.nn as nn
from torch.func import functional_call
from sklearn.datasets import load_digits

X, y = load_digits(return_X_y=True)
X = torch.tensor(X / 16.0, dtype=torch.float32)
y = torch.tensor(y)
perm = torch.randperm(len(X), generator=torch.Generator().manual_seed(0))
idx_train, idx_test = perm[:1200], perm[1200:]

def make(seed):
    torch.manual_seed(seed)
    return nn.Sequential(nn.Linear(64, 200), nn.ReLU(),
                         nn.Linear(200, 200), nn.ReLU(),
                         nn.Linear(200, 10))

def train(model, seed, steps=480):
    opt = torch.optim.SGD(model.parameters(), lr=0.1)
    g = torch.Generator().manual_seed(seed)
    for _ in range(steps):
        b = idx_train[torch.randint(len(idx_train), (50,), generator=g)]
        loss = nn.functional.cross_entropy(model(X[b]), y[b])
        opt.zero_grad(); loss.backward(); opt.step()
    return {k: v.detach() for k, v in model.state_dict().items()}

w1 = train(make(1), seed=10)                 # 兩個端點:同樣的資料,不同的初始權重
w2 = train(make(2), seed=20)
net = make(0)

@torch.no_grad()
def evaluate(w):
    loss = nn.functional.cross_entropy(functional_call(net, w, (X[idx_train],)), y[idx_train]).item()
    acc = (functional_call(net, w, (X[idx_test],)).argmax(1) == y[idx_test]).float().mean().item()
    return f"loss {loss:.3f}, 測試 {acc:.1%}"

def line(t):
    return {k: (1 - t) * w1[k] + t * w2[k] for k in w1}

theta = {k: v.clone().requires_grad_(True) for k, v in line(0.5).items()}   # 彎點從直線中點出發

def bezier(t):
    return {k: (1 - t) ** 2 * w1[k] + 2 * t * (1 - t) * theta[k] + t ** 2 * w2[k] for k in w1}

opt = torch.optim.SGD(theta.values(), lr=0.1)
g = torch.Generator().manual_seed(0)
for _ in range(1200):
    t = torch.rand((), generator=g).item()   # 每一步隨機抽一個 t
    b = idx_train[torch.randint(len(idx_train), (50,), generator=g)]
    loss = nn.functional.cross_entropy(functional_call(net, bezier(t), (X[b],)), y[b])
    opt.zero_grad(); loss.backward(); opt.step()   # 只更新彎點 theta,端點不動

for t in [0, 0.25, 0.5, 0.75, 1]:
    print(f"t={t:<4}  直線: {evaluate(line(t))}  |  曲線: {evaluate(bezier(t))}")
text
t=0     直線: loss 0.119, 測試 95.5%  |  曲線: loss 0.119, 測試 95.5%
t=0.25  直線: loss 0.415, 測試 93.6%  |  曲線: loss 0.142, 測試 95.5%
t=0.5   直線: loss 0.949, 測試 88.9%  |  曲線: loss 0.156, 測試 95.3%
t=0.75  直線: loss 0.420, 測試 95.0%  |  曲線: loss 0.140, 測試 95.3%
t=1     直線: loss 0.119, 測試 94.8%  |  曲線: loss 0.119, 測試 94.8%

functional_call lets one network structure evaluate externally supplied weights, including any point on the curve. Both endpoints have loss 0.119. The straight midpoint reaches 0.949; the learned curve's midpoint reaches just 0.156 and keeps test accuracy around 95%.

This is a tiny network with a modest straight-line barrier. The paper's larger networks have much higher walls.

Discussion

Curves in larger networks

Original Figure 1: ResNet-164 training loss on CIFAR-100 in three two-dimensional planes. Left: three independently trained solutions occupy separate low-loss regions. Middle: a quadratic Bézier curve; right: a one-bend polygonal chain. Both connect the lower two solutions through low-loss regions, while their straight dashed segment crosses high loss.
Figure 1 from the original paper (Garipov et al.), reproduced from PDF page 2.

This figure shows ResNet-164 training loss on CIFAR-100. Each panel slices parameter space in two dimensions, with its horizontal axis passing through two solutions. Red means low loss; blue means high loss.

The left plane is defined by three independently trained solutions, each in its own red region separated by blue, matching the isolated-valley intuition. The middle and right use different planes containing the learned curves. Red channels now connect the lower two solutions, while their straight dashed segment still crosses blue.

Original Figure 2: position t along each curve is horizontal. Left: straight-line training loss rises to about 5 while Bézier and polygonal curves stay near the bottom. Middle: straight-line test error approaches 100%, while the curved paths stay in the twenties. Right: ensembling an endpoint with a point along the polygonal chain reaches roughly 22% error after t around 0.4.
Figure 2 from the original paper (Garipov et al.), reproduced from PDF page 5.

Figure 2 plots the same experiment as curves. The red straight segment sends training loss and test error soaring in the left and middle panels; the green Bézier curve and blue polygonal chain remain almost flat.

Barrier height depends on architecture. On CIFAR-10, VGG-16 reaches 90% midpoint test error. ResNet-158 and Wide ResNet-28-10 peak at 80% and 66%. The authors relate this to earlier observations that residual-network landscapes are more regular than those of classical models such as VGG.

The connecting curve is also nonunique. Training two polygonal chains with different seeds between the same VGG-16 endpoints gives bend points 29.6 apart, versus endpoint distance 50.

Points along the curve make different predictions

If loss stays low along the curve, could its points merely be different parameterizations of the same model?

The right panel addresses this. Ensemble the endpoint ϕθ(0)\phi_\theta(0) with a point ϕθ(t)\phi_\theta(t) by averaging their outputs. Along the polygonal chain, ensemble error starts declining at t≈0.1t\approx0.1; by t≥0.4t\ge0.4, it matches the ensemble of the independently trained endpoints. Moving only a short distance yields a model with meaningfully different predictions and high accuracy.

If curve points represented the same function—for example, by reciprocal rescaling around ReLU—the ensemble would gain nothing. The appendix constructs such unhelpful connecting curves too.

From curves to ensembles

This observation leads to the paper's second contribution: Fast Geometric Ensembling (FGE).

If a small move near one solution finds a different, useful model, explicitly learning a connecting curve becomes unnecessary. FGE trains with a standard learning-rate schedule for roughly 80% of the budget, then uses short cycles of two to four epochs. High rates move outward; low rates return toward the basin floor. Save weights at the low point of each cycle and ensemble the resulting models.

Compared with Snapshot Ensembles, the distinction is step size. Snapshot Ensembles uses 20–40-epoch cycles throughout training; FGE uses short cycles only near the end. On CIFAR-100 with Preactivation-ResNet-164, successive FGE checkpoints are about 7 apart in Euclidean parameter distance, versus about 40 for Snapshot Ensembles.

The ImageNet result is straightforward. Starting from pretrained ResNet-50 at 23.87% top-1 error, five extra FGE epochs with two-epoch cycles produce a four-model ensemble, including the original, at 23.31%: a reduction of 0.56 percentage points.

FGE still runs every ensemble member during inference.

The cycle and checkpoint timing

Read the original figure's left side from top to bottom: learning rate, test error and parameter distance from the start. Black circles mark saved checkpoints. The right compares results under equal training budgets: crosses are individual snapshots; connected diamonds are ensembles.

FGE's short learning-rate cycles, low-rate checkpoint positions and accuracy comparisons with Snapshot Ensembles under equal training budgets.
Figure 3 from the original paper (Garipov et al.), reproduced from PDF page 6. Both sets of plots are retained; the original caption is omitted.

FGE uses a triangular learning rate. Let a cycle contain an even number cc of minibatch updates, with within-cycle position u∈(0,1]u\in(0,1] and bounds α1>α2\alpha_1>\alpha_2:

α(u)={(1−2u)α1+2uα2,0<u≤1/2,(2−2u)α2+(2u−1)α1,1/2<u≤1.\alpha(u)= \begin{cases} (1-2u)\alpha_1+2u\alpha_2,&0<u\le 1/2,\\ (2-2u)\alpha_2+(2u-1)\alpha_1,&1/2<u\le1. \end{cases}

The rate falls from high to low, then rises again. Checkpoints are saved halfway through the cycle, at u=1/2u=1/2 when the learning rate equals α2\alpha_2. Error temporarily rises near the high-rate phase; the desired snapshots have returned to a low-error region.

In the CIFAR experiments, VGG uses two-epoch cycles with α1=0.01\alpha_1=0.01, α2=0.0005\alpha_2=0.0005; ResNet and WRN use four-epoch cycles with α1=0.05\alpha_1=0.05, α2=0.0005\alpha_2=0.0005. For example, with ResNet's standard budget of B=150B=150 epochs, the normal schedule runs through epoch 125, then six models are collected over epochs 126–147. The snapshots share the earlier trajectory, saving repeated training from scratch.

This clarifies the roles of curve search and FGE. Curve search fixes two endpoints and trains a bend to test whether a low-loss channel exists. FGE starts from one model and uses short cycles to collect nearby solutions. Running FGE requires neither a second independently trained endpoint nor a learned Bézier curve.

Accuracy under the same budget

Separate training budget from the number of models used at inference. The paper's BB is the epoch budget for training one conventional model to convergence; 2B2B doubles that budget. The table selects CIFAR-100 results at 1B1B; its metric is test error, where lower is better.

ArchitectureIndependent training (Ind)Snapshot EnsemblesFGE
VGG-16, B=200B=20027.4 ± 0.126.4 ± 0.125.7 ± 0.1
Preactivation ResNet-164, B=150B=15021.5 ± 0.420.9 ± 0.220.2 ± 0.1
WRN-28-10, B=200B=20019.2 ± 0.217.9 ± 0.217.7 ± 0.2

Test error (%), compiled from original Table 1. Each row uses the same training budget, while inference model counts vary by method.

All three beat Snapshot Ensembles, with reductions of 0.7 percentage points for both VGG and ResNet. At 2B2B, CIFAR-100 WRN scores are Ind 17.48%, SSE 17.30%, FGE 16.95%. But for CIFAR-10 WRN at 3B3B, Ind reaches 3.31% and FGE 3.52%. Advantages vary with dataset, architecture and budget.

The authors also measure prediction diversity. On CIFAR-100 with ResNet-164, independently trained models disagree on 19.97% of test images, versus 14.57% for two snapshots from one FGE run. Individual FGE snapshots are slightly weaker too: about 78.0% accuracy versus 78.5% for full independent training. FGE's advantage is collecting more useful snapshots in the same time, offsetting those per-model and diversity deficits.

Finally, BatchNorm matters. Changing weights also changes intermediate activation distributions. The appendix reestimates BatchNorm statistics before evaluating loss along the curve. Reusing endpoint running means and variances can mix statistics mismatch into the apparent barrier. Ensure each point's network can infer correctly before interpreting the landscape.

Conclusion

A simple training procedure connects independently trained solutions with a one-bend path of nearly constant loss, even when their straight segment crosses a high wall.

The picture of isolated valleys gives way to low-loss channels between solutions, with curved paths. Directly averaging independently trained networks can fail because averaging follows the straight segment.

FGE leaves the inference cost of an ensemble. Two weeks later, the same group introduced SWA: average those saved weights into a single model.