Large Batch Training of Convolutional Networks
- Publication
- Publication venue not yet verified
- First submitted to arXiv
- arXiv:1708.03888
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{optimizerslars2017,
title = {{Large Batch Training of Convolutional Networks}},
author = {Yang You and Igor Gitman and Boris Ginsburg},
year = {2017},
eprint = {1708.03888},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/1708.03888}
}
In June 2017, Goyal and colleagues at Facebook trained ResNet-50 on ImageNet in one hour, using 256 GPUs and a batch size of 8,192, with no loss in accuracy. Their recipe was two rules. Scale the learning rate up by the same factor as the batch, and warm up with a small learning rate for the first few epochs.
Two months later, two NVIDIA interns and NVIDIA's Boris Ginsburg applied the same recipe to AlexNet, and training broke down as soon as the batch went past 2K. This 8-page technical report starts from that failure.
Defining the problem
Data-parallel training splits a batch across many GPUs, each computing gradients that are then averaged. Add more GPUs and keep each one fully loaded, and the global batch grows. With the number of epochs fixed, making the batch times larger means times fewer update steps; the most direct way to make up for that is to make the learning rate times larger too. The paper illustrates this intuition with . Two consecutive steps with a small batch roughly equal one step with a large batch at twice the learning rate, provided the gradient at the second step is about the same as at the first.
That assumption is least reliable early in training, which is why Goyal et al. added warmup. The authors followed the recipe with the BVLC version of AlexNet, training for 100 epochs:
| Batch | Learning rate | Top-1 (%) |
|---|---|---|
| 512 | 0.02 | 58.8 |
| 4,096 | 0.05 | 53.1 |
| 8,192 | 0.03 | 44.8 |
Compiled from Table 1(a) of the paper, taking the learning rate with the highest accuracy for each batch size; 2.5 epochs of warmup.
The learning rate simply won't go up. At batch 4K, any learning rate above 0.06 diverges even with warmup, so it has to stay at 0.05; at 8K it stops at 0.03, whereas linear scaling says it should be 16 times that of batch 512, or 0.32.
Others had hit the same wall. Hoffer et al. switched to a more conservative square-root scaling paired with a special Ghost BatchNorm, and AlexNet at batch 8K still reached only 53.93%, far from the small-batch level. When Krizhevsky himself proposed linear scaling, he only pushed AlexNet to 1K, losing about 1 percentage point.
The authors first replaced AlexNet's LRN layers with BatchNorm, calling the result AlexNet-BN. With that change, and without even needing warmup, batch 8K can use a learning rate of 0.3 and reaches 58.0% accuracy, just 2.2 percentage points short of batch 512's 60.2% (down from a gap of 14). Is the remaining 2.2 the generalization gap Keskar et al. described? The authors compared the training and test losses at batch 256 and 8K and found the gap between the two about the same at both batch sizes, so they concluded that the problem is that training itself falls short, and the model simply hasn't learned enough.
So why won't the learning rate go up?
Each layer's appetite
After the first iteration of AlexNet-BN, the authors compared each layer's weight norm with its gradient norm :
| Layer | Ratio | ||
|---|---|---|---|
| conv1 | 0.098 | 0.017 | 5.76 |
| conv2 | 0.16 | 0.002 | 83.5 |
| conv3 | 0.196 | 0.0015 | 127 |
| conv4 | 0.196 | 0.0013 | 148 |
| conv5 | 0.16 | 0.0002 | 69 |
| fc6 | 6.4 | 0.005 | 1345 |
| fc7 | 6.4 | 0.013 | 489 |
| fc8 | 0.316 | 0.016 | 19 |
Compiled from Table 2 of the paper, weights only; for the bias terms the ratios range from 8.48 to 117.
This ratio can be read directly as a learning rate, the one at which a single update is exactly as large as the weights themselves. Set the learning rate to 1, and conv1 changes about 17% of its own weights in one step (), while fc6 moves by only 0.07% ().
With one learning rate shared across the whole network, you have to accommodate the layer that's easiest to wreck. conv1's ceiling is more than two hundred times lower than fc6's. Raise the learning rate a little and conv1 breaks first; lower it to protect conv1, and layers like fc6 barely move. The paper also observes that this ratio is highest early in training and drops quickly after a few epochs. Warmup works because a small learning rate is safe for every layer, and once the weights have grown a bit, the learning rate can be raised.
The authors took a different tack. If every layer has a different appetite, give each layer its own learning rate.
Solving the problem
One yardstick per layer
LARS (Layer-wise Adaptive Rate Scaling) writes the update for layer as the global learning rate times the layer's own local LR :
is called the trust coefficient, which "defines how much we trust the layer to change its weights during one update." The paper's AlexNet experiments all use ; for ResNet-50 it isn't stated separately.
Combine the two equations and compute the size of the update:
The gradient's magnitude cancels out. Every layer's update at every step is times its own weight norm; the gradient only determines the direction. Take the that AlexNet-BN uses at batch 8K. Before the learning rate decays, the amount fed into the update at each step is 1.4% of the layer's weight norm (not counting momentum accumulation), for conv1 and fc6 alike. The authors also point out that since the update size no longer tracks the gradient size, vanishing and exploding gradients can be partly mitigated.
Compared with adaptive methods like Adam and RMSProp, LARS differs in two ways. One is granularity. It has one learning rate per layer, not one per parameter, which the paper considers more stable. The other is the reference point. It controls the update size by the size of the weights themselves, whereas Adam uses the history of gradient statistics.
Adding weight decay and momentum
With weight decay , it goes into the denominator too:
The full training procedure (Algorithm 1 in the paper, with added to as in Eq. (6)) does the following for each layer at every step:
The global learning rate follows a polynomial decay of degree 2, with momentum . Note the order. The trust ratio uses the norm of this step's gradient, and the scaled result is then accumulated into momentum. When the later LAMB paper restated LARS, it took the norm of the momentum instead, so the two aren't quite the same.

This figure shows the local LR that LARS computes automatically. Look at the vertical scales first. The conv1 weights go up to 17.5, the conv5 weights stay below 1.75, and the two bias terms are smaller still, at most 0.6 and 0.08. Within the same network, the learning rates the layers actually use differ by one to two orders of magnitude. Then look at the three lines. Later in training, the larger the batch, the lower the curve is pushed. The paper only says that the local LR "strongly depends on the layer and batch size," with no further explanation. A learning rate that varies by layer, by batch size and over the course of training is hard to reproduce with a hand-written schedule.
Discussion
Pushing AlexNet to 32K
The authors retrained both networks with LARS, each for 100 epochs, with SGD momentum 0.9, weight decay 0.0005 and , on an NVIDIA DGX-1. Batches of 16K and 32K didn't fit, so they were simulated by splitting each batch into chunks and accumulating gradients before updating.
| Batch | AlexNet LR | AlexNet (%) | AlexNet-BN LR | AlexNet-BN (%) |
|---|---|---|---|---|
| 512 | 2 | 58.7 | 2 | 60.2 |
| 4K | 10 | 58.5 | 10 | 60.4 |
| 8K | 10 | 58.2 | 14 | 60.1 |
| 16K | 14 | 55.0 | 23 | 59.3 |
| 32K | TBD | TBD | 22 | 57.8 |
Compiled from Table 3 of the paper; AlexNet warms up for 2 epochs and AlexNet-BN for 5. TBD is as written in the original.
AlexNet at batch 8K climbs from 44.8% without LARS back to 58.2%, only 0.5 percentage points behind batch 512. AlexNet-BN holds out further, losing 0.9 at 16K and 2.4 at 32K.

This figure shows AlexNet-BN training at batch 8K. On the left, without LARS, the orange line trails from start to finish; on the right, with LARS, it still trails for the first 60 epochs, then catches up and edges ahead. The large batch still starts more slowly, and LARS lets it catch up within the same number of epochs.
Two more observations concern tuning. First, the usable learning-rate range is wide. For AlexNet-BN at 16K, base learning rates from 13 to 22 all give about 59.3%, and at 32K, rates from 17 to 28 all give about 57.5%. Second, the remaining gap at 32K can be closed by training longer, with 100, 125, 150 and 200 epochs giving 57.8%, 59.2%, 59.5% and 59.9%, respectively. The authors' explanation is that once the batch gets large enough, the stochastic gradient is already very close to the true gradient, so a larger batch yields no extra information. Note that 59.9% at 200 epochs is still below 60.2%, and training for twice as many epochs isn't what you want from a large batch in the first place.
ResNet-50
ResNet-50 runs for 90 epochs with momentum 0.9 and weight decay 0.0001, using LARS with 5 epochs of warmup and degree-2 polynomial decay:
| Batch | Global LR | Top-1 (%) |
|---|---|---|
| 256 (no LARS) | 0.2 | 73.0 |
| 8K | 0.6 | 72.7 |
| 16K | 2.5 | 73.0 |
| 32K | 2.9 | 72.3 |
Compiled from Table 4 of the paper.
32K is only 0.7 percentage points below the baseline. But the baseline here is just 73%, because the authors used only minimal data augmentation (resizing to 256×256, then random 224×224 crops and horizontal flips), and testing used only a single center crop; Goyal et al.'s baseline is 76.4%. Moreover, the paper doesn't say what hardware the large-batch ResNet-50 runs used, while AlexNet's 16K and 32K are explicitly simulated with gradient accumulation; nowhere in the paper is there wall-clock time from actually scaling out to thousands of GPUs.
LARS doesn't work for every task, either. A year and a half later, the same first author used LARS to pre-train BERT at Google; at batch 16K the SQuAD F1 fell to 89.6, and at 32K training diverged outright. That failure led him to the next optimizer.
Put to the test again
LARS went on to become standard equipment for large batches. In MLPerf Training's ResNet-50 benchmark, the closed division allows only two optimizers, SGD with momentum or LARS, and the winners of two rounds both used LARS, with batches of 32,768 and 65,536. Self-supervised learning adopted it widely too. SimCLR states, "To stabilize the training, we use the LARS optimizer for all batch sizes," and BYOL uses it as well.
In 2021, Nado et al. at Google went back and asked, in A Large Batch Optimizer Reality Check, whether LARS really handles large batches better than ordinary optimizers. They pointed out that the LARS paper's ResNet-50 accuracy was on the low side, and that LARS was never compared with other optimizers at the same batch size. They took the fastest LARS pipeline from MLPerf, changed only the optimizer and hyperparameters, switched to Nesterov momentum, and ran each 50 times at batch 32,768 for 2,512 steps (64 epochs). LARS had a median of 75.97%, with 35 of 50 runs reaching the 75.9% threshold; Nesterov's two configurations got 75.97% (34 runs) and 75.92% (29 runs).
Catching up wasn't easy. They found that the MLPerf version of "LARS" actually applies LARS only to ordinary weights, handing BatchNorm parameters and bias terms to plain heavy-ball momentum, with no L2 regularization on them either; this detail isn't in the LARS paper, yet it is "extremely important" for both optimizers. On the Nesterov side, they also had to retune label smoothing, BatchNorm initialization and the shape of the warmup curve to catch up. Training accuracy is worth a look too. Nesterov reached 78.97% and LARS 78.07%, yet their test accuracy was about the same. Nado et al. therefore suggest that LARS's layer-wise normalization may act more like a form of regularization. With the budget relaxed to 6,000 steps and the simplest cosine schedule, Nesterov reached 76.7% and LARS 76.9%, and several configurations cleared 75.9% easily. Their conclusion is that large batches aren't inherently that unstable; the hard part is getting every hyperparameter right under an extremely tight step budget.
The NFNet authors went down this road too. NFNet's adaptive gradient clipping (AGC) also looks at the ratio of gradient norm to weight norm, and the paper describes it as a relaxation of normalized optimizers like LARS, capping the update without entirely ignoring the gradient's magnitude. They tried using LARS directly; large batches trained just as stably, but accuracy got worse.
Conclusion
LARS contributed a table and a one-line formula. The table shows AlexNet-BN's per-layer varying by more than two hundredfold, and the formula fixes each layer's update to a proportion of its own weights. With that, AlexNet at batch 8K went from 44.8% back up to 58.2%, and ResNet-50 at 32K lost only 0.7 percentage points. It didn't prove that ordinary optimizers can't do the same; a little over three years later, Nado et al. matched it with Nesterov at the same number of steps.
If you implement LARS yourself, exclude BatchNorm parameters and bias terms from the trust ratio and weight decay, as MLPerf and later self-supervised implementations do. And when comparing it with other optimizers, please put real effort into tuning a momentum SGD baseline at the same number of steps.
The two TBD cells for AlexNet at 32K in Table 3 stayed in through the final v3, and were never filled in.