Back to paper library
arXiv 2019ICLR 2020OptimizersPaper notes

LAMB

LARS in sheep's clothing

Original paper

Large Batch Optimization for Deep Learning: Training BERT in 76 minutes

Paper authors

Yang You · Jing Li · Sashank Reddi …

Show all 10 authors

Yang You · Jing Li · Sashank Reddi · Jonathan Hseu · Sanjiv Kumar · Srinadh Bhojanapalli · Xiaodan Song · James Demmel · Kurt Keutzer · Cho-Jui Hsieh

Publication
ICLR 2020Conference
First submitted to arXiv
arXiv:1904.00962
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{optimizerslamb2019,
  title = {{Large Batch Optimization for Deep Learning: Training BERT in 76 minutes}},
  author = {Yang You and Jing Li and Sashank Reddi and Jonathan Hseu and Sanjiv Kumar and Srinadh Bhojanapalli and Xiaodan Song and James Demmel and Kurt Keutzer and Cho-Jui Hsieh},
  year = {2019},
  eprint = {1904.00962},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1904.00962}
}
On this page

When LARS was submitted to ICLR 2018, it was rejected. The decision was specific, noting that it was validated on only one task, ImageNet, that it added hyperparameters, and that scaling the learning rate layer by layer had no theoretical grounding.

A year later, LARS's first author, Yang You, went to Google Brain as a student researcher and came back with a paper that answered each of these points. The new task was BERT, the new optimizer was LAMB, and there were two convergence theorems on top. The title doesn't hold back either, putting "76 minutes" right in it.

Defining the problem

BERT-Large pre-training has two stages, first 900,000 steps on sequences of length 128, then the last 100,000 steps at length 512, with a batch of 512 throughout. On 16 TPUv3 chips it takes about 3 days (the paper measured 81.4 hours).

The most direct way to cut that time is more hardware and a bigger batch, but neither of the two ready-made routes works. One is LARS, which brought ImageNet training with ResNet-50 down to minutes but falters on BERT:

Batch5124K8K16K32K
LARS90.7290.5490.5589.59Diverged
LAMB91.7591.1491.2691.3591.48

Compiled from Table 2 of the paper, selected columns; SQuAD v1.1 dev-set F1. The official BERT-Large F1 is 90.395.

The other is AdamW, which BERT already uses. The authors tried three ways of tuning it: BERT's defaults, LAMB's hyperparameters copied over, and their own retuning. At 16K, F1 reached only 88.1, short of the 90.4 target; in the appendix's tuning table, every setting with a shorter warmup diverged. With a mixed batch of 64K for the first stage and 32K for the second, "even after extensive tuning of the hyperparameters, we fail to get any reasonable result."

Can you keep Adam's element-wise adaptivity, add LARS's layer-wise trust ratio, and get BERT through 32K? That's the question this paper sets out to answer.

Solving the problem

A two-step makeover

The authors first abstract LARS into a general strategy that wraps any base optimizer. If the base algorithm (SGD or Adam) updates each layer as xt+1=xt+ηtutx_{t+1} = x_t + \eta_t u_t, it becomes:

  1. Normalize each layer's update to unit length first: ut/∥ut∥u_t / \|u_t\|.
  2. Multiply the learning rate, layer by layer, by ϕ(∥xt∥)\phi(\|x_t\|), where ϕ\phi is some function of the weight norm.

With SGD as the base, the update for layer ii is

xt+1(i)=xt(i)−ηt ϕ(∥xt(i)∥)∥gt(i)∥ gt(i)x_{t+1}^{(i)} = x_t^{(i)} - \eta_t \,\frac{\phi(\|x_t^{(i)}\|)}{\|g_t^{(i)}\|}\, g_t^{(i)}

The first step keeps only the direction and throws away the gradient's magnitude. That makes the update biased, but the authors argue that with a very large batch, the gradient's direction is largely reliable and the bias is small; the benefit is that a gradient that suddenly becomes extremely large or small doesn't translate directly into the update. The second step makes the update's scale follow the weights. In practice, the authors use

ϕ(z)=min⁡{max⁡{z,γl},γu}\phi(z) = \min\{\max\{z, \gamma_l\}, \gamma_u\}

which clamps the weight norm between a lower and an upper bound, so that layers fresh from initialization with very small or very large norms don't get absurd ratios. The paper doesn't say what γl\gamma_l and γu\gamma_u are actually set to.

If ϕ(z)=z\phi(z) = z, the overall learning rate gets multiplied by ∥xt(i)∥/∥gt(i)∥\|x_t^{(i)}\|/\|g_t^{(i)}\|, which is exactly LARS's trust ratio. The authors also interpret this ratio as a rough estimate of the inverse of the layer's gradient Lipschitz constant, so layers with high curvature automatically take small steps.

LAMB

Swap the base for Adam, and you get LAMB (Layer-wise Adaptive Moments optimizer for Batch training). Each step first computes the first and second moments as in Adam, with bias correction:

mt=β1mt−1+(1−β1) gt,m^t=mt/(1−β1t)vt=β2vt−1+(1−β2) gt2,v^t=vt/(1−β2t)\begin{aligned} m_t &= \beta_1 m_{t-1} + (1-\beta_1)\, g_t, & \hat m_t &= m_t/(1-\beta_1^t) \\ v_t &= \beta_2 v_{t-1} + (1-\beta_2)\, g_t^2, & \hat v_t &= v_t/(1-\beta_2^t) \end{aligned} rt=m^tv^t+ϵr_t = \frac{\hat m_t}{\sqrt{\hat v_t} + \epsilon}

Then decoupled weight decay is added in, and the whole layer is scaled by the trust ratio:

xt+1(i)=xt(i)−ηt ϕ(∥xt(i)∥)∥rt(i)+λxt(i)∥(rt(i)+λxt(i))x_{t+1}^{(i)} = x_t^{(i)} - \eta_t \, \frac{\phi(\|x_t^{(i)}\|)}{\|r_t^{(i)} + \lambda x_t^{(i)}\|} \left(r_t^{(i)} + \lambda x_t^{(i)}\right)

There are two levels of adaptivity here. v^t\sqrt{\hat v_t} adjusts step sizes element-wise within a layer, and the trust ratio evens out the size of the update relative to the weights across layers. Putting weight decay inside the norm was a deliberate choice by the authors, so the decay term and Adam's update together decide how far the layer moves. The defaults are β1=0.9\beta_1=0.9, β2=0.999\beta_2=0.999, ϵ=10−6\epsilon=10^{-6} and λ=0.01\lambda=0.01, with BERT's original linear decay η0(1−t/T)\eta_0(1-t/T) for the learning rate. The authors also note that with β1=β2=0\beta_1=\beta_2=0, LAMB degenerates into signSGD, with the learning rate multiplied by the square root of the layer's dimension.

Figure 12 from the paper: a bar chart of the trust ratio for each of 52 layers at iteration 50 of training ResNet-50 on ImageNet with LAMB. Layer 0 is the highest, at about 0.10; most other layers fall between 0.006 and 0.065, lower overall toward later layers, with the layers in each residual block alternating high and low.
Original Figure 12, You et al., from PDF page 21, arXiv license, scholarly quotation.

This appendix figure records each layer's trust ratio at iteration 50 of ResNet-50 training. The horizontal axis is the layer index and the vertical axis is the ratio. The first layer is the highest, at about 0.10; the lowest layers toward the end are only around 0.006, a difference of more than tenfold, and the layers within a residual block alternate high and low. The authors read this as the trust ratio helping "the slow learners to train faster."

Convergence analysis

This is the part LARS had been asked to supply. For non-convex stochastic optimization with L2L_2 regularization, the authors assume the loss is LiL_i-smooth with respect to layer ii, and define L∞=max⁡iLiL_\infty = \max_i L_i (the steepest layer) and Lavg=∑iLi/hL_{avg} = \sum_i L_i / h (the average over hh layers).

The baseline is SGD. When the batch size bb equals the total number of steps TT and the learning rate is chosen appropriately, SGD's bound depends on L∞L_\infty. The intuition is the same as in LARS's table of per-layer ratios. With one learning rate shared across the network, you can only accommodate the steepest layer.

For LARS (Theorem 2) and LAMB (Theorem 3), the bound depends on LavgL_{avg} instead, under conditions that include:

  • b=Tb = T, meaning the batch is on the same order as the total number of steps;
  • the analysis covers only β1=0\beta_1 = 0 with weight decay λ=0\lambda = 0;
  • ϕ\phi has positive lower and upper bounds, αl≤ϕ≤αu\alpha_l \le \phi \le \alpha_u;
  • LAMB's LavgL_{avg}-dependent bound also requires β2=0\beta_2 = 0 and the same number of parameters in every layer; for β2>0\beta_2 > 0 the authors obtain only a worse bound, adding that they believe the actual behavior for β2>0\beta_2 > 0 should be better than for β2=0\beta_2 = 0, and leaving this to future work.

The two sides also use different convergence criteria. SGD measures the squared gradient norm, while LARS measures the square of the sum of the per-layer gradient norms. The authors explain in the appendix that LARS's bound beats SGD's only when the gradient is "denser" than the curvature and the noise; otherwise SGD's upper bound is better. In other words, the two theorems show that layer-wise scaling helps under certain conditions, and the β1=0.9\beta_1 = 0.9, β2=0.999\beta_2 = 0.999 used in the experiments fall outside the theorems' scope.

Discussion

From 3 days to 76 minutes

The BERT experiments reuse the official data (2.5 billion words of Wikipedia plus 800 million words of BooksCorpus) and the SQuAD v1.1 fine-tuning pipeline, leaving the fine-tuning settings unchanged. LAMB tunes only the learning rate, and doesn't even retune that as the batch grows, scaling it by the square-root rule instead, with 6.25×10−46.25\times10^{-4} at batch 512, multiplied by 2\sqrt 2 with each doubling, up to 5×10−35\times10^{-3} at 32K; the warmup ratio goes from 1/320 of the total steps all the way up to 1/5. The number of epochs is fixed, so the larger the batch, the fewer the steps.

BatchStepsTPUv3TimeF1
512 (official)1000k1681.4 h90.395
5121000k1682.8 h91.752
4K125k128693.6 m91.137
16K31,250512200.0 m91.345
32K15,6251024101.2 m91.475
64K/32K8,599102476.19 m90.584

Compiled from Table 1 of the paper, selected rows.

32K is the most a full TPUv3 Pod (1,024 chips) can fit at sequence length 512. Of the 15,625 steps, 14,063 run at length 128 and 1,562 at length 512, taking about 100 minutes, a 49.1× speedup bought with 64 times the resources.

To go faster still, the authors used different batches for the two stages. The first stage has short sequences and could go up to a batch of 131,072, but beyond 65,536 there was no further speedup, so the first stage uses 65,536 and the second goes back to 32,768. Shrinking the batch throws training into chaos, so the authors re-warm the learning rate from 0 at the start of the second stage, then decay it as usual. That makes 8,599 steps in total, and 76 minutes.

Figure 8 from the paper: speedup at different numbers of TPUs, with black for ideal linear speedup and blue for actual results. From 16 to 128 chips the two are close; at 512 chips it's about 25× versus 32×; at 1024 chips, about 49× versus 64×; the rightmost blue bar, 1024-mixed, is about 65×, slightly above the ideal 64×.
Original Figure 8, You et al., from PDF page 20, arXiv license, scholarly quotation.

The black bars are ideal linear speedup, and the blue ones are measured. With 1,024 chips and a 32K batch, it gets only 49×; the authors attribute the gap to the communication volume of gradient synchronization, since BERT has 300 million parameters and ResNet-50 only 25 million. The mixed batch on the far right measures 65.2×, and exceeding the ideal isn't surprising, since it takes almost half as many steps as a fixed 32K.

The 76-minute figure has to be read together with its F1. The mixed batch's 90.584 is above official BERT's 90.395, but it's the lowest of all of LAMB's own settings; the others all fall between 91.1 and 91.9.

ImageNet

On the ResNet-50 side, the authors wanted to show that an adaptive method like LAMB can match momentum SGD. At batch 16K and 90 epochs, fully tuned AdaGrad, Adam and AdamW reach only 55.38%, 66.04% and 67.27%; even with Goyal et al.'s warmup and step decay added, they only get to 72.01%, 73.48% and 73.07%. Momentum SGD as reported by Goyal et al. gets 75.20%, and LAMB gets 76.66%. With the batch ranging from 512 to 32K and no retuning at all, LAMB's top-1 stays between 76.42% and 77.11%, lowest at 32K; against LARS at 32K, it's 76.4% versus 76.3%.

Reality check

Two years later, Nado et al. at Google redid the BERT experiments in A Large Batch Optimizer Reality Check, with the conclusion right there in the subtitle, "Traditional, Generic Optimizers Suffice Across Batch Sizes."

Keeping the same two-stage setup and the same number of steps, they swapped LAMB back to Adam and retuned it carefully:

BatchStepsLAMBAdam
32K15,62591.4891.58
64K/32K8,59990.5891.04
64K7,818–90.46

Compiled from Table 3 of Nado et al.; SQuAD v1.1 dev-set F1. The LAMB numbers are taken from the LAMB paper.

At batch 32K, Adam's pre-training loss is also lower, 1.277 versus 1.342. Using 64K throughout, Adam reaches 90.46 in just 7,818 steps, more than seven hundred steps fewer than LAMB's mixed batch.

Where do the two sides' conditions differ? Nado et al. point out two implementation details that may have put the Adam in the LAMB paper at a disadvantage. First, the Adam in BERT's open-source code has no bias correction, and bias correction itself has a warmup effect. Second, BERT's learning-rate schedule has a cliff where decay begins, because the decay is mistakenly applied during warmup; this code lives on in more than three thousand copies of BERT training code on GitHub. The tuning effort differed too. Nado et al. searched the learning rate, β1\beta_1, β2\beta_2, the power of the warmup curve and weight decay separately for each pre-training stage, about 30 trials per stage, with 130 for the second stage at 64K, and ϵ\epsilon fixed at 10−1110^{-11}. On the other hand, they didn't rerun LAMB, and they compared step counts rather than measured wall-clock time; their argument is that Adam is computationally simpler, so the same number of steps won't be slower on an optimized system.

On ResNet-50 (batch 32,768, 6,000 steps), they also applied LAMB unmodified to all parameters, and got only 27.3%; switching BatchNorm parameters and bias terms to Adam brought it back to 76.3%. Layer-wise normalization can indeed hurt some parameters, and the LAMB paper doesn't explain how these parameters are handled. In their discussion they write that "the LARS and LAMB update rules have no more to do with the batch size (or 'large' batches) than the Momentum or Adam update rules"; the convergence bounds rest on assumptions far from practice (no momentum, β2=0\beta_2 = 0), and a similar bound can be proved for Adam too.

Another angle comes from MoCo v3. When Chen, Xie and He trained ViTs with a self-supervised method in 2021, they also tried LAMB. For ViT-B/16 at batch 4096, LAMB reached 72.5% at a learning rate of 5×10−45\times10^{-4}, slightly ahead of AdamW; at learning rates of 6×10−46\times10^{-4} and 8×10−48\times10^{-4}, accuracy fell by 1.6 and 6.0 percentage points. The authors found it too sensitive to the learning rate to be convenient for ablations.

Conclusion

To get from 81 hours to 76 minutes, LAMB moved LARS's layer-wise trust ratio onto Adam, and added 1,024 TPUv3 chips, mixed batches and a fresh warmup for the second stage; it also supplied the convergence theorems LARS lacked. Later scrutiny showed that as long as Adam's implementation has no defects and every stage is tuned properly, Adam is no worse than LAMB at the same batch size and step count; and the conditions under which the theorems hold are far from real training.

Before switching to LAMB, check which version your library implements, including whether it has bias correction, ϕ\phi and gradient pre-normalization, and whether LayerNorm and bias terms are excluded from the trust ratio. Then, at the same number of steps, put real effort into tuning an Adam with bias correction as your baseline.

In the end, the 76-minute record was matched in fewer steps by Adam, the very optimizer LAMB set out to replace.