Back to paper library
arXiv 2014ICLR 2015OptimizersPaper notes

Adam

Adam's first step

Original paper

Adam: A Method for Stochastic Optimization

Paper authors

Diederik P. Kingma · Jimmy Ba

Publication
ICLR 2015Conference
First submitted to arXiv
arXiv:1412.6980
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{optimizersadam2014,
  title = {{Adam: A Method for Stochastic Optimization}},
  author = {Diederik P. Kingma and Jimmy Ba},
  year = {2014},
  eprint = {1412.6980},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1412.6980}
}
On this page

A footnote on the first page of this paper says the two authors contributed equally, and that the author order was decided by a coin flip over Google Hangout.

After the coin flip, Diederik Kingma went first and Jimmy Ba second, and the paper went up on arXiv in December 2014. The text accompanying Algorithm 1 includes the line "Good default settings for the tested machine learning problems," followed by four numbers, α=0.001\alpha=0.001, β1=0.9\beta_1=0.9, β2=0.999\beta_2=0.999, ϵ=10−8\epsilon=10^{-8}.

Open PyTorch ten years later, and the defaults of torch.optim.Adam are still those four numbers.

This note takes Algorithm 1 apart line by line. What is each of the two moving averages estimating? Why divide by 1−βt1-\beta^t? How far does a single step actually go? What did the paper's experiments show, and which part was later overturned?

Defining the problem

When we train a neural network, all we have are gradients computed on mini-batches, so every step carries noise, and regularization such as dropout adds another layer of randomness on top. With parameters routinely numbering in the millions, second-order methods are too expensive, and the paper restricts itself to first-order methods from the outset.

Plain SGD uses the same learning rate for every parameter. The trouble is that gradient magnitudes can differ across parameters by several orders of magnitude, so a single learning rate is too large for some parameters and too small for others.

Two approaches to this already existed at the time.

  • AdaGrad keeps, for each parameter, the sum of its past squared gradients and divides the gradient by the square root of that sum, θt+1=θt−α gt/∑i=1tgi2\theta_{t+1}=\theta_t-\alpha\, g_t\big/\sqrt{\sum_{i=1}^{t} g_i^2}. Rarely seen features have a small sum of squared gradients and therefore take larger steps, which works especially well on sparse data. The price is that the sum only ever grows, so the learning rate keeps shrinking.
  • RMSProp replaces the sum with an exponential moving average and looks only at recent gradients. The learning rate doesn't keep shrinking, and it copes better with objectives that change over time.

Adam wants the advantages of both. Section 5 of the paper also points out that RMSProp has no bias correction. When β2\beta_2 is close to 1 (exactly the setting sparse gradients call for), RMSProp without correction takes very large steps at the start and often diverges outright.

So the paper asks whether we can build an optimizer that uses only first-order gradients and only two extra parameter-sized copies of memory, handles both sparse gradients and changing objectives, and stays steady from the very start of training.

Solving the problem

Algorithm 1

First, the whole algorithm. All operations are element-wise; gt2g_t^2 means gt⊙gtg_t\odot g_t, and βt\beta^t is β\beta raised to the power tt:

gt←∇θft(θt−1)mt←β1 mt−1+(1−β1) gtvt←β2 vt−1+(1−β2) gt2m^t←mt / (1−β1t)v^t←vt / (1−β2t)θt←θt−1−α m^t / (v^t+ϵ)\begin{aligned} g_t &\leftarrow \nabla_\theta f_t(\theta_{t-1}) \\ m_t &\leftarrow \beta_1\, m_{t-1} + (1-\beta_1)\, g_t \\ v_t &\leftarrow \beta_2\, v_{t-1} + (1-\beta_2)\, g_t^2 \\ \hat m_t &\leftarrow m_t \,/\, (1-\beta_1^t) \\ \hat v_t &\leftarrow v_t \,/\, (1-\beta_2^t) \\ \theta_t &\leftarrow \theta_{t-1} - \alpha\, \hat m_t \,/\, \big(\sqrt{\hat v_t}+\epsilon\big) \end{aligned}

Initially m0=v0=0m_0=v_0=0 and t=0t=0, and each iteration begins by incrementing tt by one.

mtm_t is the exponential moving average of the gradient, which the paper calls an estimate of the first moment, that is, the mean of the gradient; vtv_t is the exponential moving average of the squared gradient, an estimate of the second moment (the uncentered variance). The name Adam comes from adaptive moment estimation (according to the acknowledgments, Ivo Danihelka and Tom Schaul came up with it).

The update is m^t/v^t\hat m_t/\sqrt{\hat v_t}. The numerator sets the direction, and because it is a moving average, it has the effect of momentum; the denominator converts each parameter to a scale of its own. This amounts to every parameter having its own learning rate α/v^t\alpha/\sqrt{\hat v_t}, except that this learning rate is re-estimated at every step from recent gradients.

The paper also gives a cheaper formulation that folds the two corrections into a single coefficient:

αt=α⋅1−β2t1−β1t,θt←θt−1−αt mt / (vt+ϵ^)\alpha_t=\alpha\cdot\frac{\sqrt{1-\beta_2^t}}{1-\beta_1^t},\qquad \theta_t\leftarrow\theta_{t-1}-\alpha_t\, m_t\,/\,\big(\sqrt{v_t}+\hat\epsilon\big)

The two forms differ only in where ϵ\epsilon is added; the ϵ^\hat\epsilon here is a different quantity from the ϵ\epsilon in Algorithm 1. This small difference later lived on in the various frameworks, and we'll come back to it in the discussion.

Bias correction

Those two divisions in the middle are the biggest difference between this paper and RMSProp, and they are what Section 3 is devoted to deriving.

Unroll the recursion for vtv_t, and it turns out to be a weighted sum of all past squared gradients:

vt=(1−β2)∑i=1tβ2 t−i gi2v_t=(1-\beta_2)\sum_{i=1}^{t}\beta_2^{\,t-i}\, g_i^2

Take the expectation of both sides. If the expected squared gradient doesn't change over time, we can pull E[gt2]\mathbb E[g_t^2] out, and what remains is a geometric series:

E[vt]=E[gt2]⋅(1−β2)∑i=1tβ2 t−i+ζ=E[gt2]⋅(1−β2t)+ζ\mathbb E[v_t]=\mathbb E[g_t^2]\cdot(1-\beta_2)\sum_{i=1}^{t}\beta_2^{\,t-i}+\zeta =\mathbb E[g_t^2]\cdot(1-\beta_2^t)+\zeta

If the second moment really is constant over time, ζ=0\zeta=0; even if it changes, ζ\zeta stays small as long as β2\beta_2 gives gradients from long ago small enough weights. The remaining factor 1−β2t1-\beta_2^t comes entirely from starting at v0=0v_0=0; divide it out, and the estimate is back on the right scale. The derivation for mtm_t is exactly the same.

Let's work out the first step with the defaults. At t=1t=1:

  • m1=0.1 g1m_1=0.1\,g_1, only a tenth of the true value;
  • v1=0.001 g12v_1=0.001\,g_1^2, which after the square root is 0.0316 ∣g1∣0.0316\,|g_1|, only about a thirtieth of the true scale.

Both are too small, but vv more so. Without correction, the first step has size α⋅0.1/0.0316≈3.16 α\alpha\cdot 0.1/0.0316\approx 3.16\,\alpha. With correction, m^1=g1\hat m_1=g_1 and v^1=g12\hat v_1=g_1^2, and every parameter's first step is exactly α⋅sign⁡(g1)\alpha\cdot\operatorname{sign}(g_1).

That is Adam's first step. Whether the gradient is 10−610^{-6} or 10310^{3}, every parameter moves by just α\alpha.

Without correction, then, how long do the oversized steps last? If every step sees the same gradient, the uncorrected step size is α (1−β1t)/1−β2t\alpha\,(1-\beta_1^t)/\sqrt{1-\beta_2^t}. Let's plot it.

Line chart with step t on the horizontal axis (log scale, from 1 to 20,000) and the step size without bias correction on the vertical axis, in units of α, with β1 fixed at 0.9. The blue line for β2 = 0.99 peaks at about 2.1α at step 13 and returns to around α after roughly two hundred steps; the orange dashed line for β2 = 0.999 peaks at about 6.6α at step 12 and is still about 1.26α at step 1,000; the green dotted line for β2 = 0.9999 peaks at about 20.7α at step 12, is still about 3.2α at step 1,000, and takes more than ten thousand steps to return to α. With bias correction, all three are a constant α.
Drawn for this note, computed from Algorithm 1 of the paper. Assumes the same gradient at every step.

All three lines peak around steps 12 and 13. By then mtm_t has built up to about 70% of its true value, while with β2=0.999\beta_2=0.999, vtv_t has built up only about 1%, which is still only a little over a tenth after the square root. The numerator has caught up; the denominator hasn't. With β2=0.999\beta_2=0.999 the peak is 6.6 α6.6\,\alpha, and it takes over 1,700 steps to get back within 1.1 α1.1\,\alpha; with β2=0.9999\beta_2=0.9999 the peak is 20.7 α20.7\,\alpha, and it takes over 17,000 steps.

Take momentum away as well (β1=0\beta_1=0), and the uncorrected first step is α/1−β2\alpha/\sqrt{1-\beta_2}, which at β2=0.9999\beta_2=0.9999 is 100 α100\,\alpha. The learning rate is set to 10−310^{-3}, yet the first step effectively uses 10−110^{-1}.

This is exactly the situation the paper describes. Sparse gradients need a large β2\beta_2 to average over enough samples, and the closer β2\beta_2 gets to 1, the worse the uncorrected initial bias becomes.

How far one step goes

Section 2.1 goes on to analyze the update itself. Set ϵ=0\epsilon=0 for now; the effective step at each iteration is Δt=α m^t/v^t\Delta_t=\alpha\,\hat m_t/\sqrt{\hat v_t}.

m^t/v^t\hat m_t/\sqrt{\hat v_t} is "the mean of the gradient" divided by "the square root of the mean of the squared gradient." Since ∣E[g]∣≤E[g2]|\mathbb E[g]|\le\sqrt{\mathbb E[g^2]}, the absolute value of this ratio usually doesn't exceed 1, so ∣Δt∣|\Delta_t| is roughly bounded by α\alpha.

There is one exception. If a parameter's gradient has been 0 all along and suddenly has a value at this step, m^t\hat m_t is about (1−β1)gt(1-\beta_1)g_t and v^t\hat v_t about (1−β2)gt2(1-\beta_2)g_t^2, and the ratio becomes (1−β1)/1−β2(1-\beta_1)/\sqrt{1-\beta_2}. The bound the paper gives therefore has two cases:

∣Δt∣≤{α⋅(1−β1)/1−β2,if 1−β1>1−β2α,otherwise|\Delta_t|\le \begin{cases} \alpha\cdot(1-\beta_1)/\sqrt{1-\beta_2}, & \text{if } 1-\beta_1>\sqrt{1-\beta_2}\\[2pt] \alpha, & \text{otherwise} \end{cases}

Plugging in the defaults, 1−β1=0.11-\beta_1=0.1 is larger than 0.001≈0.0316\sqrt{0.001}\approx0.0316, so the default setting falls into the first case, with a worst case of 3.16 α3.16\,\alpha. This only happens for extremely sparse parameters; the paper says that when the sparsity is less extreme, the effective step will be smaller than this.

This property has three practical consequences.

  1. α\alpha is the step size. It is roughly the distance moved in parameter space at each step, and the paper likens it to drawing a trust region around the current parameters. If we know roughly how far a good solution is from the initial values, we can estimate the order of magnitude of α\alpha.
  2. It doesn't depend on the gradient's scale. Multiply the gradient by cc, and m^t\hat m_t becomes cc times larger and v^t\hat v_t becomes c2c^2 times larger, leaving the ratio unchanged. Multiply the loss function by 1000, and SGD's steps grow 1000-fold, while Adam isn't affected at all.
  3. Automatic annealing. The paper borrows the term "signal-to-noise ratio" for m^t/v^t\hat m_t/\sqrt{\hat v_t}. Near an optimum, the gradient direction often flips back and forth, so the mean approaches 0 while the mean of the squares is still there; the ratio shrinks, and the step shrinks along with it.

Relation to AdaGrad and RMSProp

Section 5 puts Adam back into the family it came from and compares.

Compared with RMSProp, there are two differences. RMSProp with momentum first divides the gradient by the scale and then applies momentum to the result; Adam estimates the gradient's first and second moments separately and then divides one by the other. The other difference is bias correction.

The relationship with AdaGrad is more interesting. Set β1=0\beta_1=0, let 1−β21-\beta_2 approach 0, and replace the learning rate with αt=α/t\alpha_t=\alpha/\sqrt t; the bias-corrected v^t\hat v_t becomes 1t∑i=1tgi2\frac1t\sum_{i=1}^t g_i^2, and substituting this back into the update gives exactly AdaGrad. This correspondence holds only if bias correction is kept; without it, β2\beta_2 approaching 1 sends 1−β2t1-\beta_2^t toward 0, and the update becomes infinite.

The paper also notes that v^t\hat v_t can be viewed as an approximation of the diagonal of the Fisher information matrix, so Adam amounts to preconditioning with its inverse square root. Compared with natural gradient, which uses the inverse of the Fisher matrix directly, this approach is somewhat more conservative.

AdaMax

Section 7.1 is an extension. Adam scales by the L2L^2 norm of the gradient, and the authors generalize this to LpL^p:

vt=β2p vt−1+(1−β2p) ∣gt∣pv_t=\beta_2^p\, v_{t-1}+(1-\beta_2^p)\,|g_t|^p

Large pp is numerically unstable, but if we let p→∞p\to\infty and set ut=lim⁡p→∞vt1/pu_t=\lim_{p\to\infty}v_t^{1/p}, only the largest term in the weighted sum survives, giving a very simple recursion:

ut=max⁡(β2 ut−1, ∣gt∣),θt←θt−1−α1−β1t⋅mtutu_t=\max\big(\beta_2\, u_{t-1},\ |g_t|\big),\qquad \theta_t\leftarrow\theta_{t-1}-\frac{\alpha}{1-\beta_1^t}\cdot\frac{m_t}{u_t}

utu_t is "a discounted record of the largest past gradient." If some step sees ∣g∣=10|g|=10, uu becomes 10 and is then multiplied by 0.999 at every step, so a thousand steps later about 3.7 remains. It needs no bias correction, because utu_t is always at least the current ∣gt∣|g_t| and never has to creep up from 0. The bound on the step is simpler too, ∣Δt∣≤α|\Delta_t|\le\alpha. The paper's suggested defaults are α=0.002\alpha=0.002, with β1\beta_1 and β2\beta_2 the same as in Adam.

Section 7.2 also proposes an exponential moving average of the parameters, θˉt←β2θˉt−1+(1−β2)θt\bar\theta_t\leftarrow\beta_2\bar\theta_{t-1}+(1-\beta_2)\theta_t, again correcting the initial bias with 1−β2t1-\beta_2^t.

Convergence proof

Section 4 analyzes Adam in the framework of online convex optimization. At each step the algorithm faces a convex function ftf_t that isn't known in advance, and the yardstick is regret, the amount by which the cumulative loss exceeds that of the best fixed parameters in hindsight:

R(T)=∑t=1T[ft(θt)−ft(θ∗)]R(T)=\sum_{t=1}^{T}\big[f_t(\theta_t)-f_t(\theta^*)\big]

Theorem 4.1 comes with a long list of conditions: bounded gradients, bounded distances between iterates, β12/β2<1\beta_1^2/\sqrt{\beta_2}<1, a learning rate decaying as αt=α/t\alpha_t=\alpha/\sqrt t, and β1\beta_1 also decaying over time as β1,t=β1λt−1\beta_{1,t}=\beta_1\lambda^{t-1}. Under these conditions, the paper obtains O(T)O(\sqrt T) regret, so the average regret R(T)/TR(T)/T goes to 0.

In practice Adam is rarely run under these conditions. The learning rate is often constant or follows some other schedule, and β1\beta_1 is fixed at 0.9. The paper itself admits in Section 6.2 that this analysis doesn't apply to non-convex problems such as neural networks.

This theorem was later shown to be wrong. In 2018, Reddi, Kale, and Kumar pointed out that at one step, the proof in the appendix treats v^t/αt−v^t−1/αt−1\sqrt{\hat v_t}/\alpha_t-\sqrt{\hat v_{t-1}}/\alpha_{t-1} as if it could never be negative. This quantity is how "the inverse of the effective learning rate" changes over time; Adam's vtv_t is a moving average that shrinks when the gradients shrink, so it can be negative. They also pointed out problems with Lemmas 10.3 and 10.4, and constructed a one-dimensional convex problem that satisfies the theorem's conditions, on which Adam converges to the worst possible point.

Discussion

The three experiments on logistic regression, multilayer perceptrons, and convolutional networks all use a batch size of 128, and the optimizers being compared start from the same initial parameters; hyperparameters such as the learning rate and momentum are searched over a dense grid, and the best setting is reported. The paper plots only loss curves, and for the first three experiments it states explicitly that these are training losses; there is no test accuracy. Keep this in mind when reading the figures, because what these experiments compare is who drives the training loss down faster.

Logistic regression

Two line charts. The left is logistic regression on MNIST, with epochs (0 to 45) on the horizontal axis and training cost on the vertical axis: Adam (red) and SGD with Nesterov momentum (green) both drop quickly to about 0.26 to 0.27, while AdaGrad (blue) is still around 0.46 at epoch 45. The right is logistic regression on IMDB bag-of-words features, with dropout added for all four methods: the three lines for Adagrad, RMSProp, and Adam all drop to about 0.25, while SGD Nesterov (red) oscillates heavily and stays near 0.35.
Original Figure 1, Kingma and Ba, from PDF page 6, arXiv license, scholarly quotation.

Logistic regression is a convex problem with no local minima to worry about, which makes it a clean setting for comparing optimizers. In this experiment the learning rate decays as α/t\alpha/\sqrt t, following the theory in Section 4.

The left panel is MNIST, classifying the raw 784-dimensional pixel vectors. Adam and SGD with momentum are about even, and both are much faster than AdaGrad.

The right panel is what Adam really sets out to prove. The authors turn IMDB movie reviews into bag-of-words vectors over the 10,000 most frequent words; each review uses only a few of those words, so the features are very sparse. To prevent overfitting, 50% dropout is applied to the features. Here AdaGrad is far ahead of SGD, and Adam is just as fast as AdaGrad. Adam has inherited AdaGrad's advantage on sparse features.

Multilayer perceptron

Next is a fully connected network with two layers of 1000 ReLU units each, again on MNIST. In the version with dropout, Adam's training loss is lower than that of AdaGrad, RMSProp, SGD with momentum, and AdaDelta.

For the version without dropout, which uses L2 weight decay instead, the authors also compare against SFO. SFO is a quasi-Newton method that operates on mini-batches and updates its curvature information at every step; it is 5 to 10 times slower than Adam, and its memory grows linearly with the number of mini-batches. Adam is faster whether measured in iterations or in wall-clock time, and since SFO assumes each sub-function is deterministic, it fails to converge once dropout is involved.

Convolutional network

Two plots of training cost for a convolutional network on CIFAR-10. The left covers only the first 3 epochs: Adam (magenta) and Adam with dropout (yellow) drop fastest, reaching about 0.6 to 0.7 by epoch 3; AdaGrad (blue) is about 1.2. The right covers 45 epochs with a log-scale vertical axis: Adam and SGD Nesterov (red) drop sharply around epoch 15 and end at about 4 to 6 times 10 to the minus 4; AdaGrad and AdaGrad with dropout stall at 0.6 to 0.8; Adam and SGD with dropout are at about 0.02 to 0.04.
Original Figure 3, Kingma and Ba, from PDF page 7, arXiv license, scholarly quotation.

The network has three stages of 5×55\times5 convolutions with 3×33\times3 max pooling at stride 2, with 64, 64, and 128 channels, followed by a fully connected layer of 1000 ReLUs; the inputs are whitened, and dropout is applied to the input layer and the fully connected layer.

Over the first 3 epochs on the left, both Adam and AdaGrad drop quickly. Stretch it out to 45 epochs on the right, though, and AdaGrad stalls early, while Adam and SGD with momentum keep going down below 10−310^{-3}.

The authors' explanation for this result is quite specific. They observed that after a few epochs of training, v^t\hat v_t becomes so small that it is nearly 0, the denominator is dominated by ϵ\epsilon, and the second moment no longer describes the geometry of the problem; what does the work in convolutional networks is mainly the first moment, which averages out mini-batch noise. Following this explanation through, when the denominator is dominated by ϵ\epsilon, Adam behaves more like SGD with momentum, which also fits how close the two curves are. The authors are candid that Adam is only a marginal improvement over SGD with momentum, and that its benefit is in automatically adjusting the learning-rate scale for different layers, with no need to hand-pick a smaller learning rate for the convolutional layers.

ϵ\epsilon is described as "a small constant to avoid division by zero," but once v^t\sqrt{\hat v_t} shrinks to the same order of magnitude, it is actually taking part in the update.

The bias-correction experiment

Two groups of small plots, each 2 rows by 3 columns, from training a variational autoencoder. The left group is after 10 epochs of training, the right group after 100 epochs; each row is β1 = 0 or 0.9, and each column is β2 = 0.99, 0.999, or 0.9999. In each small plot the horizontal axis is log10(α), from −5 to −1, and the vertical axis is the loss, ranging from 95 to 140. The red line is with bias correction, the green line without. At β2 = 0.99 the two lines nearly overlap; the closer β2 is to 1, the earlier the green line shoots off the plot. At 10 epochs with β1 = 0, the green lines for β2 = 0.999 and 0.9999 are almost entirely off the plot, and the red line bottoms out at about 110. After 100 epochs, the red line's lowest point in all six plots is around 100.
Original Figure 4, Kingma and Ba, from PDF page 8, arXiv license, scholarly quotation.

This is the figure in the paper that tests the design most directly. The authors train a variational autoencoder (VAE, with a single hidden layer of 500 softplus units and a 50-dimensional Gaussian latent variable), sweeping β1∈{0,0.9}\beta_1\in\{0,0.9\}, β2∈{0.99,0.999,0.9999}\beta_2\in\{0.99,0.999,0.9999\}, and α\alpha from 10−510^{-5} to 10−110^{-1}. The paper treats Adam with bias correction removed as a form of RMSProp with momentum.

In each small plot, the horizontal axis is the learning rate and the vertical axis is the loss; red is with correction and green is without.

  • In the β2=0.99\beta_2=0.99 column, the red and green lines roughly overlap; only in the β1=0\beta_1=0 row does the green line shoot off the plot first, at the largest learning rate. With this setting, the bias from skipping the correction disappears within one or two hundred steps and has little effect on training as a whole.
  • Moving right to β2=0.9999\beta_2=0.9999, the green line shoots off the plot already at the larger learning rates. The worst case is the first row of the left group, β1=0\beta_1=0, where the green lines for β2=0.999\beta_2=0.999 and 0.99990.9999 are almost entirely off the plot; without correction, the first step in these two cells is 31.6 α31.6\,\alpha and 100 α100\,\alpha, respectively.
  • In the right group, trained to 100 epochs, the red line's lowest point is around 100 in all six plots. In the β2=0.9999\beta_2=0.9999 column, the red line bottoms out at α=10−3\alpha=10^{-3}; at that same learning rate, the green line has already shot off the plot for β1=0\beta_1=0 and climbed to about 110 for β1=0.9\beta_1=0.9. The red line has a much wider range of usable learning rates.

The authors conclude that when β2\beta_2 is close to 1, the lack of bias correction makes training unstable, especially in the first few epochs, and that the best results come from settings with a very small 1−β21-\beta_2 combined with bias correction. Across all the hyperparameters they swept, Adam was never worse than RMSProp.

The authors add one more observation. Late in training, hidden units specialize and the gradients become sparser, which is when the benefit of a large β2\beta_2 shows most clearly. In other words, a β2\beta_2 close to 1 is the setting we wanted all along, and bias correction lets us use it with confidence.

Where ϵ\epsilon goes

Back to the difference between the two formulations from earlier. Algorithm 1 adds ϵ\epsilon to the corrected v^t\sqrt{\hat v_t}, while the cheaper form adds ϵ^\hat\epsilon to the uncorrected vt\sqrt{v_t}. Converting between them gives ϵ^=ϵ1−β2t\hat\epsilon=\epsilon\sqrt{1-\beta_2^t}, and the two differ most early in training.

Different frameworks picked different forms. PyTorch follows Algorithm 1, with a default of eps=1e-8; Keras's documentation states that its epsilon is the paper's ϵ^\hat\epsilon, with a default of 1e-7. When moving hyperparameters between these two frameworks, the thing called epsilon on each side is actually a different quantity.

This seemingly unimportant constant gets tuned in large-model training, too. OLMo 2 lowered AdamW's ϵ\epsilon from 10−510^{-5} to 10−810^{-8} and reported that gradient norms early in training became lower and more stable as a result.

The years without bias correction

The derivation in Section 3 sounds like a minor detail, but it has a real-world sequel.

BERTAdam, the optimizer in BERT's official code, dropped those two bias-correction lines, and this version was then copied into widely used libraries such as Hugging Face Transformers (which only switched back to standard Adam in July 2019) and AllenNLP. In 2020, Zhang et al.'s "Revisiting Few-sample BERT Fine-tuning" traced why BERT fine-tuning is unstable on small datasets and found this to be one of the main causes.

They fine-tuned on four small datasets, RTE, MRPC, STS-B, and CoLA, with 50 random seeds each. Without bias correction, many fine-tuning runs never got off the ground; on RTE, for example, 48% of runs had an accuracy below 55%, close to random guessing. With bias correction restored, the variance of the results shrank substantially.

The reason is the same as in the step-size plot above. Fine-tuning on these small datasets usually takes fewer than a thousand steps, so the entire run sits in the stretch where the bias is largest. On MNLI, which trains for many more steps, the difference with and without correction falls within the margin of error (validation accuracy of 86.0 without correction and 85.9 with it).

Three questions that followed

Adam quickly became the default optimizer, and over the next few years, the challenges to it fell roughly along three lines:

  1. Convergence. This is the proof error mentioned earlier. Reddi et al. used an example with a large gradient once every three steps to show that vtv_t "forgets" rare large gradients, and proposed AMSGrad, which keeps the historical maximum.
  2. Generalization. In 2017, Wilson et al. found on several vision and language tasks that Adam drives the training loss down quickly but often loses to well-tuned SGD on the test set.
  3. Regularization. Loshchilov and Hutter pointed out that in Adam, writing L2 regularization into the gradient is not equivalent to applying weight decay directly, and proposed AdamW.

Conclusion

Adam uses two exponential moving averages to estimate the mean and the mean square of the gradient, and dividing one by the other gives each parameter a step size of its own; it then uses 1−βt1-\beta^t to correct the bias from starting at 0, so that even the first step lands on the scale of α\alpha. The paper's most convincing evidence is the VAE experiment, which shows that the closer β2\beta_2 is to 1, the more bias correction matters; the other experiments compare only training loss, and on convolutional networks its margin over SGD with momentum is small. The convergence theorem was shown to be wrong three years later.

If you're implementing Adam yourself, or reusing someone else's training code, first check whether those two bias-correction lines are still there, then check whether ϵ\epsilon is added before or after bias correction, and what its value is. The fewer training steps you have, the more these two details affect the results; and when moving hyperparameters between frameworks, remember that the epsilon on each side may not be the same quantity.

In 2025, this paper won the ICLR Test of Time Award, seven years after its convergence proof was shown to be flawed. As for the author order, that still comes down to the coin.