Back to paper library
ICLR 2018arXiv 2019OptimizersPaper notes

AMSGrad

Out of sight, out of mind

Original paper

On the Convergence of Adam and Beyond

Paper authors

Sashank J. Reddi · Satyen Kale · Sanjiv Kumar

Publication
ICLR 2018Conference
First submitted to arXiv
arXiv:1904.09237
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{optimizersamsgrad2019,
  title = {{On the Convergence of Adam and Beyond}},
  author = {Sashank J. Reddi and Satyen Kale and Sanjiv Kumar},
  year = {2019},
  eprint = {1904.09237},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1904.09237}
}
On this page

Let's start with a little game.

The parameter xx has to stay between −1-1 and 11. Every three steps make up one round; the loss at the first step is 3x3x, and at the second and third steps it is −x-x. A round adds up to 3x−x−x=x3x-x-x=x, so the smaller xx the better, and the answer is x=−1x=-1.

The gradients are easy to work out, +3+3 at the first step and −1-1 at the next two. SGD takes one net step in the negative direction each round and ends up at −1-1.

Adam ends up at x=+1x=+1, the worst point in the whole interval.

This example comes from Sashank Reddi, Satyen Kale, and Sanjiv Kumar at Google. They used it to prove that the convergence theorem in the original Adam paper is wrong, and then, working from where the proof goes wrong, derived a fix that changes a single line, AMSGrad.

Defining the problem

Adam's claimed convergence

The Adam paper analyzes convergence in the framework of online convex optimization. At each step, the algorithm first picks a parameter xtx_t, and only then sees this step's convex loss function ftf_t. The yardstick is regret, the amount by which the cumulative loss exceeds that of "the best fixed parameter in hindsight":

RT=∑t=1Tft(xt)−min⁡x∈F∑t=1Tft(x)R_T=\sum_{t=1}^{T}f_t(x_t)-\min_{x\in\mathcal F}\sum_{t=1}^{T}f_t(x)

If the average regret RT/TR_T/T goes to 0, then on average the algorithm has caught up with the best fixed solution. Theorem 4.1 of the Adam paper claims that under conditions including bounded gradients, β12/β2<1\beta_1^2/\sqrt{\beta_2}<1, and a learning rate of αt=α/t\alpha_t=\alpha/\sqrt t, Adam's regret is O(T)O(\sqrt T) and its average regret goes to 0.

The game at the top of this note is exactly an online convex optimization problem. Each ftf_t is a linear function, and the feasible set is [−1,1][-1,1]. Adam stays at +1+1 and incurs a fixed amount of extra loss every round compared with x=−1x=-1, so its average regret never goes to 0.

Can the effective learning rate grow?

Reddi et al. point to one particular quantity. Write the various adaptive methods in a common framework, where each update is xt+1=xt−αtmt/Vtx_{t+1}=x_t-\alpha_t m_t/\sqrt{V_t}, so that αt/Vt\alpha_t/\sqrt{V_t} is the actual learning rate of each parameter. Now look at how its inverse changes over time:

Γt+1=Vt+1αt+1−Vtαt\Gamma_{t+1}=\frac{\sqrt{V_{t+1}}}{\alpha_{t+1}}-\frac{\sqrt{V_t}}{\alpha_t}

For SGD, VtV_t is fixed and αt\alpha_t decreases. For AdaGrad, VtV_t is the average of past squared gradients, and paired with α/t\alpha/\sqrt t, what actually sits in the denominator is the sum of squared gradients, which only ever grows. For both methods Γt\Gamma_t is never negative, and the learning rate can only shrink.

Adam and RMSProp use exponential moving averages. As soon as recent gradients get smaller, vtv_t shrinks with them and the learning rate grows instead, so Γt\Gamma_t can be negative.

The paper opens by describing what people see in practice. In problems with very large output spaces, some mini-batches bring large, informative gradients, but they show up only rarely; the exponential moving average makes their influence fade quickly, and training converges poorly. This paper sets out to turn that intuition into a provable counterexample.

Solving the problem

Counterexample: a large gradient every three steps

Theorem 1 of the paper is the general version of the opening game. The loss functions are, in order,

ft(x)={Cx,t mod 3=1−x,otherwisex∈[−1,1], C>2f_t(x)= \begin{cases} Cx, & t \bmod 3 = 1\\ -x, & \text{otherwise} \end{cases} \qquad x\in[-1,1],\ C>2

The gradients over one round sum to C−2>0C-2>0, and the optimum is x=−1x=-1. Adam is set to β1=0\beta_1=0, β2=1/(1+C2)\beta_2=1/(1+C^2), αt=α/t\alpha_t=\alpha/\sqrt t, with α<1−β2\alpha<\sqrt{1-\beta_2}; these parameters satisfy the Adam paper's requirement that β12/β2<1\beta_1^2/\sqrt{\beta_2}<1. The paper's appendix proves by induction that, starting from x1=1x_1=1, xx returns to 11 at the end of every round.

Why? Take C=3C=3, which makes β2=0.1\beta_2=0.1, and compute one round directly. After a few rounds vtv_t settles into a fixed cycle, and the table below shows one round of that cycle. The last two columns are gt/vtg_t/\sqrt{v_t}, where a positive value pushes xx toward −1-1 and a negative value pushes it toward +1+1. The actual update also gets multiplied by αt\alpha_t, but αt\alpha_t is about the same for the three steps within a round, so we'll ignore it for now:

Stepgtg_tvtv_tAdamAMSGrad
1+3+38.21+1.05+1.05+1.05+1.05
2−1-11.72−0.76-0.76−0.35-0.35
3−1-11.07−0.97-0.97−0.35-0.35
Round+1+1−0.68-0.68+0.35+0.35

Computed for this note with C=3C=3, β1=0\beta_1=0, β2=0.1\beta_2=0.1, the setting of Theorem 1. AMSGrad's denominator uses the largest vv so far, which is 8.21.

Look at the Adam column first. The gradient at the first step is +3+3, but it also pumps vtv_t up to 8.21, and 3/8.21≈1.053/\sqrt{8.21}\approx1.05, so the large gradient gets shrunk by nearly a factor of 3. When the paper says the large gradient CC is scaled down "by a factor of almost CC," this is what it means. For the next two steps the gradient is only −1-1, and vtv_t falls back to 1.72 and then 1.07, keeping only a tenth of its value at each step; the learning rate grows nearly threefold within two steps, and the two small gradients together push by 1.731.73. Over the round, the net effect is a move of 0.680.68 in the wrong direction.

This is what Γt<0\Gamma_t<0 looks like in concrete terms. Two steps later, the lesson taught by the large gradient is mostly forgotten.

Running this setup with α=0.5\alpha=0.5 for three hundred thousand steps, Adam stays at x=1x=1 with an average regret of 0.6670.667, exactly the lower bound (2C−4)/3(2C-4)/3 given in the paper; under the same conditions, AMSGrad reaches −0.999-0.999.

The paper adds a few more points. The example holds just as well with a constant step size αt=α\alpha_t=\alpha. Adding ϵ\epsilon to the denominator doesn't rescue Adam either; Theorem 6 in the appendix constructs a similar counterexample for any ϵ>0\epsilon>0. On the same problem, SGD and AdaGrad both have average regret going to 0.

With the β\beta values people actually use

β2=0.1\beta_2=0.1 is far too contrived; in practice, β2\beta_2 is always close to 1. The paper goes on to prove two stronger results.

Theorem 2 says that as long as β1<β2\beta_1<\sqrt{\beta_2} (which nearly every practical setting satisfies), there exists an online convex problem on which Adam's average regret doesn't go to 0. The construction is the same, except that the period stretches from 3 to CC; once every CC steps there is a gradient of CC, and all the rest are −1-1. CC has to be large enough that β2\beta_2's memory can't make it through a full period.

How large is large enough? The appendix lists three conditions, one of which is β2(C−2)/2C2≤1\beta_2^{(C-2)/2}C^2\le1. Plugging in numbers ourselves, CC must be at least 3,218 for β2=0.99\beta_2=0.99 (Appendix B also requires CC to be even; the inequality alone gives 3,217) and at least 42,622 for β2=0.999\beta_2=0.999. In other words, a gradient more than forty thousand times larger than usual, showing up once every forty-some thousand steps.

Theorem 3 moves to stochastic optimization. Each step is sampled independently, seeing CxCx with probability p=(1+δ)/(C+1)p=(1+\delta)/(C+1) and −x-x otherwise. The expected loss is δx\delta x and the optimum is still −1-1, but when CC is large enough, the expected displacement of Adam at each step is positive, and the parameter drifts away from the optimum.

The authors conclude that to avoid this behavior, ϵ\epsilon, β1\beta_1, and β2\beta_2 would all have to depend on the problem; high-dimensional problems might even need a separate set for each dimension, which defeats the whole point of adaptive methods needing no tuning. They admit the examples are contrived, but argue that similar phenomena in real settings would at least slow convergence down.

Where the proof goes wrong

With a counterexample in hand, we know some step in Adam's original proof has to fail. In Section 4 and Appendix D, the paper points out that Adam's proof assumes Γt\Gamma_t is positive semi-definite, that is, that the learning rate never grows; this assumption doesn't hold for Adam. In addition, Lemmas 10.3 and 10.4 in Adam's proof also have problems.

The learning rates of SGD and AdaGrad are non-increasing to begin with, so they never run into this. This observation points straight to the fix.

AMSGrad

Since the problem is that vtv_t can shrink, let the denominator remember the historical maximum:

mt=β1t mt−1+(1−β1t) gtvt=β2 vt−1+(1−β2) gt2v^t=max⁡(v^t−1, vt)xt+1=ΠF,V^t(xt−αt mt/v^t)\begin{aligned} m_t &= \beta_{1t}\, m_{t-1} + (1-\beta_{1t})\, g_t\\ v_t &= \beta_2\, v_{t-1} + (1-\beta_2)\, g_t^2\\ \hat v_t &= \max(\hat v_{t-1},\, v_t)\\ x_{t+1} &= \Pi_{\mathcal F,\sqrt{\hat V_t}}\big(x_t-\alpha_t\, m_t/\sqrt{\hat v_t}\big) \end{aligned}

The only difference from Adam is the third line. vtv_t is still updated as an exponential moving average, so its estimate of the gradient scale still looks only at the recent past; but the v^t\hat v_t actually used as the denominator never decreases, so the learning rate αt/v^t\alpha_t/\sqrt{\hat v_t} is non-increasing and Γt\Gamma_t is guaranteed to be positive semi-definite. The extra cost is just one vector the same size as the parameters.

Go back to the AMSGrad column of the table above. The denominator stays at 8.21, the two −1-1 gradients each push by only 0.350.35, and the net effect over a round becomes a move of 0.350.35 in the right direction.

The authors compare it with Adam and AdaGrad in one scenario. Suppose that at some step vt−1>gt2>0v_{t-1}>g_t^2>0, meaning this step's gradient is smaller than the recent average:

  • Adam aggressively increases the learning rate, which is exactly what sank it in the counterexample above;
  • AdaGrad slightly decreases the learning rate, and over the long run the learning rate can get so small that the parameters barely move;
  • AMSGrad neither increases nor decreases it.

The convergence guarantee for AMSGrad (Theorem 4) reuses the Adam paper's analysis framework with the problem above fixed. The conditions are αt=α/t\alpha_t=\alpha/\sqrt t, β1t≤β1\beta_{1t}\le\beta_1, β1/β2<1\beta_1/\sqrt{\beta_2}<1, and a bounded feasible-set diameter and bounded gradients, and the regret bound picks up an extra 1+log⁡T\sqrt{1+\log T}; the paper writes that choosing β1t=β1λt−1\beta_{1t}=\beta_1\lambda^{t-1}, or the milder β1/t\beta_1/t, guarantees O(T)O(\sqrt T) regret.

Two details are worth noting. The proof needs β1\beta_1 to decrease over time, but the authors also state that in practice β1\beta_1 is usually kept constant. And the Adam in this paper has bias correction removed; a footnote says this is for simplicity and that the counterexample argument applies equally to the corrected version. AMSGrad's Algorithm 2 has no bias correction either.

The paper also offers another route, AdamNC, which keeps the structure the same but lets β2\beta_2 increase over time. With β2t=1−1/t\beta_{2t}=1-1/t, vtv_t is exactly the average of all past squared gradients, which amounts to AdaGrad with momentum.

Discussion

Synthetic experiments

Three line charts, all with iterations (in millions) on the horizontal axis. The left shows average regret in the online setting: Adam (blue dashed) levels off at about 0.21, while AMSGrad (green solid) keeps falling to near 0. The middle shows the parameter x in the online setting: Adam rises quickly to 1 and stays there, while AMSGrad stays at −1. The right shows the parameter x in the stochastic setting: Adam stays near 1, while AMSGrad works its way down from 1 and is around −0.85 at about ten million steps.
Original Figure 1, Reddi et al., from PDF page 6, arXiv license, scholarly quotation.

The experiment uses a scaled-up version of the counterexample. Once every 101 steps the loss is 1010x1010x, and for the other 100 steps it is −10x-10x; the gradients over one period sum to +10+10, and the optimum is still −1-1. This time the usual β1=0.9\beta_1=0.9 and β2=0.99\beta_2=0.99 are used, with the same parameters for both methods.

The left panel is the average regret. Adam's blue dashed line flattens out around 0.21 and stops falling, while AMSGrad's green line falls to near 0. The middle panel shows the parameter itself, with Adam at +1+1 and AMSGrad at −1-1.

The right panel is the stochastic version, where each step sees 1010x1010x with probability 1% and −10x-10x otherwise. Adam again stays at +1+1. AMSGrad moves down slowly; at ten million steps it is at roughly −0.85-0.85, still short of −1-1.

Real data

The top row's three plots are all on MNIST, with 0 to 5000 iterations on the horizontal axis: in the training and test loss of logistic regression, AMSGrad (green) is slightly below Adam (blue dashed), ending at about 0.25 and 0.27; in the training loss of a one-hidden-layer network, AMSGrad is also slightly lower. The bottom row's two plots are CIFARNET on CIFAR-10, with up to about 2 million iterations on the horizontal axis: the training loss is about 0.3 for AMSGrad and about 0.4 for Adam; the test loss of both oscillates around 0.6, with AMSGrad slightly lower.
Original Figure 2, Reddi et al., from PDF page 7, arXiv license, scholarly quotation.

There are three real-data experiments, all with a batch size of 128 and β1=0.9\beta_1=0.9, with β2\beta_2 chosen from {0.99,0.999}\{0.99,0.999\}:

  1. Logistic regression on MNIST (top row, left and middle). A convex problem, so the learning rate decays as α/t\alpha/\sqrt t as the theory prescribes, with α\alpha and β2\beta_2 chosen by grid search. AMSGrad's training and test losses are both slightly lower, and the authors also mention that it is less sensitive to changes in its parameters.
  2. A one-hidden-layer network on MNIST (top row, right). 100 ReLUs and a constant learning rate. AMSGrad's training loss stays slightly below Adam's the whole way.
  3. CIFARNET on CIFAR-10 (bottom row). Two layers of 64-channel 6×66\times6 convolutions, followed by fully connected layers of 384 and 192 units, with max pooling, local response normalization, and dropout in between. The gap in training loss is the most visible, roughly 0.3 versus 0.4; the test loss of both is around 0.6, with AMSGrad slightly lower.

Keep the scope of these results in mind. The figures show only loss, not accuracy (the text says AMSGrad's accuracy is also better, but gives no numbers); each curve appears to be a single run, without error bars; and the models are all small. In their own list of contributions, the authors call this a preliminary empirical study.

The practical response

AMSGrad was quickly added to the major frameworks. Adam in both PyTorch and Keras has an amsgrad parameter, off by default.

Once people actually tried it, the feedback was lukewarm. In a July 2018 post, fast.ai's Sylvain Gugger and Jeremy Howard compared it on a Wide ResNet on CIFAR-10, ResNet-50 fine-tuned on Stanford Cars, and AWD-LSTM and QRNN language models on WikiText-2, writing that amsgrad consistently failed to bring any gain in accuracy or any other relevant metric, and calling it "very disappointing."

AMSGrad's own proof was later found to have a problem too. In 2019, Phuong and Phong pointed out that its proof treats β1t\beta_{1t} and β1\beta_1 as if they were equal, the same slip found in Adam's original proof; they gave a new proof and also proposed another variant, AdamX.

Does Adam converge after all?

The counterexample stands, and yet Adam really does work well in practice. How do the two fit together? The answers from several follow-up works differ in what they fix first:

WorkWhat is fixed firstConclusion
Reddi et al. 2018Pick β\beta first, then construct the problemAdam can fail to converge
Zhang et al. 2022Give the problem first, then pick β\betaConverges if β2\beta_2 is large enough
ADOPT 2024Change the algorithmConverges for any β2\beta_2

Zhang et al. 2022 put a finger on the question of order. Reddi et al. fix (β1,β2)(\beta_1,\beta_2) first and then construct a bad problem for that choice; as computed above, β2=0.999\beta_2=0.999 needs a period of more than forty thousand steps. In practice, though, the problem is there first, and then we tune the hyperparameters. They prove that for a fixed problem, as long as β2\beta_2 is large enough (with a threshold that depends on the problem) and β1<β2\beta_1<\sqrt{\beta_2}, Adam with its update rule unchanged converges to a neighborhood of critical points whose size is proportional to the variance of the stochastic gradients; under stronger conditions it converges to the critical points themselves. This analysis targets non-convex finite-sum problems with the samples randomly reshuffled every epoch, doesn't need the bounded-gradient assumption, and covers the default β1=0.9\beta_1=0.9. Conversely, when β2\beta_2 is too small, there is a large region of (β1,β2)(\beta_1,\beta_2) that diverges.

Around the same time, Défossez et al. gave a short convergence proof for smooth non-convex problems with bounded gradients, and also pointed out that Adam with default parameters, strictly speaking, does not converge, behaving more like SGD with a constant step size.

ADOPT 2024 takes a fresh look at AMSGrad. It points out that the convergence proofs for AMSGrad on non-convex problems all assume uniformly bounded stochastic gradients, and that models which inject Gaussian noise into their gradients, such as variational autoencoders and diffusion models, don't satisfy this condition. The trouble lies in taking the maximum, whose expectation is hard to control. ADOPT instead normalizes the current gradient with the previous step's vt−1v_{t-1}, so the denominator doesn't contain the current gradient, and it normalizes first and accumulates momentum afterward. Assuming only that the gradient's second moment is finite, it achieves an O(1/T)O(1/\sqrt T) convergence rate for any β2\beta_2.

One last footnote. A July 2026 preprint, "On the Convergence of Adam, Revisited," claims that in the same online setting as Reddi et al., even the condition β1<β2\beta_1<\sqrt{\beta_2} isn't needed, and that for any β1,β2\beta_1,\beta_2 one can find an example where Adam's average regret doesn't vanish, still using a three-step cycle with slopes c,−1,−1c,-1,-1; the result also extends to variants such as AdamW and Muon. This doesn't contradict Zhang et al.'s positive result, since the two answer the question in opposite orders.

Conclusion

Using a linear function with three-step rounds, this paper proved that Adam's original convergence theorem is wrong, and traced the problem to the exponential moving average allowing the learning rate to grow. AMSGrad uses the historical maximum as the denominator, which makes the learning rate non-increasing and restores the convergence guarantee for convex problems. The gains on real data are small, the experiments only go as far as a preliminary study, and later independent tests mostly saw no benefit. What this paper left to later research is the Γt\Gamma_t angle of analysis, along with the question of which side gets fixed first.

In practice, amsgrad=True is just a switch. If you suspect your data has rare but important large gradients, you can put it on the list to compare, alongside raising β2\beta_2; the former adds only one parameter-sized state, and the latter adds no cost at all. Zhang et al. proved that a large region diverges when β2\beta_2 is too small. Going by that result, if you lowered β2\beta_2 to make the optimizer react faster and training becomes unstable, the first thing to do is move it back close to 1.

The little three-step game, for its part, never retired. Eight years later it's still the same c,−1,−1c,-1,-1, only this time without even assuming β1<β2\beta_1<\sqrt{\beta_2}, and it drags AdamW and Muon along for the ride.