On the Variance of the Adaptive Learning Rate and Beyond
- Publication
- ICLR 2020Conference
- First submitted to arXiv
- arXiv:1908.03265
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{optimizersradam2019,
title = {{On the Variance of the Adaptive Learning Rate and Beyond}},
author = {Liyuan Liu and Haoming Jiang and Pengcheng He and Weizhu Chen and Xiaodong Liu and Jianfeng Gao and Jiawei Han},
year = {2019},
eprint = {1908.03265},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/1908.03265}
}
If warmup is the answer, what is the question?
That is how RAdam introduces itself on GitHub. By then, warmup was standard equipment for training Transformers. You raised the learning rate slowly from near 0 over the first few thousand steps, and only then would Adam train. There was no theoretical explanation at the time for why it had to be done this way, and how long to warm up was mostly settled by trial and error.
In August 2019, Liyuan Liu and colleagues from the University of Illinois and Microsoft released this paper. They traced the cause to Adam's adaptive learning rate. Early in training it has seen too few gradients, so its variance grows out of control. Along the way they proposed a fix called Rectified Adam, or RAdam for short.
About a week after the paper went up on arXiv, Less Wright introduced it on the fast.ai forums and on Medium as a "New State of the Art AI Optimizer," and people on the forum soon put it to the test. Two months after that, Jerry Ma and Denis Yarats published a rebuttal, arguing that RAdam, taken apart, is just Adam plus a fixed warmup.
This note looks at both sides.
Defining the problem
Adam's adaptive learning rate
The authors first write the various adaptive optimizers in a single framework (Algorithm 1). Each step computes a momentum and an adaptive learning rate , then updates . All operations are element-wise. For Adam, is:
That is, the reciprocal of the square root of the second-moment estimate, with the in the numerator serving as bias correction. Parameters whose gradients stay large take smaller steps; parameters whose gradients stay small take larger ones.
What happens without warmup

Look first at the blue Adam-vanilla line and at Adam-warmup at the bottom. With the same Transformer and the same IWSLT'14 German-to-English data, the training loss stalls around 10 without warmup and drops below 3 with it. The other three lines are the authors' controlled experiments, which we'll get to shortly.
The authors plotted the distribution of absolute gradient values at each step and found that without warmup, the distribution gets distorted within the first 10 steps, its mass crowding into very small values. Their reading is that Adam makes harmful updates in the first few steps that carry the model into a bad region, and warmup works by reducing the impact of these updates.
So why are the first few updates harmful?
Solving the problem
The first step has only one sample
Start with the most extreme case, . Here , so the entire adaptive learning rate is estimated from a single gradient.
If the gradient follows , then follows a scaled inverse chi-squared distribution with 1 degree of freedom, and the variance of diverges. The intuition is simple enough. If happens to land very close to 0, becomes enormous.
A small learning rate can hold this variance down, since . That is the authors' explanation of warmup. Early in training, the adaptive learning rate's variance is too large, and warmup suppresses it with a small .
Two controlled experiments
If the problem really is the variance, then reducing the variance some other way should also fix it. The authors designed two Adam variants.
- Adam-2k: for the first 2,000 iterations, only is updated, while momentum and parameters stay frozen. In effect, it looks at 2,000 extra samples to get an accurate second-moment estimate before training starts.
- Adam-eps: raises the in the denominator from to , directly capping .
In the figure above, Adam-2k has no warmup, yet it overlaps with Adam-warmup, and the gradient distribution is no longer distorted. Adam-eps also avoids getting stuck at 10, but ends up around 3.6, worse than warmup. The authors conjecture that a large introduces bias into the adaptive learning rate and slows optimization down, so a more principled method is needed.
Variance falls as samples grow
Next comes computing the variance. Adam uses an exponential moving average (EMA), which is hard to analyze, so the authors approximate it with a simple moving average (SMA) and assume follows a scaled inverse chi-squared distribution with degrees of freedom. Here can be read as "how many samples the second-moment estimate effectively uses."
Theorem 1 says that under this assumption, the variance of decreases monotonically as grows, and gives a closed form for (Eq. (2)). In other words, the fewer the samples, the less reliable the adaptive learning rate; for , the paper treats the variance as intractable.
How many samples is an EMA worth?
So how long an SMA is the EMA at step equivalent to? The authors require the two to share the same "center of mass," meaning the weighted average of the samples' positions in time must match, and solve for:
is the limit as . Plugging in the default gives . For the first few steps, works out to:
| 1 | 2 | 3 | 4 | 5 | |
|---|---|---|---|---|---|
| 1.000 | 1.999 | 2.999 | 3.997 | 4.996 |
Computed from the paper's definition of , with .
Early in training, is almost exactly , which matches intuition. However many gradients it has seen is how many samples it effectively uses. The closer is to 1, the longer the EMA's memory, and the longer takes to approach its limit.
The rectification term
With in hand, the authors set the goal of keeping the adaptive learning rate's variance the same at every step, equal to its minimum at . They do this by multiplying by a scaling factor .
The closed form in Eq. (2) is numerically unstable, so the authors use a first-order approximation instead, giving Eq. (5):
Plug in and in turn and divide, and cancels out:
This is the rectification in RAdam's name. Let's compute a few points to see what it looks like:
| Step | 5 | 10 | 100 | 1,000 | 2,000 | 5,000 |
|---|---|---|---|---|---|---|
| 0.017 | 0.049 | 0.215 | 0.645 | 0.828 | 0.983 |
Computed from in Section 4.2 of the paper, with .
It starts out tiny, reaches about 0.65 at a thousand steps, and only nears 1 at five thousand. Multiply the learning rate by this curve, and it looks just like a warmup.
The full algorithm
Put together as Algorithm 2, each step does the following:
- Compute the gradient , and update the second moment and the first moment .
- Bias-correct the first moment, .
- Compute .
- If , compute and , and update .
- Otherwise, skip the adaptive learning rate and simply set .
Item 5 is where RAdam degenerates into SGD with momentum. From the table above, with only through 4 have , so the first 4 steps are bias-corrected momentum SGD, and the rectified Adam only kicks in from step 5. Ma and Yarats later proved that for any , is exactly equivalent to . The paper also notes that if , then , and RAdam degenerates entirely into SGD with momentum.
Relation to warmup
In Section 4.3, the authors compare directly with linear warmup. Linear warmup amounts to multiplying the learning rate by , which is similar in form to , so they take this as verifying that "warmup works as a variance reduction technique." The difference is that RAdam needs no warmup-length hyperparameter , and adjusts automatically with .
They also stress that this fix is orthogonal to other techniques for stabilizing training, such as gradient clipping, a larger , initialization and normalization layers, and can be used alongside them.
Discussion
Experimental setup
The experiments fall into three groups, and the Adam-family methods all use AdamW's decoupled form of weight decay.
- Language modeling: One Billion Word, a two-layer LSTM with 2,048 hidden units, learning rate 0.001, batch size 128.
- Image classification: ResNet-20 on CIFAR-10 (186 epochs) and ResNet-18 on ImageNet (90 epochs), with step-wise learning-rate decay and weight decay .
- Machine translation: fairseq's Transformer, on IWSLT'14 German–English in both directions and on WMT'16 English-to-German; Adam uses 4,000 steps of linear warmup.
In the language modeling and image experiments, Adam and RAdam both set to .
Against vanilla Adam
Without warmup, RAdam beats vanilla Adam on all three datasets. Test perplexity on One Billion Word drops from 36.92 to 35.70; RAdam is slower than Adam for the first few epochs, then converges faster. Test accuracy for image classification is as follows:
| Dataset | SGD | Adam | RAdam |
|---|---|---|---|
| CIFAR-10 | 91.51 | 90.54 | 91.38 |
| ImageNet | 69.86 | 66.54 | 67.62 |
Compiled from Table 1 of the paper; in %.
RAdam is better than Adam, but it doesn't beat SGD on either dataset. The authors also report ImageNet training accuracy, with SGD, Adam and RAdam at 69.57, 69.12 and 70.30, RAdam the highest. It trains better than SGD yet tests worse, so the gap appears to come from generalization. That said, this is an inference worked back from the training accuracy; the paper itself doesn't analyze it this way.
Insensitive to the learning rate

As the learning rate varies from 0.1 to 0.003, RAdam's four test-accuracy curves nearly lie on top of each other; Adam drops to about 86% at a learning rate of 0.1, and SGD's four lines each go their own way. The authors read this as rectification letting RAdam handle a wider range of learning rates. Still, the README's FAQ candidly says "the robustness of RAdam is not infinity," and the learning rate still needs tuning.
Against warmup
Translation is where warmup is hardest to do without, so here the opponent is a tuned Adam with warmup:
| Task | Adam + warmup | RAdam |
|---|---|---|
| IWSLT'14 De→En | 34.66 | 34.76 |
| IWSLT'14 En→De | 28.56 | 28.48 |
| WMT'16 En→De | 27.03 | 27.27 |
Compiled from Table 2 of the paper; BLEU, with IWSLT averaged over 5 random seeds.
It's a tie, and the authors' own conclusion is that RAdam "achieves similar performance to that of previous state-of-the-art warmup heuristics." One setting here only shows up in the appendix. The translation experiments set to 0.9995, and the appendix gives the reason as making . Plug it into and you get exactly 3,999, lined up with the 4,000-step warmup of the Adam runs. So in this comparison, the length of RAdam's "automatic warmup" was aligned through .
On CIFAR-10, Adam with warmup reaches a best accuracy of 91.29%, on par with RAdam's 91.38%; but the warmup length has options of 100, 200, 500 and 1,000 steps, and a learning rate of 0.1 with 100 warmup steps gets only 90.13%, against RAdam's 91.06% at the same learning rate. Stated more precisely, then, the authors' claim is that RAdam has one fewer hyperparameter to tune, while the best results themselves are about the same.
The momentum SGD steps at the start
Do the steps where it degenerates into momentum SGD matter? Appendix C runs a substitution experiment on CIFAR-10 at a learning rate of 0.1 (averaged over 5 runs). RAdam throughout gets a test accuracy of 91.08%; replacing the first 4 steps with plain Adam drops it to 89.98%; using SGD for the first 4 steps and unrectified Adam for steps 5 to 8 gives 90.29%. The authors see this as consistent with the theory, with earlier updates doing more damage, but also say this part is only a minor element of the algorithm's design; the focus is on the warmup mechanism and the derivation of the rectification term.
Ma and Yarats's rebuttal
In October 2019, Jerry Ma (University of Chicago) and Denis Yarats (New York University, Facebook AI Research) posted On the Adequacy of Untuned Warmup for Adaptive Optimization (AAAI 2021) on arXiv. The abstract gets straight to the point, stating that they "refute this analysis." The argument has three layers.
First, RAdam is a fixed warmup. depends only on and , and not at all on what the gradients actually look like. As mentioned above, covers exactly the first 4 steps, so RAdam can be summed up in one line as four steps of momentum SGD followed by Adam under a fixed warmup schedule. They also tried replacing the first 4 steps with doing nothing, or with Adam at a very small learning rate, and the differences were negligible.
Second, the variance argument has a hole. RAdam assumes the gradient has zero mean, but early in training the model is far from a minimum, and the gradient's mean is usually larger than its standard deviation; on EMNIST they measured that it takes until around step 50 for most parameters' coefficient of variation to stay above 1. More importantly, Adam's actual update is , and and are highly correlated. At step 1 they are exactly equal, so every parameter's update is exactly (with ). In other words, at the very step where the adaptive learning rate's variance "diverges," the variance of the actual update is 0.
Third, what matters is the size of the update. They ran a simulation in which the gradient around a minimum is pure zero-mean noise, and ran Adam to watch the update magnitudes. These start at and only settle at around after about 40 steps. So even when the model is already sitting at a minimum, Adam takes oversized steps for its first few dozen iterations; SGD in the same situation needs no warmup, which partly explains why Adam needs warmup more than SGD does.
Building on this analysis, they propose two rules of thumb that need no tuning:
The first one's time constant was chosen to mimic Adam's second-moment bias-correction term , and the authors themselves say this intuition is purely speculative; the second is a linear warmup over steps, which comes to 2,000 steps for .

The three curves don't have quite the same shape. RAdam rises fastest over the first few hundred steps, but after about 1,500 steps it ends up last. Ma and Yarats compare them by the "effective warmup period" , that is, how many steps' worth of learning rate the warmup suppresses in total; at all three come to about 1,000 steps (by the formula, RAdam's is about 1,033), and they stay nearly identical across values of .
The experiments back this up. On ResNet-50 on ImageNet, a Transformer language model on WikiText-103, and a Transformer on WMT16 English-to-German, they swept 9 to 12 combinations of learning rate and each, and the three methods mostly differ by only a few tenths. Take ImageNet top-1 error as an example (5 random seeds):
| Learning rate | Exponential | Linear | RAdam | |
|---|---|---|---|---|
| 0.99 | 27.9 | 28.0 | 28.4 | |
| 0.997 | 27.9 | 27.9 | 28.3 | |
| 0.999 | 28.2 | 28.3 | 28.4 |
Compiled from Table 1 of Ma & Yarats, showing the three rows with learning rate ; in %.
In these three rows, RAdam is actually slightly worse. The biggest gap in the whole table is at learning rate and , where RAdam's error is about 1 percentage point higher than the two rules of thumb. Their recommendation is simply to use a linear warmup over steps whenever you use Adam.
Who's right?
Both papers agree that Adam needs a smaller learning rate for its first few thousand steps; where they part ways is why.
On RAdam's side, the evidence is controlled experiments like Adam-2k, plus ties with tuned warmup on translation and images. Ma and Yarats, for their part, point out that is just a fixed schedule determined by , and use the update-magnitude analysis and experiments across three tasks to show that an untuned linear warmup does the same job. The final version of the RAdam paper doesn't respond to this rebuttal; the README's FAQ does concede that when the gradient's mean is much larger than its variance, the variance problem may not be large, and offers this as a possible explanation for why some models need warmup more than others.
The RAdam paper did check the zero-mean assumption itself with a simulation in Section 5.3. Gradients are sampled from with six values of (0, 0.001, 0.01, 0.1, 1 and 10) and , running 5,000 trajectories of 6,000 steps each; in all six cases, the variance of the raw adaptive learning rate is large early on and stays roughly constant after rectification. The same section also confirms that the first-order approximation in Eq. (5) and the closed form in Eq. (2) nearly coincide. What this simulation measures is still the variance of the adaptive learning rate itself; it doesn't touch the correlation between and that Ma and Yarats pointed out.
Ma and Yarats didn't fully settle the question either. Their simulation shows the update magnitudes settling in about 40 steps, and by that logic a few hundred steps of warmup should be enough; yet for both RAdam and their rules of thumb, the effective warmup is around 1,000 steps, and some hand-tuned warmups stretch to ten thousand. In their own discussion they write that why Adam needs such a long warmup is still an unresolved question.
From an engineering standpoint, both sides reach the same practical conclusion, that RAdam is equivalent to a warmup about steps long. This has a consequence that's easy to miss. Later large language models often set to 0.95, GPT-3 and LLaMA among them; then , and by the formula RAdam's built-in effective warmup is only about 24 steps, whereas LLaMA actually uses 2,000 warmup steps.
The rectification term itself has stuck around, though. PyTorch ships torch.optim.RAdam; the official Schedule-Free package also took in a community-contributed RAdam version in late 2024, whose description says it does away with warmup too.
Conclusion
RAdam starts from Adam's adaptive learning rate, uses an SMA approximation to estimate , the effective number of samples at step , and then uses the rectification term to bring the variance to a consistent level; for the first few steps, where , it degenerates straight into momentum SGD. On translation and image classification, it matches a tuned Adam with warmup without a separate warmup-length setting (though was specifically aligned in the translation experiments). Ma and Yarats then pointed out that is just a fixed warmup determined by , and that a simple linear warmup gets almost the same results.
If you already have a tuned warmup, switching to RAdam probably won't buy you much; if you don't, RAdam and "linear warmup over steps" are two roughly equally reliable starting points. Just remember that with RAdam, changing also changes the warmup length.
The README opens with "If warmup is the answer, what is the question?" Ma and Yarats's answer is that Adam's early updates are too large; as for why the warmup has to last so long, they admit they don't know either.