Back to paper library
arXiv 2019ICLR 2020OptimizersPaper notes

RAdam

Fitting a rectifier

Original paper

On the Variance of the Adaptive Learning Rate and Beyond

Paper authors

Liyuan Liu · Haoming Jiang · Pengcheng He …

Show all 7 authors

Liyuan Liu · Haoming Jiang · Pengcheng He · Weizhu Chen · Xiaodong Liu · Jianfeng Gao · Jiawei Han

Publication
ICLR 2020Conference
First submitted to arXiv
arXiv:1908.03265
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{optimizersradam2019,
  title = {{On the Variance of the Adaptive Learning Rate and Beyond}},
  author = {Liyuan Liu and Haoming Jiang and Pengcheng He and Weizhu Chen and Xiaodong Liu and Jianfeng Gao and Jiawei Han},
  year = {2019},
  eprint = {1908.03265},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1908.03265}
}
On this page

If warmup is the answer, what is the question?

That is how RAdam introduces itself on GitHub. By then, warmup was standard equipment for training Transformers. You raised the learning rate slowly from near 0 over the first few thousand steps, and only then would Adam train. There was no theoretical explanation at the time for why it had to be done this way, and how long to warm up was mostly settled by trial and error.

In August 2019, Liyuan Liu and colleagues from the University of Illinois and Microsoft released this paper. They traced the cause to Adam's adaptive learning rate. Early in training it has seen too few gradients, so its variance grows out of control. Along the way they proposed a fix called Rectified Adam, or RAdam for short.

About a week after the paper went up on arXiv, Less Wright introduced it on the fast.ai forums and on Medium as a "New State of the Art AI Optimizer," and people on the forum soon put it to the test. Two months after that, Jerry Ma and Denis Yarats published a rebuttal, arguing that RAdam, taken apart, is just Adam plus a fixed warmup.

This note looks at both sides.

Defining the problem

Adam's adaptive learning rate

The authors first write the various adaptive optimizers in a single framework (Algorithm 1). Each step computes a momentum mt=ϕ(g1,…,gt)m_t = \phi(g_1, \dots, g_t) and an adaptive learning rate lt=ψ(g1,…,gt)l_t = \psi(g_1, \dots, g_t), then updates θt=θt−1−αtmtlt\theta_t = \theta_{t-1} - \alpha_t m_t l_t. All operations are element-wise. For Adam, ψ\psi is:

ψ(g1,…,gt)=1−β2t(1−β2)∑i=1tβ2t−igi2\psi(g_1, \dots, g_t) = \sqrt{\frac{1 - \beta_2^t}{(1 - \beta_2)\sum_{i=1}^{t}\beta_2^{t-i} g_i^2}}

That is, the reciprocal of the square root of the second-moment estimate, with the 1−β2t1-\beta_2^t in the numerator serving as bias correction. Parameters whose gradients stay large take smaller steps; parameters whose gradients stay small take larger ones.

What happens without warmup

Figure 1 from the paper: training loss of a Transformer on the IWSLT'14 German-to-English dataset, with 0 to 70,000 iterations on the horizontal axis and 0 to 9 on the vertical axis. The blue Adam-vanilla line (no warmup) stays above 9 throughout; the black Adam-eps line falls to about 3.6; the Adam-2k, RAdam and Adam-warmup lines almost coincide, falling to about 3, and are labeled Overlapped on the plot.
Original Figure 1, Liu et al., from PDF page 1, arXiv license, scholarly quotation.

Look first at the blue Adam-vanilla line and at Adam-warmup at the bottom. With the same Transformer and the same IWSLT'14 German-to-English data, the training loss stalls around 10 without warmup and drops below 3 with it. The other three lines are the authors' controlled experiments, which we'll get to shortly.

The authors plotted the distribution of absolute gradient values at each step and found that without warmup, the distribution gets distorted within the first 10 steps, its mass crowding into very small values. Their reading is that Adam makes harmful updates in the first few steps that carry the model into a bad region, and warmup works by reducing the impact of these updates.

So why are the first few updates harmful?

Solving the problem

The first step has only one sample

Start with the most extreme case, t=1t = 1. Here ψ(g1)=1/g12=1/∣g1∣\psi(g_1) = \sqrt{1/g_1^2} = 1/|g_1|, so the entire adaptive learning rate is estimated from a single gradient.

If the gradient follows N(0,σ2)\mathcal{N}(0, \sigma^2), then 1/g121/g_1^2 follows a scaled inverse chi-squared distribution with 1 degree of freedom, and the variance of 1/g12\sqrt{1/g_1^2} diverges. The intuition is simple enough. If g1g_1 happens to land very close to 0, 1/∣g1∣1/|g_1| becomes enormous.

A small learning rate can hold this variance down, since Var[αx]=α2Var[x]\mathrm{Var}[\alpha x] = \alpha^2 \mathrm{Var}[x]. That is the authors' explanation of warmup. Early in training, the adaptive learning rate's variance is too large, and warmup suppresses it with a small α\alpha.

Two controlled experiments

If the problem really is the variance, then reducing the variance some other way should also fix it. The authors designed two Adam variants.

  • Adam-2k: for the first 2,000 iterations, only ψ\psi is updated, while momentum and parameters stay frozen. In effect, it looks at 2,000 extra samples to get an accurate second-moment estimate before training starts.
  • Adam-eps: raises the ϵ\epsilon in the denominator from 10−810^{-8} to 10−410^{-4}, directly capping ψ\psi.

In the figure above, Adam-2k has no warmup, yet it overlaps with Adam-warmup, and the gradient distribution is no longer distorted. Adam-eps also avoids getting stuck at 10, but ends up around 3.6, worse than warmup. The authors conjecture that a large ϵ\epsilon introduces bias into the adaptive learning rate and slows optimization down, so a more principled method is needed.

Variance falls as samples grow

Next comes computing the variance. Adam uses an exponential moving average (EMA), which is hard to analyze, so the authors approximate it with a simple moving average (SMA) and assume ψ2\psi^2 follows a scaled inverse chi-squared distribution with ρ\rho degrees of freedom. Here ρ\rho can be read as "how many samples the second-moment estimate effectively uses."

Theorem 1 says that under this assumption, the variance of ψ\psi decreases monotonically as ρ\rho grows, and gives a closed form for ρ>4\rho > 4 (Eq. (2)). In other words, the fewer the samples, the less reliable the adaptive learning rate; for ρ≤4\rho \le 4, the paper treats the variance as intractable.

How many samples is an EMA worth?

So how long an SMA is the EMA at step tt equivalent to? The authors require the two to share the same "center of mass," meaning the weighted average of the samples' positions in time must match, and solve for:

ρt=21−β2−1−2tβ2t1−β2t,ρ∞=21−β2−1\rho_t = \frac{2}{1-\beta_2} - 1 - \frac{2t\beta_2^t}{1-\beta_2^t}, \qquad \rho_\infty = \frac{2}{1-\beta_2} - 1

ρ∞\rho_\infty is the limit as t→∞t \to \infty. Plugging in the default β2=0.999\beta_2 = 0.999 gives ρ∞=1999\rho_\infty = 1999. For the first few steps, ρt\rho_t works out to:

tt12345
ρt\rho_t1.0001.9992.9993.9974.996

Computed from the paper's definition of ρt\rho_t, with β2=0.999\beta_2 = 0.999.

Early in training, ρt\rho_t is almost exactly tt, which matches intuition. However many gradients it has seen is how many samples it effectively uses. The closer β2\beta_2 is to 1, the longer the EMA's memory, and the longer ρt\rho_t takes to approach its limit.

The rectification term rtr_t

With ρt\rho_t in hand, the authors set the goal of keeping the adaptive learning rate's variance the same at every step, equal to its minimum CvarC_{var} at ρ∞\rho_\infty. They do this by multiplying by a scaling factor rt=Cvar/Var[ψ]r_t = \sqrt{C_{var}/\mathrm{Var}[\psi]}.

The closed form in Eq. (2) is numerically unstable, so the authors use a first-order approximation instead, giving Eq. (5):

Var[ψ]≈ρt2(ρt−2)(ρt−4)σ2\mathrm{Var}[\psi] \approx \frac{\rho_t}{2(\rho_t - 2)(\rho_t - 4)\sigma^2}

Plug in ρt\rho_t and ρ∞\rho_\infty in turn and divide, and σ\sigma cancels out:

rt=(ρt−4)(ρt−2)ρ∞(ρ∞−4)(ρ∞−2)ρtr_t = \sqrt{\frac{(\rho_t - 4)(\rho_t - 2)\rho_\infty}{(\rho_\infty - 4)(\rho_\infty - 2)\rho_t}}

This is the rectification in RAdam's name. Let's compute a few points to see what it looks like:

Step tt5101001,0002,0005,000
rtr_t0.0170.0490.2150.6450.8280.983

Computed from rtr_t in Section 4.2 of the paper, with β2=0.999\beta_2 = 0.999.

It starts out tiny, reaches about 0.65 at a thousand steps, and only nears 1 at five thousand. Multiply the learning rate by this curve, and it looks just like a warmup.

The full algorithm

Put together as Algorithm 2, each step does the following:

  1. Compute the gradient gtg_t, and update the second moment vt=β2vt−1+(1−β2)gt2v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2 and the first moment mt=β1mt−1+(1−β1)gtm_t = \beta_1 m_{t-1} + (1-\beta_1) g_t.
  2. Bias-correct the first moment, m^t=mt/(1−β1t)\hat m_t = m_t / (1 - \beta_1^t).
  3. Compute ρt\rho_t.
  4. If ρt>4\rho_t > 4, compute lt=(1−β2t)/vtl_t = \sqrt{(1-\beta_2^t)/v_t} and rtr_t, and update θt=θt−1−αtrtm^tlt\theta_t = \theta_{t-1} - \alpha_t r_t \hat m_t l_t.
  5. Otherwise, skip the adaptive learning rate and simply set θt=θt−1−αtm^t\theta_t = \theta_{t-1} - \alpha_t \hat m_t.

Item 5 is where RAdam degenerates into SGD with momentum. From the ρt\rho_t table above, with β2=0.999\beta_2 = 0.999 only t=1t = 1 through 4 have ρt≤4\rho_t \le 4, so the first 4 steps are bias-corrected momentum SGD, and the rectified Adam only kicks in from step 5. Ma and Yarats later proved that for any 0.8≤β2<10.8 \le \beta_2 < 1, ρt≤4\rho_t \le 4 is exactly equivalent to t≤4t \le 4. The paper also notes that if β2≤0.6\beta_2 \le 0.6, then ρ∞≤4\rho_\infty \le 4, and RAdam degenerates entirely into SGD with momentum.

Relation to warmup

In Section 4.3, the authors compare rtr_t directly with linear warmup. Linear warmup amounts to multiplying the learning rate by min⁡(t,Tw)/Tw\min(t, T_w)/T_w, which is similar in form to rtr_t, so they take this as verifying that "warmup works as a variance reduction technique." The difference is that RAdam needs no warmup-length hyperparameter TwT_w, and adjusts automatically with β2\beta_2.

They also stress that this fix is orthogonal to other techniques for stabilizing training, such as gradient clipping, a larger ϵ\epsilon, initialization and normalization layers, and can be used alongside them.

Discussion

Experimental setup

The experiments fall into three groups, and the Adam-family methods all use AdamW's decoupled form of weight decay.

  • Language modeling: One Billion Word, a two-layer LSTM with 2,048 hidden units, learning rate 0.001, batch size 128.
  • Image classification: ResNet-20 on CIFAR-10 (186 epochs) and ResNet-18 on ImageNet (90 epochs), with step-wise learning-rate decay and weight decay 10−410^{-4}.
  • Machine translation: fairseq's Transformer, on IWSLT'14 German–English in both directions and on WMT'16 English-to-German; Adam uses 4,000 steps of linear warmup.

In the language modeling and image experiments, Adam and RAdam both set (β1,β2)(\beta_1, \beta_2) to (0.9,0.999)(0.9, 0.999).

Against vanilla Adam

Without warmup, RAdam beats vanilla Adam on all three datasets. Test perplexity on One Billion Word drops from 36.92 to 35.70; RAdam is slower than Adam for the first few epochs, then converges faster. Test accuracy for image classification is as follows:

DatasetSGDAdamRAdam
CIFAR-1091.5190.5491.38
ImageNet69.8666.5467.62

Compiled from Table 1 of the paper; in %.

RAdam is better than Adam, but it doesn't beat SGD on either dataset. The authors also report ImageNet training accuracy, with SGD, Adam and RAdam at 69.57, 69.12 and 70.30, RAdam the highest. It trains better than SGD yet tests worse, so the gap appears to come from generalization. That said, this is an inference worked back from the training accuracy; the paper itself doesn't analyze it this way.

Insensitive to the learning rate

Figure 6 from the paper: training of ResNet-20 on CIFAR-10, with three columns for RAdam, Adam and SGD, test accuracy on the top row and training loss on the bottom row, epochs on the horizontal axis, and four colors for learning rates 0.1, 0.03, 0.01 and 0.003. In the RAdam column the four lines almost completely overlap, all ending near 91% accuracy; in the Adam column the pink line for learning rate 0.1 is clearly worse, at about 86%; in the SGD column the four lines spread out, and the smaller the learning rate, the worse.
Original Figure 6, Liu et al., from PDF page 7, arXiv license, scholarly quotation.

As the learning rate varies from 0.1 to 0.003, RAdam's four test-accuracy curves nearly lie on top of each other; Adam drops to about 86% at a learning rate of 0.1, and SGD's four lines each go their own way. The authors read this as rectification letting RAdam handle a wider range of learning rates. Still, the README's FAQ candidly says "the robustness of RAdam is not infinity," and the learning rate still needs tuning.

Against warmup

Translation is where warmup is hardest to do without, so here the opponent is a tuned Adam with warmup:

TaskAdam + warmupRAdam
IWSLT'14 De→En34.6634.76
IWSLT'14 En→De28.5628.48
WMT'16 En→De27.0327.27

Compiled from Table 2 of the paper; BLEU, with IWSLT averaged over 5 random seeds.

It's a tie, and the authors' own conclusion is that RAdam "achieves similar performance to that of previous state-of-the-art warmup heuristics." One setting here only shows up in the appendix. The translation experiments set β2\beta_2 to 0.9995, and the appendix gives the reason as making ν=4000\nu = 4000. Plug it into ρ∞=2/(1−β2)−1\rho_\infty = 2/(1-\beta_2) - 1 and you get exactly 3,999, lined up with the 4,000-step warmup of the Adam runs. So in this comparison, the length of RAdam's "automatic warmup" was aligned through β2\beta_2.

On CIFAR-10, Adam with warmup reaches a best accuracy of 91.29%, on par with RAdam's 91.38%; but the warmup length has options of 100, 200, 500 and 1,000 steps, and a learning rate of 0.1 with 100 warmup steps gets only 90.13%, against RAdam's 91.06% at the same learning rate. Stated more precisely, then, the authors' claim is that RAdam has one fewer hyperparameter to tune, while the best results themselves are about the same.

The momentum SGD steps at the start

Do the steps where it degenerates into momentum SGD matter? Appendix C runs a substitution experiment on CIFAR-10 at a learning rate of 0.1 (averaged over 5 runs). RAdam throughout gets a test accuracy of 91.08%; replacing the first 4 steps with plain Adam drops it to 89.98%; using SGD for the first 4 steps and unrectified Adam for steps 5 to 8 gives 90.29%. The authors see this as consistent with the theory, with earlier updates doing more damage, but also say this part is only a minor element of the algorithm's design; the focus is on the warmup mechanism and the derivation of the rectification term.

Ma and Yarats's rebuttal

In October 2019, Jerry Ma (University of Chicago) and Denis Yarats (New York University, Facebook AI Research) posted On the Adequacy of Untuned Warmup for Adaptive Optimization (AAAI 2021) on arXiv. The abstract gets straight to the point, stating that they "refute this analysis." The argument has three layers.

First, RAdam is a fixed warmup. rtr_t depends only on tt and β2\beta_2, and not at all on what the gradients actually look like. As mentioned above, ρt≤4\rho_t \le 4 covers exactly the first 4 steps, so RAdam can be summed up in one line as four steps of momentum SGD followed by Adam under a fixed warmup schedule. They also tried replacing the first 4 steps with doing nothing, or with Adam at a very small learning rate, and the differences were negligible.

Second, the variance argument has a hole. RAdam assumes the gradient has zero mean, but early in training the model is far from a minimum, and the gradient's mean is usually larger than its standard deviation; on EMNIST they measured that it takes until around step 50 for most parameters' coefficient of variation to stay above 1. More importantly, Adam's actual update is m^t/v^t\hat m_t / \sqrt{\hat v_t}, and ∣mt∣|m_t| and vt\sqrt{v_t} are highly correlated. At step 1 they are exactly equal, so every parameter's update is exactly ±α\pm\alpha (with ϵ=0\epsilon = 0). In other words, at the very step where the adaptive learning rate's variance "diverges," the variance of the actual update is 0.

Third, what matters is the size of the update. They ran a simulation in which the gradient around a minimum is pure zero-mean noise, and ran Adam to watch the update magnitudes. These start at 1⋅α1\cdot\alpha and only settle at around 0.15α0.15\alpha after about 40 steps. So even when the model is already sitting at a minimum, Adam takes oversized steps for its first few dozen iterations; SGD in the same situation needs no warmup, which partly explains why Adam needs warmup more than SGD does.

Building on this analysis, they propose two rules of thumb that need no tuning:

ωtexpo=1−exp⁡(−(1−β2) t),ωtlinear=min⁡(1,1−β22 t)\omega^{\text{expo}}_t = 1 - \exp\big(-(1-\beta_2)\,t\big), \qquad \omega^{\text{linear}}_t = \min\left(1, \frac{1-\beta_2}{2}\,t\right)

The first one's time constant 1/(1−β2)1/(1-\beta_2) was chosen to mimic Adam's second-moment bias-correction term 1−β2t≈1−e−(1−β2)t1-\beta_2^t \approx 1 - e^{-(1-\beta_2)t}, and the authors themselves say this intuition is purely speculative; the second is a linear warmup over 2/(1−β2)2/(1-\beta_2) steps, which comes to 2,000 steps for β2=0.999\beta_2 = 0.999.

A line chart drawn for this note, with β2 = 0.999, training steps 0 to 6,000 on the horizontal axis, and the factor applied to the learning rate, 0 to 1, on the vertical axis. The solid blue line is RAdam's rectification term r_t, which rises fastest at first, reaches about 0.65 at 1,000 steps, and then slowly approaches 1; the dashed orange line is a 2,000-step linear warmup, which reaches 1 at step 2,000; the dotted green line is the exponential warmup, similar in shape to r_t, slightly lower for the first 1,100 or so steps and slightly higher afterward. A note on the chart says that for t ≤ 4, ρ_t ≤ 4 and RAdam uses no adaptive learning rate.
Drawn for this note, computed from Algorithm 2 of the paper and Eqs. (13) and (14) of Ma & Yarats.

The three curves don't have quite the same shape. RAdam rises fastest over the first few hundred steps, but after about 1,500 steps it ends up last. Ma and Yarats compare them by the "effective warmup period" ∑t(1−ωt)\sum_t (1-\omega_t), that is, how many steps' worth of learning rate the warmup suppresses in total; at β2=0.999\beta_2 = 0.999 all three come to about 1,000 steps (by the formula, RAdam's is about 1,033), and they stay nearly identical across values of β2\beta_2.

The experiments back this up. On ResNet-50 on ImageNet, a Transformer language model on WikiText-103, and a Transformer on WMT16 English-to-German, they swept 9 to 12 combinations of learning rate and β2\beta_2 each, and the three methods mostly differ by only a few tenths. Take ImageNet top-1 error as an example (5 random seeds):

Learning rateβ2\beta_2ExponentialLinearRAdam
10−310^{-3}0.9927.928.028.4
10−310^{-3}0.99727.927.928.3
10−310^{-3}0.99928.228.328.4

Compiled from Table 1 of Ma & Yarats, showing the three rows with learning rate 10−310^{-3}; in %.

In these three rows, RAdam is actually slightly worse. The biggest gap in the whole table is at learning rate 10−210^{-2} and β2=0.999\beta_2 = 0.999, where RAdam's error is about 1 percentage point higher than the two rules of thumb. Their recommendation is simply to use a linear warmup over 2/(1−β2)2/(1-\beta_2) steps whenever you use Adam.

Who's right?

Both papers agree that Adam needs a smaller learning rate for its first few thousand steps; where they part ways is why.

On RAdam's side, the evidence is controlled experiments like Adam-2k, plus ties with tuned warmup on translation and images. Ma and Yarats, for their part, point out that rtr_t is just a fixed schedule determined by β2\beta_2, and use the update-magnitude analysis and experiments across three tasks to show that an untuned linear warmup does the same job. The final version of the RAdam paper doesn't respond to this rebuttal; the README's FAQ does concede that when the gradient's mean is much larger than its variance, the variance problem may not be large, and offers this as a possible explanation for why some models need warmup more than others.

The RAdam paper did check the zero-mean assumption itself with a simulation in Section 5.3. Gradients are sampled from N(μ,1)\mathcal{N}(\mu, 1) with six values of μ\mu (0, 0.001, 0.01, 0.1, 1 and 10) and β2=0.999\beta_2 = 0.999, running 5,000 trajectories of 6,000 steps each; in all six cases, the variance of the raw adaptive learning rate is large early on and stays roughly constant after rectification. The same section also confirms that the first-order approximation in Eq. (5) and the closed form in Eq. (2) nearly coincide. What this simulation measures is still the variance of the adaptive learning rate itself; it doesn't touch the correlation between mtm_t and vtv_t that Ma and Yarats pointed out.

Ma and Yarats didn't fully settle the question either. Their simulation shows the update magnitudes settling in about 40 steps, and by that logic a few hundred steps of warmup should be enough; yet for both RAdam and their rules of thumb, the effective warmup is around 1,000 steps, and some hand-tuned warmups stretch to ten thousand. In their own discussion they write that why Adam needs such a long warmup is still an unresolved question.

From an engineering standpoint, both sides reach the same practical conclusion, that RAdam is equivalent to a warmup about 2/(1−β2)2/(1-\beta_2) steps long. This has a consequence that's easy to miss. Later large language models often set β2\beta_2 to 0.95, GPT-3 and LLaMA among them; then ρ∞=39\rho_\infty = 39, and by the formula RAdam's built-in effective warmup is only about 24 steps, whereas LLaMA actually uses 2,000 warmup steps.

The rectification term itself has stuck around, though. PyTorch ships torch.optim.RAdam; the official Schedule-Free package also took in a community-contributed RAdam version in late 2024, whose description says it does away with warmup too.

Conclusion

RAdam starts from Adam's adaptive learning rate, uses an SMA approximation to estimate ρt\rho_t, the effective number of samples at step tt, and then uses the rectification term rtr_t to bring the variance to a consistent level; for the first few steps, where ρt≤4\rho_t \le 4, it degenerates straight into momentum SGD. On translation and image classification, it matches a tuned Adam with warmup without a separate warmup-length setting (though β2\beta_2 was specifically aligned in the translation experiments). Ma and Yarats then pointed out that rtr_t is just a fixed warmup determined by β2\beta_2, and that a simple linear warmup gets almost the same results.

If you already have a tuned warmup, switching to RAdam probably won't buy you much; if you don't, RAdam and "linear warmup over 2/(1−β2)2/(1-\beta_2) steps" are two roughly equally reliable starting points. Just remember that with RAdam, changing β2\beta_2 also changes the warmup length.

The README opens with "If warmup is the answer, what is the question?" Ma and Yarats's answer is that Adam's early updates are too large; as for why the warmup has to last so long, they admit they don't know either.