Back to paper library
arXiv 2018ICML 2018OptimizersPaper notes

Adafactor

Decluttering memory

Original paper

Adafactor: Adaptive Learning Rates with Sublinear Memory Cost

Paper authors

Noam Shazeer · Mitchell Stern

Publication
ICML 2018Conference
First submitted to arXiv
arXiv:1804.04235
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{optimizersadafactor2018,
  title = {{Adafactor: Adaptive Learning Rates with Sublinear Memory Cost}},
  author = {Noam Shazeer and Mitchell Stern},
  year = {2018},
  eprint = {1804.04235},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1804.04235}
}
On this page

Open the T5 paper to its training setup, and the optimizer line reads "For optimization, we use AdaFactor." It was later used for fine-tuning FLAN and Flan-T5, too, and for pre-training the 540-billion-parameter PaLM.

Go back to the source, and the Adafactor paper has just one experimental task, an English-to-German Transformer, with each training run taking under two hours on a single TPU v2. The problem Noam Shazeer and Mitchell Stern set out to solve is stated on the first page. Adam uses too much memory.

Defining the problem

Adam stores two extra numbers for every parameter, the first moment mm and the second moment vv. The paper says this triples the memory requirement, with one copy each for the weights, mm and vv. Take a 1024×40961024 \times 4096 feed-forward weight matrix. It has 4,194,304 parameters, so Adam has to store another 8,388,608 numbers.

In 2018, the authors observed that compute was growing much faster than memory capacity, and memory had become one of the limits on model size. Counting only weights and optimizer state, Adam's two extra copies mean the same memory can hold a model only a third as large.

The authors set out to do four things:

  1. shrink the second moment from nmnm numbers to n+mn+m;
  2. drop the first moment, β1=0\beta_1 = 0;
  3. deal with the instability that shows up once momentum is gone;
  4. make the step size proportional to the scale of the parameters.

The first two save memory, the third is the trouble those savings create, and the fourth is an improvement the authors threw in along the way.

Solving the problem

Keep only row sums and column sums

Suppose a layer's weights form an n×mn \times m matrix. Adam maintains an exponential average of the squared gradients, VV, of the same size. If we store only two small matrices R∈Rn×kR \in \mathbb{R}^{n\times k} and S∈Rk×mS \in \mathbb{R}^{k\times m} with V≈RSV \approx RS, the memory shrinks a lot. The question is how to factor it.

The most common low-rank approximation is truncated SVD, which is optimal under the Frobenius norm, but it has two problems. First, the top kk singular vectors don't decompose over addition. The singular vectors of a sum of two matrices bear no simple relation to those of each matrix, so it can't be combined with an exponential average. Second, the approximated entries can be negative, while Adam needs to take the square root of VV and then its reciprocal.

The authors turn instead to the generalized KL divergence common in nonnegative matrix factorization, also called I-divergence:

d(p,q)=plog⁡pq−p+qd(p, q) = p \log \frac{p}{q} - p + q

It is never less than 0, and equals 0 only when p=qp = q. The goal is to minimize ∑i,jd(Vij,[RS]ij)\sum_{i,j} d(V_{ij}, [RS]_{ij}) subject to RR and SS both being nonnegative. General rank kk requires alternating optimization; rank 1 has a closed-form solution (Lemma 1 in the paper):

V^=(V1m)(1n⊤V)1n⊤V1m\hat V = \frac{(V 1_m)(1_n^\top V)}{1_n^\top V 1_m}

V1mV 1_m is the sum of each row, 1n⊤V1_n^\top V is the sum of each column, and the denominator is the sum of all entries. In plain terms, the estimate at row ii, column jj is "sum of row ii × sum of column jj ÷ total sum."

Let's work through a 2×32 \times 3 example:

V=[936312]⇒row sums=(18, 6),    column sums=(12, 4, 8),    total=24V = \begin{bmatrix} 9 & 3 & 6 \\ 3 & 1 & 2 \end{bmatrix} \quad\Rightarrow\quad \text{row sums}=(18,\,6),\;\; \text{column sums}=(12,\,4,\,8),\;\; \text{total}=24

The top-left entry is 18×12/24=918 \times 12 / 24 = 9 and the bottom-right is 6×8/24=26 \times 8 / 24 = 2, so the whole matrix comes back intact, because the second row happens to be a third of the first, which makes VV itself rank 1. Change the bottom-right 2 to 8, and the row sums become (18,12)(18, 12), the column sums (12,4,14)(12, 4, 14) and the total 30, so the bottom-right estimate is only 12×14/30=5.612 \times 14 / 30 = 5.6. The factorization remembers the overall size of each row and each column, but not that one particular cell stands out.

This solution uses only the row sums and column sums, both linear functions of VV, so "average, then sum" equals "sum, then average." In practice, you just keep exponential averages of the row sums and column sums directly, without ever reconstructing the full VV. For that 1024×40961024 \times 4096 weight matrix, the second moment goes from 4,194,304 numbers to 5,120.

The authors write the β1=0\beta_1 = 0 version as Algorithm 2:

Rt=β2Rt−1+(1−β2) (Gt2) 1mCt=β2Ct−1+(1−β2) 1n⊤(Gt2)V^t=RtCt/1n⊤Rt1−β2t\begin{aligned} R_t &= \beta_2 R_{t-1} + (1-\beta_2)\,(G_t^2)\,1_m \\ C_t &= \beta_2 C_{t-1} + (1-\beta_2)\,1_n^\top (G_t^2) \\ \hat V_t &= \frac{R_t C_t / 1_n^\top R_t}{1 - \beta_2^t} \end{aligned}

and then update with Gt/(V^t+ϵ)G_t / (\sqrt{\hat V_t} + \epsilon), as in Adam. Biases and other vector-shaped parameters keep the original element-wise VV, since they're small to begin with.

Drop the momentum, and things break

The second step is simpler. Set β1=0\beta_1 = 0, and don't store mm at all.

All the experiments use a Transformer on WMT'14 English-to-German, for 100,000 steps, with each batch holding about 4,096 source tokens plus 4,096 target tokens, roughly an eighth of the original Transformer paper's batch. Each setting is run twice, once with warmup, at learning rate 0.1⋅min⁡(10−6t, 1/t)0.1 \cdot \min(10^{-6} t,\, 1/\sqrt t), and once without, starting directly at 0.1⋅min⁡(10−2, 1/t)0.1 \cdot \min(10^{-2},\, 1/\sqrt t). The metric is BLEU on the newstest2013 dev set.

SettingWith warmupNo warmup
(A) Adam, β1=0\beta_1=025.60.1
(B) Adam, β1=0.9\beta_1=0.925.423.1
(C) Factored second moment, β1=0\beta_1=025.40.2
(D) Row means only25.20.3
(E) Column means only0.30.5

Compiled from Table 2 of the paper; all settings use β2=0.999\beta_2 = 0.999.

With warmup, dropping momentum (A versus B) and then factoring the second moment (C) change BLEU by only a few tenths. (D) and (E) are two even leaner controls that store only the row means or only the column means. Row means do about as well as the baseline, while column means break down entirely. The authors' explanation is that this model's word embeddings and output softmax share one weight matrix, with each row corresponding to a token; rows for frequent tokens get far larger gradients than rows for rare ones, and storing only column means lumps every token onto the same scale.

The trouble is in the right-hand column. Without warmup, (B), with momentum, still gets 23.1, while (A), without momentum, drops straight to 0.1.

Stale second moments

The authors suspect β2\beta_2. A small β2\beta_2 means the second moment decays fast, which Reddi et al. had already shown causes convergence problems; a large β2\beta_2 means the estimate comes from gradients long past, and it can't keep up when the model changes quickly.

Both show up. At β2=0.999\beta_2 = 0.999 it's 25.6 with warmup and 0.1 without; at β2=0.9\beta_2 = 0.9 the two are 18.4 and 15.6, stable but not learning well.

To check whether the estimate is lagging, the authors define a quantity:

RMS(Ut)=Mean⁡x∈X ⁣(gxt2v^xt)\mathrm{RMS}(U_t) = \sqrt{\operatorname{Mean}_{x \in X}\!\left(\frac{g_{xt}^2}{\hat v_{xt}}\right)}

Ut=−gt/v^tU_t = -g_t / \sqrt{\hat v_t} is the update before it's multiplied by the learning rate. If v^\hat v is estimated accurately, each parameter's g2/v^g^2 / \hat v should be around 1, and the RMS over the whole matrix should be close to 1 as well.

Figure 1 from the paper: RMS(U_t) for one Transformer weight matrix over the first 3,000 steps. The red line for β2=0.9 hovers around 1 throughout; the blue line for β2=0.999 drops to about 0.33 in the first 300 steps, spikes to about 2.24 around step 800, then slowly falls back, returning to around 1 after 2,000 steps.
Original Figure 1, Shazeer and Stern, from PDF page 5, arXiv license, scholarly quotation.

The red line is β2=0.9\beta_2 = 0.9, staying around 1 throughout. The blue line is β2=0.999\beta_2 = 0.999. It first drops to about 0.33 over the first 300 steps, meaning updates smaller than expected; then around step 800 it spikes to 2.24, with updates more than twice as large as expected. Both runs had warmup and neither diverged, so this isn't a case of training blowing up and then producing larger updates; the authors think the oversized updates cause the instability, and admit that this figure is only supporting evidence, not proof.

Update clipping

If the problem is updates that are too large, then limit the update directly:

U^t=Utmax⁡(1,  RMS(Ut)/d)\hat U_t = \frac{U_t}{\max\left(1,\; \mathrm{RMS}(U_t) / d\right)}

When the RMS exceeds the threshold dd, the whole matrix's update is scaled down proportionally to dd. This differs from the usual gradient clipping. Gradient clipping limits the gradient, but methods like Adam then divide each parameter by v^\sqrt{\hat v}, so once v^\hat v goes stale, a clipped gradient can still be blown up into a huge update; update clipping limits the step that actually gets applied.

With d=1d = 1, the no-warmup 0.1 is rescued to 21.5; at d=2d = 2 the threshold is too loose, and it's still only 0.2. With warmup, clipping makes no clear difference either way.

Let β2\beta_2 rise with the step count

Another route is to make β2\beta_2 small at first and large later. The authors first point out that Adam's bias correction is itself equivalent to an increasing β2\beta_2:

β^2t=β2 1−β2t−11−β2t\hat\beta_{2t} = \beta_2 \,\frac{1 - \beta_2^{t-1}}{1 - \beta_2^{t}}

It is 0 at t=1t = 1, then gradually approaches β2\beta_2. The authors propose a different schedule:

β^2t=1−t−c\hat\beta_{2t} = 1 - t^{-c}

With the c=0.8c = 0.8 the paper adopts, it's 0.84 at step 10, 0.975 at step 100 and 0.996 at step 1,000, passes 0.999 around step 5,600, and is 0.9994 at step 10,000. Early in training it behaves like fast decay, which is stable; later it behaves like slow decay, which gives more accurate estimates.

This schedule has two more benefits. β^21=0\hat\beta_{21} = 0, so the weights on all past gradients sum to exactly 1, and no bias correction is needed. Also, using the convergence condition for infinite products, the authors show that only c≤1c \le 1 lets the weight on every old gradient eventually go to 0; with c>1c > 1, β2\beta_2 rises too fast, and early gradients keep a sizable share forever. c=1c = 1 is exactly the arithmetic mean of all squared gradients.

SettingWith warmupNo warmup
(G) β2=0.9\beta_2 = 0.918.415.6
(H) β2=0.999\beta_2 = 0.999, clipping d=1d=125.421.5
(I) β2=0.999\beta_2 = 0.999, clipping d=2d=225.70.2
(J) Factored, β2=0.999\beta_2 = 0.999, clipping d=1d=125.622.4
(K) β^2t=1−t−0.5\hat\beta_{2t} = 1-t^{-0.5}25.621.1
(L) β^2t=1−t−0.8\hat\beta_{2t} = 1-t^{-0.8}25.60.1
(N) β^2t=1−t−0.8\hat\beta_{2t} = 1-t^{-0.8}, clipping d=1d=125.922.4

Compiled from Table 2 of the paper; all with β1=0\beta_1 = 0.

c=0.5c = 0.5 on its own holds up without warmup; c=0.8c = 0.8 on its own isn't enough and needs clipping too. (N) gets the table's highest with-warmup score, 25.9, and (J) shows that a factored second moment plus clipping can be about as stable as Adam with momentum.

Relative step size

The last piece has nothing to do with memory. Adam's learning rate is an absolute step size; the authors think the relative change in the parameters matters more, so they multiply the step size by the parameters' own scale:

αt=max⁡(ϵ2,  RMS(Xt−1)) ρt,ρt=min⁡ ⁣(10−2,  1t)\alpha_t = \max\bigl(\epsilon_2,\; \mathrm{RMS}(X_{t-1})\bigr)\,\rho_t, \qquad \rho_t = \min\!\left(10^{-2},\; \frac{1}{\sqrt t}\right)

RMS(Xt−1)\mathrm{RMS}(X_{t-1}) is the current root mean square of this parameter matrix, and ϵ2=10−3\epsilon_2 = 10^{-3} is a floor, so that parameters initialized to 0 can still leave the origin. The paper's acknowledgments specifically credit Geoffrey Hinton's view that training goes smoothly when each update is about 10−210^{-2} to 10−310^{-3} times the size of the parameters.

The authors tested three word-embedding initializations (50,000 steps, batches of 16,384 tokens, embeddings not shared with the softmax). The original Transformer initializes with standard deviation 1/dmodel1/\sqrt{d_{model}} and multiplies by dmodel\sqrt{d_{model}} at compute time, so the same absolute learning rate suits both the embeddings and the other weights; Adam gets 26.4 and Adafactor 26.6. With the more naive initializations, Adam drops to 25.8 and 24.2, while Adafactor still gets 26.4 and 25.4. When the parameter scale is poorly set, relative step sizes suffer less.

Putting it together

Algorithm 4 from the paper: Adafactor's full procedure for weight matrices, computing in order the relative step size α_t, the gradient G_t, exponential averages of the row sums R_t and column sums C_t, the factored second-moment estimate, the unscaled update U_t and update clipping, and finally updating the parameters.
Original Algorithm 4, Shazeer and Stern, from PDF page 8, arXiv license, scholarly quotation.

Chain the five pieces together and you get Algorithm 4. Line 3 is the relative step size, lines 5 and 6 are the row-sum and column-sum averages with ϵ1\epsilon_1 added, line 7 reconstructs the second moment, and line 9 is update clipping. The paper's recommended hyperparameters are ϵ1=10−30\epsilon_1 = 10^{-30}, ϵ2=10−3\epsilon_2 = 10^{-3}, d=1d = 1, ρt=min⁡(10−2,1/t)\rho_t = \min(10^{-2}, 1/\sqrt t) and β^2t=1−t−0.8\hat\beta_{2t} = 1 - t^{-0.8}, with no momentum. For an n×mn \times m matrix, the whole optimizer stores only n+mn + m extra numbers.

Discussion

The full version's scores

SettingWith warmupNo warmup
(A) Adam, β1=0\beta_1=025.60.1
(B) Adam, β1=0.9\beta_1=0.925.423.1
(O) Adafactor25.025.5
(P) Adafactor with momentum added back24.925.3
(Q) SGD, best learning rate24.39.1

Compiled from Table 2 of the paper; (O) and (P) use the relative step size ρt\rho_t, and (Q) uses lr=200⋅stlr = 200 \cdot s_t with warmup and lr=10⋅stlr = 10 \cdot s_t without; every larger learning rate diverged without warmup.

The full Adafactor gets the table's highest no-warmup score, 25.5, and 25.0 with warmup, 0.6 below (A) without momentum and 0.4 below (B) with momentum. The authors themselves note that the relative step sizes can't be compared directly with the step sizes in the other rows, so they can't tell whether the stability comes from the relative step size. SGD also uses no extra memory, but its best is only 24.3, and it's very sensitive to the learning rate.

These results come from just one translation task, one model, 100,000 steps, and a batch eight times smaller than the original setup. The paper ends by saying the saved memory "should enable training of significantly larger models," but it has no large-model experiments at all, and no measurements of memory or speed.

Adafactor after the paper

The real large-scale tests were done by others.

T5 adopted Adafactor directly. Its pre-training learning rate is held at 0.01 for the first ten thousand steps and then decays as the inverse square root, which is exactly this paper's ρt=min⁡(10−2,1/t)\rho_t = \min(10^{-2}, 1/\sqrt t). T5's published training config also keeps the paper's β1=0\beta_1 = 0, an update clipping threshold of 1.0, the factored second moment and parameter scaling.

DeepMind tried another direction while training Gopher. At 1.4 billion parameters, Adafactor was stable and worked well, but at 7.1 billion it began to show mild loss divergence; Adam's maximum learning rate was 1.2×10−41.2\times10^{-4}, and Adafactor was still unstable even at 6×10−56\times10^{-5}. They still ended up fine-tuning Gopher with Adafactor, because its hardware requirements are much lower.

When Google trained the 2-billion-parameter ViT-G, they said Adafactor "did not work out of the box" and changed three things. They added back a half-precision first moment, disabled step-size scaling relative to the weight norm, and capped β2\beta_2 at 0.999. After the changes, it performed on par with or slightly better than Adam.

PaLM was more direct. The paper states that it uses Adafactor "without factorization," which is effectively Adam with parameter scaling; momentum is β1=0.9\beta_1 = 0.9, the second moment uses β2=1−k−0.8\beta_2 = 1 - k^{-0.8}, and the learning rate is 10−210^{-2} for the first ten thousand steps, then decays as 1/k1/\sqrt k. They specifically explain that this β2\beta_2 schedule is more stable than a fixed 0.99, because embeddings of rare tokens can have poorly estimated second moments over short windows.

Update clipping also got carried off on its own. In 2023, Wortsman et al. found that loss spikes in large CLIP training always came 1 to 8 steps after the second moment underestimated the squared gradient, and proposed StableAdamW, which "is AdamW with the update clipping technique introduced in AdaFactor." They also wrote that large-scale training has largely moved away from Adafactor in its original form, speculating that the reason is the factored second moment or the lack of momentum.

On the library side, Hugging Face Transformers has had Adafactor for a long time, while PyTorch only added torch.optim.Adafactor in version 2.5. Both use relative step sizes by default (Hugging Face lets you turn this off); PyTorch's version treats lr as the upper bound on ρt\rho_t, and notes that it deliberately departs from the paper in where ϵ1\epsilon_1 goes.

Conclusion

Adafactor uses a rank-1 factorization by row and column sums to shrink the second moment from nmnm to n+mn+m, then drops momentum. The authors traced the resulting instability to stale second moments, and handled it with update clipping and a 1−t−0.81-t^{-0.8} schedule for β2\beta_2. All this held on one translation task, and later held up at T5's scale too; at billions of parameters, Gopher reported it unstable, and PaLM and ViT-G put momentum back.

If you use Adafactor in Hugging Face and want to control the learning rate yourself, remember to set relative_step=False and scale_parameter=False. Otherwise the optimizer will switch to its built-in ρt\rho_t, or multiply your learning rate by the parameters' RMS, and the learning-rate schedule you wrote won't take effect as written.

The Adafactor that PaLM used ended up dropping even the factor.