Back to paper library
arXiv 2024ICLR 2025OptimizersPaper notes

SOAP

Adam, rotated

Original paper

SOAP: Improving and Stabilizing Shampoo using Adam

Paper authors

Nikhil Vyas · Depen Morwani · Rosie Zhao …

Show all 8 authors

Nikhil Vyas · Depen Morwani · Rosie Zhao · Mujin Kwun · Itai Shapira · David Brandfonbrener · Lucas Janson · Sham Kakade

Publication
ICLR 2025Conference
First submitted to arXiv
arXiv:2409.11321
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{optimizerssoap2024,
  title = {{SOAP: Improving and Stabilizing Shampoo using Adam}},
  author = {Nikhil Vyas and Depen Morwani and Rosie Zhao and Mujin Kwun and Itai Shapira and David Brandfonbrener and Lucas Janson and Sham Kakade},
  year = {2024},
  eprint = {2409.11321},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.11321}
}
On this page

Everyone who uses Shampoo has to make the same decision. How often should the most expensive step run?

Shampoo updates its two statistic matrices LL and RR every step, but what actually multiplies the gradient is their matrix powers, which take an eigendecomposition or an iterative root solver. That step is so expensive that implementations compute it only every ff steps and reuse the old one in between.

Nikhil Vyas, Depen Morwani and colleagues at Harvard measured what happens as ff grows. On a 360M-parameter language model, Shampoo's final test loss is about 2.824 at f=1f=1, about 2.868 at f=10f=10 and about 2.932 at f=100f=100; AdamW in the same setup gets about 2.943. Stretch the interval to save compute, and Shampoo gives back almost its entire advantage over AdamW.

In August 2024, Shampoo had just won the external tuning track of AlgoPerf. This paper sets out to make it a bit cheaper, and a bit easier to tune.

Defining the problem

Shampoo keeps a left and a right preconditioner for an m×nm\times n weight matrix:

L←L+GG⊤,R←R+G⊤G,W←W−η L−1/4 G R−1/4L \leftarrow L + GG^\top,\qquad R \leftarrow R + G^\top G,\qquad W \leftarrow W - \eta\,L^{-1/4}\,G\,R^{-1/4}

The original exponent is −1/4-1/4; later implementations mostly use −1/2-1/2. Besides the learning rate and momentum, a Shampoo you can actually use (such as Meta's DistributedShampoo) needs a moving-average coefficient for the statistics, an ϵ\epsilon for the matrix roots, separate exponents for 1D and 2D parameters, a choice of which optimizer grafting takes its step size from, and how often to update the preconditioner.

There are two problems. First, performance drops as soon as the frequency is stretched, because between two eigendecompositions the whole preconditioner is frozen and knows nothing about how the gradient's scale changes. Second, there are too many hyperparameters.

The authors start by working out what Shampoo actually does in its eigenbasis. If it can be split into an expensive part and a cheap part, maybe the expensive part can be computed slowly while the cheap part is updated every step.

Solving the problem

Shampoo is a rotated Adafactor

First, idealize Shampoo. Use the 1/21/2 power; take LL and RR as expectations over the whole dataset, L=E[GG⊤]L=\mathbb E[GG^\top] and R=E[G⊤G]R=\mathbb E[G^\top G]; and multiply by a scalar correction based on Trace(L)\mathrm{Trace}(L). The update is then

W←W−η Tr(L)  L−1/2 G R−1/2W \leftarrow W - \eta\,\sqrt{\mathrm{Tr}(L)}\;L^{-1/2}\,G\,R^{-1/2}

Eigendecompose LL and RR as L=QLΛQL⊤L = Q_L\Lambda Q_L^\top and R=QRMQR⊤R=Q_R M Q_R^\top, with eigenvalues λ1,…,λm\lambda_1,\dots,\lambda_m and μ1,…,μn\mu_1,\dots,\mu_n. Substituting:

L−1/2 G R−1/2=QL  [Λ−1/2 (QL⊤G QR) M−1/2]  QR⊤L^{-1/2}\,G\,R^{-1/2} = Q_L\;\Big[\Lambda^{-1/2}\,\big(Q_L^\top G\,Q_R\big)\,M^{-1/2}\Big]\;Q_R^\top

Read this line in three steps. First rotate the gradient into the eigenbasis, G′=QL⊤GQRG' = Q_L^\top G Q_R; in that coordinate system, multiply entry (i,j)(i,j) by (λiμj)−1/2(\lambda_i\mu_j)^{-1/2}; finally rotate back to the original coordinates.

The middle step is an elementwise scaling, and the scaling factor is “a quantity for row ii” times “a quantity for column jj,” which is rank 1. That's exactly what Adafactor does. To save memory, Adafactor doesn't store a second moment for every entry; it stores only the sum AiA_i of each row and the sum CjC_j of each column, and reconstructs V^ij=AiCj/∑iAi\hat V_{ij} = A_iC_j/\sum_i A_i.

If you run Adafactor in the rotated coordinates, what is the row sum? Let uiu_i and vjv_j be the ii-th and jj-th column vectors of QLQ_L and QRQ_R. Then

Ai=E[∑j(ui⊤G vj)2]=E[∥ui⊤G∥2]=ui⊤ E[GG⊤] ui=λiA_i = \mathbb E\Big[\sum_j \big(u_i^\top G\,v_j\big)^2\Big] = \mathbb E\big[\|u_i^\top G\|^2\big] = u_i^\top\,\mathbb E[GG^\top]\,u_i = \lambda_i

The second equality uses the fact that the vjv_j form an orthonormal basis. Likewise Cj=μjC_j=\mu_j and ∑iAi=Tr(L)\sum_iA_i=\mathrm{Tr}(L). Adafactor's scaling for (i,j)(i,j) is (λiμj/Tr(L))−1/2(\lambda_i\mu_j/\mathrm{Tr}(L))^{-1/2}, identical to the Shampoo update above.

This is the paper's Claim 1, Shampoo with the 1/21/2 power is equivalent to running Adafactor in the eigenbasis of Shampoo's preconditioner. The expensive part is finding the basis (the eigendecomposition); the cheap part is the per-coordinate scaling within it.

From Adafactor to Adam

Seen this way, the root of Shampoo's frequency problem becomes visible. Recomputing only every ff steps freezes the basis and the scaling together, yet the scaling is just a few elementwise moving averages, cheap to update every step. The basis can change slowly; there's no reason for the scaling to wait along with it.

Since the scaling is going to be updated every step anyway, the authors simply swap Adafactor for Adam, using the full elementwise second moment with no rank-1 factorization. The algorithm (Algorithm 3) is below, where ⊙\odot is elementwise multiplication:

G′=QL⊤ G QRM←β1M+(1−β1) GM′=QL⊤ M QRV←β2V+(1−β2) G′⊙G′W←W−η  QL (M′V^+ϵ) QR⊤L←β2L+(1−β2) GG⊤,R←β2R+(1−β2) G⊤G\begin{aligned} G' &= Q_L^\top\,G\,Q_R \\ M &\leftarrow \beta_1 M + (1-\beta_1)\,G \\ M' &= Q_L^\top\,M\,Q_R \\ V &\leftarrow \beta_2 V + (1-\beta_2)\,G'\odot G' \\ W &\leftarrow W - \eta\; Q_L\,\Big(\frac{M'}{\sqrt{\hat V}+\epsilon}\Big)\,Q_R^\top \\ L &\leftarrow \beta_2 L + (1-\beta_2)\,GG^\top,\qquad R \leftarrow \beta_2 R + (1-\beta_2)\,G^\top G \end{aligned}

The basis is updated only every ff steps, using one step of power iteration followed by a QR decomposition, Q←QR(PQ)Q \leftarrow \mathrm{QR}(PQ), where PP is LL or RR and QQ is the previous basis. Only the very first basis comes from a full eigendecomposition.

A few details are worth pausing on.

The momentum MM is stored in the original coordinates and projected over every step. Since the basis changes every ff steps, momentum stored in the old coordinates would stop lining up the moment the basis changes. The second moment VV, by contrast, lives directly in the rotated coordinates and isn't rotated when the basis changes; the abstract describes this basis as “slowly changing.” When the basis changes, the official code reorders VV by eigenvalue size so that the same direction maps to the same slot.

One-dimensional parameters (biases, LayerNorm weights) just run AdamW. The first embedding layer and the final output layer have one dimension equal to the vocabulary size, and storing an eigenvector matrix that large is too expensive, so the QQ on that side is fixed to the identity. With the identity on both sides, SOAP falls back to Adam, which is why the paper describes SOAP as “running Adam in a rotated space.”

Fewer hyperparameters

The name SOAP stands for ShampoO with Adam in the Preconditioner's eigenbasis. Compared with AdamW, it adds just one hyperparameter, the frequency ff of basis updates. Compared with Shampoo, all of these go away:

  • The exponent of the matrix power, since the scaling is handled by Adam's V\sqrt{V};
  • The ϵ\epsilon added to the eigenvalues when taking roots, replaced by Adam's own ϵ\epsilon;
  • Grafting, since Adam sets the step size per coordinate in the rotated space and doesn't need to borrow one from another optimizer;
  • The moving-average coefficient for LL and RR, which simply shares Adam's β2\beta_2.

The cost

Memory is the same as DistributedShampoo's. An m×nm\times n layer stores four square matrices, LL, QLQ_L, RR and QRQ_R, plus the gradient, MM and VV, for a total of 2m2+2n2+3mn2m^2+2n^2+3mn; AdamW needs only 3mn3mn.

Each Shampoo step multiplies by a preconditioner once on each side; SOAP has to project over and then project back, an extra round of left and right matrix multiplications, so its per-step compute is slightly higher than Shampoo's. What that buys is the freedom to stretch the frequency.

Left half of original Figure 7: SOAP's training-time overhead relative to AdamW (vertical axis, log scale) against preconditioning frequency (horizontal axis, log scale). The overhead is about 27% at frequency 1, about 9% at 3, about 3.7% at 10, about 2.4% at 32 and about 2% at 100.
Original Figure 7 (left half), Vyas et al., from PDF page 13, arXiv license, scholarly quotation.

Changing the basis every step adds about 27% to training time; at f=10f=10 it's about 3.7%, and at f=100f=100 about 2%. The curve never reaches 0, because whether or not the basis changes, every step updates LL and RR in fp32 and does two projections. These numbers were measured on a single H100, with gradient accumulation used to reach the large batch.

Discussion

Experimental setup

All experiments are language-model pretraining. The data is C4 with the T5 tokenizer. The models are decoder-only, built on the OLMo codebase, in three sizes of 210M, 360M and 660M parameters (excluding embeddings), with widths of 1024, 1024 and 1408 and depths of 12, 24 and 24, using RoPE and QK LayerNorm in bf16 mixed precision. Sequences are 1024 tokens long, with about 2 million tokens per batch; training runs for about 20 times as many tokens as parameters, which is 3,200 steps for 360m and 6,400 steps for 660m.

All three optimizers first get a learning-rate sweep. AdamW additionally sweeps β1\beta_1 and β2\beta_2 at 360m; Shampoo, run as DistributedShampoo, additionally sweeps ϵ\epsilon, the moving-average coefficient of the statistics and four exponent settings, and each of these sweeps improves the loss by less than 0.004. SOAP sweeps only the learning rate. The shared settings are β1=0.95\beta_1=0.95, decoupled weight decay of 10−410^{-4}, z-loss of 10−410^{-4}, and warmup followed by cosine decay to 0.1 times the peak.

How “faster” is measured also needs spelling out. A cosine schedule depends on the total number of steps, so comparing intermediate checkpoints from the same run isn't fair. The authors run separate full SOAP trainings at 0.5, 0.625, 0.75 and 0.875 times the length, fit final loss against steps with a+bN−βa+bN^{-\beta}, and find the length at which it catches up with the final losses of AdamW and Shampoo.

Main results

Original Figure 1: the left and middle panels show training loss for the 660M-parameter model with a 2-million-token batch at frequency 10, against steps and against wall-clock time normalized to AdamW = 1, respectively; blue AdamW is highest, green Shampoo in the middle, orange SOAP lowest, and light orange is SOAP with a shortened schedule, with a dashed line marking where it catches up with AdamW's endpoint. The right panel shows final test loss of the 360M-parameter model against preconditioning frequency: green Shampoo rises with frequency from about 2.82 to 2.93, orange SOAP only to 2.87, and the blue dashed line at the top is AdamW.
Original Figure 1, Vyas et al., from PDF page 3, arXiv license, scholarly quotation.

Start with the left panel. The blue, green and orange lines are AdamW, Shampoo and SOAP, each run for 6,400 steps, and SOAP ends lowest. The light orange line is SOAP with a shortened schedule, and the dashed line shows it catching up with AdamW's endpoint at about 3,600 steps. The middle panel switches the horizontal axis to wall-clock time, and the gaps between the lines shrink a little, because every SOAP and Shampoo step costs more than an AdamW step.

Using the paper's efficiency estimates, the numbers for three settings are:

SettingTo match AdamWTo match Shampoo
360m, 2M0.50 / 0.520.75 / 0.74
660m, 2M0.565 / 0.620.80 / 0.78
360m, 256K0.72 / 0.830.875 / 0.90

Compiled from the tick labels in Figure 2 of the paper; each cell is “step ratio / time ratio,” and the batch sizes in the first column are in tokens. The time ratios against Shampoo are our conversion (Shampoo's full runs take 1.06, 1.12 and 1.11 times as long as AdamW's, respectively).

With large batches, SOAP needs only 50% to 57% of AdamW's steps and about 52% to 62% of its time, and saves a bit more than 20% relative to Shampoo. That's where the abstract's claim comes from, that SOAP “reduces the number of iterations by over 40% and wall clock time by over 35% compared to AdamW, with approximately 20% improvements in both metrics compared to Shampoo.” In a further run taking 360m to 100 times as many tokens as parameters (33.5 billion), SOAP keeps its advantage over AdamW.

Frequency ablation

The numbers at the start of this note come from the right half of Figure 1, shown again here.

Right half of original Figure 1: final test loss of the 360M-parameter model with a 2-million-token batch against preconditioning frequency (1, 3, 10, 32, 100). Green triangles are Shampoo, rising steadily from about 2.824 to 2.932; orange triangles are SOAP, from about 2.820 and still at 2.823 at f=10, rising to 2.870 at f=100; the blue dashed line at the top is AdamW, at about 2.943.
Original Figure 1 (right half), Vyas et al., from PDF page 3, arXiv license, scholarly quotation. Green line Shampoo, orange line SOAP, blue dashed line AdamW.

Both lines sit below AdamW's dashed line, so for every ff from 1 to 100, both beat AdamW. At f=1f=1 the two are nearly identical (2.820 vs 2.824), as expected when recomputing every step leaves Adafactor versus Adam as the only difference. After that Shampoo climbs all the way to 2.932, while SOAP is still at 2.823 at f=10f=10 and only rises to 2.870 at f=100f=100.

These results support the authors' explanation. SOAP is at 2.823 at f=10f=10, and at f=32f=32 it's only about 0.018 above f=1f=1, still far below Shampoo's 2.868 and 2.898 at the same frequencies. What really hurts Shampoo is that the scaling freezes along with the basis.

Batch size

Original Figure 4: the left panel shows the steps the 360M-parameter model needs to reach a loss of 2.842 against batch size, both axes on a log scale, with blue AdamW and orange SOAP, solid lines for actual and dashed lines for ideal linear scaling; the right panel shows training loss with a 256K-token batch at frequency 80, with blue AdamW, green Shampoo and orange SOAP.
Original Figure 4, Vyas et al., from PDF page 10, arXiv license, scholarly quotation.

Ideally, doubling the batch size halves the steps needed to reach the same loss; that's the dashed line on the left. The farther a solid line drifts from its dashed line, the more of the extra batch is wasted. AdamW's blue solid line drifts further and further away from 512 onward, and at 2048 it needs nearly twice the ideal number of steps; SOAP's orange solid line stays closer to its own dashed line, which is why the authors say SOAP has a larger critical batch size. For fairness, every time the batch shrinks by a factor of kk, the frequency grows by kk, keeping the eigendecomposition's share of the overhead constant.

The right panel is the smallest batch, 256K. SOAP needs 25% fewer steps than AdamW and about 10% fewer than Shampoo. Compared with the table above, the advantage shrinks noticeably at small batch sizes. The authors also call this a preliminary result, and holding “batch times frequency” fixed may not be the best trade-off. Second-order methods gaining less at small batch sizes is consistent with earlier work, and the official README also recommends using as large a batch as you can.

Memory-saving variants and GaLore

SOAP uses 2m2+2n22m^2+2n^2 more memory than AdamW. The paper tries two ways to cut it. Running Adafactor instead of Adam in the rotated space saves one mnmn copy with almost no change in loss. Rotating only the shorter side and using the identity on the other increases the loss somewhat more, but it still ties or slightly beats Shampoo. With both tricks combined, memory is 2min⁡(m,n)2+2mn2\min(m,n)^2+2mn, which on a standard Transformer is already less than AdamW's 3mn3mn, and it still beats AdamW.

The full-rank version of GaLore is very similar to SOAP, differing mainly in three places. GaLore's projection comes from the SVD of the current gradient, while SOAP uses a moving average of GG⊤GG^\top; GaLore stores momentum in the projected space and doesn't rotate it back when the basis changes, while SOAP stores it in the original space; and GaLore projects only one side. On the 210M-parameter model, GaLore's best result is 3.12, beating AdamW but losing to Shampoo's 3.10, so the authors consider each of SOAP's changes necessary.

The idea had actually been written down before. In Appendix B of their 2020 Distributed Shampoo paper, Anil et al. describe an algorithm essentially equivalent to SOAP on 2D layers, with no experiments attached, only a remark that unpublished experiments showed no improvement. SOAP's authors acknowledge this candidly in their related work and suggest the difference comes down to implementation details.

Limitations

The paper lists its own limitations clearly. The largest model has 660M parameters, two orders of magnitude smaller than the large models of the time; the authors assume the conclusions carry over to larger scales, but that “remains to be validated.” Only language models were tested. Throughput was measured on a single GPU, with comparisons of distributed implementations left for future work. All LL and RR operations run in fp32, and PyTorch's QR doesn't support lower precision, so the floor on the overhead can't be pushed down for now.

An independent check a year later

In 2025, Wen, Hall, Ma and Liang at Stanford published “Fantastic Pretraining Optimizers and Where to Find Them,” putting AdamW and ten newer optimizers through models from 130M to 1.2B parameters at 1 to 8 times the Chinchilla data budget, and comparing them after tuning each one carefully.

On 300M- and 520M-parameter models, they measured SOAP at 1.2 to 1.3 times faster than a well-tuned AdamW; at 1.2B parameters, SOAP and Muon were both down to about 1.1 times. They also found that once the data-to-model ratio reaches 8 times or more, SOAP and Kron overtake Muon, which is the strongest when data is scarce. Their SOAP implementation additionally splits weights into blocks and keeps momentum in bf16 at 1.2B parameters, so it differs slightly from the original.

Conclusion

SOAP starts from the equivalence that Shampoo with the 1/21/2 power is Adafactor running in Shampoo's own eigenbasis. Once the expensive basis is separated from the cheap scaling, the basis can be updated every 10 steps or even less often, while the scaling is updated every step the way Adam does it; the hyperparameters shrink to just one more than AdamW, the frequency. On 360M- and 660M-parameter language models with large batches, it needs about half the steps of AdamW and 38% to 48% less wall-clock time.

To try it, you can copy the official soap.py straight into your project and start from the README settings, β=(0.95,0.95)\beta=(0.95, 0.95) and frequency 10. First measure how much slower each step is than AdamW, then decide whether to stretch the frequency. When memory is tight, one-sided rotation and the Adafactor variant are fallbacks the paper has already tested.

The Fantastic paper singles SOAP out in its main text as one of the few papers that independently swept hyperparameters for its baseline, and its reported speedup came closest to what they measured. In optimizer papers, praise like that is rare.