SOAP: Improving and Stabilizing Shampoo using Adam
- Publication
- ICLR 2025Conference
- First submitted to arXiv
- arXiv:2409.11321
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{optimizerssoap2024,
title = {{SOAP: Improving and Stabilizing Shampoo using Adam}},
author = {Nikhil Vyas and Depen Morwani and Rosie Zhao and Mujin Kwun and Itai Shapira and David Brandfonbrener and Lucas Janson and Sham Kakade},
year = {2024},
eprint = {2409.11321},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2409.11321}
}
Everyone who uses Shampoo has to make the same decision. How often should the most expensive step run?
Shampoo updates its two statistic matrices and every step, but what actually multiplies the gradient is their matrix powers, which take an eigendecomposition or an iterative root solver. That step is so expensive that implementations compute it only every steps and reuse the old one in between.
Nikhil Vyas, Depen Morwani and colleagues at Harvard measured what happens as grows. On a 360M-parameter language model, Shampoo's final test loss is about 2.824 at , about 2.868 at and about 2.932 at ; AdamW in the same setup gets about 2.943. Stretch the interval to save compute, and Shampoo gives back almost its entire advantage over AdamW.
In August 2024, Shampoo had just won the external tuning track of AlgoPerf. This paper sets out to make it a bit cheaper, and a bit easier to tune.
Defining the problem
Shampoo keeps a left and a right preconditioner for an weight matrix:
The original exponent is ; later implementations mostly use . Besides the learning rate and momentum, a Shampoo you can actually use (such as Meta's DistributedShampoo) needs a moving-average coefficient for the statistics, an for the matrix roots, separate exponents for 1D and 2D parameters, a choice of which optimizer grafting takes its step size from, and how often to update the preconditioner.
There are two problems. First, performance drops as soon as the frequency is stretched, because between two eigendecompositions the whole preconditioner is frozen and knows nothing about how the gradient's scale changes. Second, there are too many hyperparameters.
The authors start by working out what Shampoo actually does in its eigenbasis. If it can be split into an expensive part and a cheap part, maybe the expensive part can be computed slowly while the cheap part is updated every step.
Solving the problem
Shampoo is a rotated Adafactor
First, idealize Shampoo. Use the power; take and as expectations over the whole dataset, and ; and multiply by a scalar correction based on . The update is then
Eigendecompose and as and , with eigenvalues and . Substituting:
Read this line in three steps. First rotate the gradient into the eigenbasis, ; in that coordinate system, multiply entry by ; finally rotate back to the original coordinates.
The middle step is an elementwise scaling, and the scaling factor is “a quantity for row ” times “a quantity for column ,” which is rank 1. That's exactly what Adafactor does. To save memory, Adafactor doesn't store a second moment for every entry; it stores only the sum of each row and the sum of each column, and reconstructs .
If you run Adafactor in the rotated coordinates, what is the row sum? Let and be the -th and -th column vectors of and . Then
The second equality uses the fact that the form an orthonormal basis. Likewise and . Adafactor's scaling for is , identical to the Shampoo update above.
This is the paper's Claim 1, Shampoo with the power is equivalent to running Adafactor in the eigenbasis of Shampoo's preconditioner. The expensive part is finding the basis (the eigendecomposition); the cheap part is the per-coordinate scaling within it.
From Adafactor to Adam
Seen this way, the root of Shampoo's frequency problem becomes visible. Recomputing only every steps freezes the basis and the scaling together, yet the scaling is just a few elementwise moving averages, cheap to update every step. The basis can change slowly; there's no reason for the scaling to wait along with it.
Since the scaling is going to be updated every step anyway, the authors simply swap Adafactor for Adam, using the full elementwise second moment with no rank-1 factorization. The algorithm (Algorithm 3) is below, where is elementwise multiplication:
The basis is updated only every steps, using one step of power iteration followed by a QR decomposition, , where is or and is the previous basis. Only the very first basis comes from a full eigendecomposition.
A few details are worth pausing on.
The momentum is stored in the original coordinates and projected over every step. Since the basis changes every steps, momentum stored in the old coordinates would stop lining up the moment the basis changes. The second moment , by contrast, lives directly in the rotated coordinates and isn't rotated when the basis changes; the abstract describes this basis as “slowly changing.” When the basis changes, the official code reorders by eigenvalue size so that the same direction maps to the same slot.
One-dimensional parameters (biases, LayerNorm weights) just run AdamW. The first embedding layer and the final output layer have one dimension equal to the vocabulary size, and storing an eigenvector matrix that large is too expensive, so the on that side is fixed to the identity. With the identity on both sides, SOAP falls back to Adam, which is why the paper describes SOAP as “running Adam in a rotated space.”
Fewer hyperparameters
The name SOAP stands for ShampoO with Adam in the Preconditioner's eigenbasis. Compared with AdamW, it adds just one hyperparameter, the frequency of basis updates. Compared with Shampoo, all of these go away:
- The exponent of the matrix power, since the scaling is handled by Adam's ;
- The added to the eigenvalues when taking roots, replaced by Adam's own ;
- Grafting, since Adam sets the step size per coordinate in the rotated space and doesn't need to borrow one from another optimizer;
- The moving-average coefficient for and , which simply shares Adam's .
The cost
Memory is the same as DistributedShampoo's. An layer stores four square matrices, , , and , plus the gradient, and , for a total of ; AdamW needs only .
Each Shampoo step multiplies by a preconditioner once on each side; SOAP has to project over and then project back, an extra round of left and right matrix multiplications, so its per-step compute is slightly higher than Shampoo's. What that buys is the freedom to stretch the frequency.

Changing the basis every step adds about 27% to training time; at it's about 3.7%, and at about 2%. The curve never reaches 0, because whether or not the basis changes, every step updates and in fp32 and does two projections. These numbers were measured on a single H100, with gradient accumulation used to reach the large batch.
Discussion
Experimental setup
All experiments are language-model pretraining. The data is C4 with the T5 tokenizer. The models are decoder-only, built on the OLMo codebase, in three sizes of 210M, 360M and 660M parameters (excluding embeddings), with widths of 1024, 1024 and 1408 and depths of 12, 24 and 24, using RoPE and QK LayerNorm in bf16 mixed precision. Sequences are 1024 tokens long, with about 2 million tokens per batch; training runs for about 20 times as many tokens as parameters, which is 3,200 steps for 360m and 6,400 steps for 660m.
All three optimizers first get a learning-rate sweep. AdamW additionally sweeps and at 360m; Shampoo, run as DistributedShampoo, additionally sweeps , the moving-average coefficient of the statistics and four exponent settings, and each of these sweeps improves the loss by less than 0.004. SOAP sweeps only the learning rate. The shared settings are , decoupled weight decay of , z-loss of , and warmup followed by cosine decay to 0.1 times the peak.
How “faster” is measured also needs spelling out. A cosine schedule depends on the total number of steps, so comparing intermediate checkpoints from the same run isn't fair. The authors run separate full SOAP trainings at 0.5, 0.625, 0.75 and 0.875 times the length, fit final loss against steps with , and find the length at which it catches up with the final losses of AdamW and Shampoo.
Main results

Start with the left panel. The blue, green and orange lines are AdamW, Shampoo and SOAP, each run for 6,400 steps, and SOAP ends lowest. The light orange line is SOAP with a shortened schedule, and the dashed line shows it catching up with AdamW's endpoint at about 3,600 steps. The middle panel switches the horizontal axis to wall-clock time, and the gaps between the lines shrink a little, because every SOAP and Shampoo step costs more than an AdamW step.
Using the paper's efficiency estimates, the numbers for three settings are:
| Setting | To match AdamW | To match Shampoo |
|---|---|---|
| 360m, 2M | 0.50 / 0.52 | 0.75 / 0.74 |
| 660m, 2M | 0.565 / 0.62 | 0.80 / 0.78 |
| 360m, 256K | 0.72 / 0.83 | 0.875 / 0.90 |
Compiled from the tick labels in Figure 2 of the paper; each cell is “step ratio / time ratio,” and the batch sizes in the first column are in tokens. The time ratios against Shampoo are our conversion (Shampoo's full runs take 1.06, 1.12 and 1.11 times as long as AdamW's, respectively).
With large batches, SOAP needs only 50% to 57% of AdamW's steps and about 52% to 62% of its time, and saves a bit more than 20% relative to Shampoo. That's where the abstract's claim comes from, that SOAP “reduces the number of iterations by over 40% and wall clock time by over 35% compared to AdamW, with approximately 20% improvements in both metrics compared to Shampoo.” In a further run taking 360m to 100 times as many tokens as parameters (33.5 billion), SOAP keeps its advantage over AdamW.
Frequency ablation
The numbers at the start of this note come from the right half of Figure 1, shown again here.

Both lines sit below AdamW's dashed line, so for every from 1 to 100, both beat AdamW. At the two are nearly identical (2.820 vs 2.824), as expected when recomputing every step leaves Adafactor versus Adam as the only difference. After that Shampoo climbs all the way to 2.932, while SOAP is still at 2.823 at and only rises to 2.870 at .
These results support the authors' explanation. SOAP is at 2.823 at , and at it's only about 0.018 above , still far below Shampoo's 2.868 and 2.898 at the same frequencies. What really hurts Shampoo is that the scaling freezes along with the basis.
Batch size

Ideally, doubling the batch size halves the steps needed to reach the same loss; that's the dashed line on the left. The farther a solid line drifts from its dashed line, the more of the extra batch is wasted. AdamW's blue solid line drifts further and further away from 512 onward, and at 2048 it needs nearly twice the ideal number of steps; SOAP's orange solid line stays closer to its own dashed line, which is why the authors say SOAP has a larger critical batch size. For fairness, every time the batch shrinks by a factor of , the frequency grows by , keeping the eigendecomposition's share of the overhead constant.
The right panel is the smallest batch, 256K. SOAP needs 25% fewer steps than AdamW and about 10% fewer than Shampoo. Compared with the table above, the advantage shrinks noticeably at small batch sizes. The authors also call this a preliminary result, and holding “batch times frequency” fixed may not be the best trade-off. Second-order methods gaining less at small batch sizes is consistent with earlier work, and the official README also recommends using as large a batch as you can.
Memory-saving variants and GaLore
SOAP uses more memory than AdamW. The paper tries two ways to cut it. Running Adafactor instead of Adam in the rotated space saves one copy with almost no change in loss. Rotating only the shorter side and using the identity on the other increases the loss somewhat more, but it still ties or slightly beats Shampoo. With both tricks combined, memory is , which on a standard Transformer is already less than AdamW's , and it still beats AdamW.
The full-rank version of GaLore is very similar to SOAP, differing mainly in three places. GaLore's projection comes from the SVD of the current gradient, while SOAP uses a moving average of ; GaLore stores momentum in the projected space and doesn't rotate it back when the basis changes, while SOAP stores it in the original space; and GaLore projects only one side. On the 210M-parameter model, GaLore's best result is 3.12, beating AdamW but losing to Shampoo's 3.10, so the authors consider each of SOAP's changes necessary.
The idea had actually been written down before. In Appendix B of their 2020 Distributed Shampoo paper, Anil et al. describe an algorithm essentially equivalent to SOAP on 2D layers, with no experiments attached, only a remark that unpublished experiments showed no improvement. SOAP's authors acknowledge this candidly in their related work and suggest the difference comes down to implementation details.
Limitations
The paper lists its own limitations clearly. The largest model has 660M parameters, two orders of magnitude smaller than the large models of the time; the authors assume the conclusions carry over to larger scales, but that “remains to be validated.” Only language models were tested. Throughput was measured on a single GPU, with comparisons of distributed implementations left for future work. All and operations run in fp32, and PyTorch's QR doesn't support lower precision, so the floor on the overhead can't be pushed down for now.
An independent check a year later
In 2025, Wen, Hall, Ma and Liang at Stanford published “Fantastic Pretraining Optimizers and Where to Find Them,” putting AdamW and ten newer optimizers through models from 130M to 1.2B parameters at 1 to 8 times the Chinchilla data budget, and comparing them after tuning each one carefully.
On 300M- and 520M-parameter models, they measured SOAP at 1.2 to 1.3 times faster than a well-tuned AdamW; at 1.2B parameters, SOAP and Muon were both down to about 1.1 times. They also found that once the data-to-model ratio reaches 8 times or more, SOAP and Kron overtake Muon, which is the strongest when data is scarce. Their SOAP implementation additionally splits weights into blocks and keeps momentum in bf16 at 1.2B parameters, so it differs slightly from the original.
Conclusion
SOAP starts from the equivalence that Shampoo with the power is Adafactor running in Shampoo's own eigenbasis. Once the expensive basis is separated from the cheap scaling, the basis can be updated every 10 steps or even less often, while the scaling is updated every step the way Adam does it; the hyperparameters shrink to just one more than AdamW, the frequency. On 360M- and 660M-parameter language models with large batches, it needs about half the steps of AdamW and 38% to 48% less wall-clock time.
To try it, you can copy the official soap.py straight into your project and start from the README settings, and frequency 10. First measure how much slower each step is than AdamW, then decide whether to stretch the frequency. When memory is tight, one-sided rotation and the Adafactor variant are fallbacks the paper has already tested.
The Fantastic paper singles SOAP out in its main text as one of the few papers that independently swept hyperparameters for its baseline, and its reported speedup came closest to what they measured. In optimizer papers, praise like that is rare.