Back to Blog
Articles

Gradient Descent, Illustrated

In 1847, Cauchy wrote down “take a step along the negative gradient” to work out the orbits of celestial bodies. When large language models are trained today, optimizer.step() still does the same thing; only the way that step is taken has grown into a whole family, from SGD and Adam to AdamW, Lion, and Muon. We draw a tilted valley from the shipping fees of twelve parcels and, with more than thirty animations computed live, see which gap each method fills in the one before it.

Open a PyTorch training script, and the loop almost always contains these two lines:

python
loss.backward()
optimizer.step()

backward() works out which direction each parameter should change in, and step() actually makes the change. This article is about the few lines inside step().

The directions have already been computed, so why not just follow them? It doesn't sound like there's much to say. Yet “how to take this one step” alone has sprouted a whole family, from SGD, Momentum, AdaGrad, Adam, and AdamW all the way to Lion and Muon. Almost every new method appeared because the one before it walked poorly on some kind of terrain.

We'll start from the very basics, with one straight line and twelve parcels. First we draw the “gradient”, then walk it through a long, narrow valley, random noise, cliffs, and saddle points, and see which gap each method fills in the one before it.

Before we set off, take a look at the whole route. Each row is a method, followed by the gap it fills:

Every method, and the gap it fillsStarting from BGD, each row names the gap the method fills.

Basics

  1. BGDThe starting point: every step reads all the data.
  2. SGD · MBGDReading all the data per step is too slow → sample one item or a small batch.
  3. SGDMZigzags and noise → accumulate a velocity.
  4. NAGMomentum overshoots → take the gradient at the look-ahead point.

One step size per parameter

  1. AdaGradOne learning rate for every parameter → scale each by its accumulated gradients.
  2. RMSProp · AdaDeltaAdaGrad's rate only ever shrinks → a moving average that can forget.
  3. Adam · NadamCombine momentum with per-parameter steps, and correct the zero-start bias.

Fixing Adam

  1. AMSGradAdam can converge to the wrong point → never let v shrink.
  2. AdamWInside Adam, L2 gets divided by √v̂ → decouple weight decay.
  3. Warmup · Cosine · SGDR · RAdam · Schedule-FreeHow should the rate change, and why does Adam need warmup? → schedules, a rectified variance, or no schedule at all.

Scale and geometry

  1. LARS · LAMBHuge batches: layers differ in ‖w‖/‖g‖ → a trust ratio per layer.
  2. Adafactor · GaLoreOptimizer state eats memory → factor it, or keep it in a low-rank subspace.
  3. SAMA low loss in a narrow valley → compare the worst loss within ρ.
  4. Shampoo · SOAP · SophiaDiagonal scaling can't fix a tilted valley → matrix preconditioners, or curvature with clipping.
  5. LionLess state and computation → keep only the sign.
  6. MuonWeights are matrices, not long lists → orthogonalize the whole update.

Every animation in this article is computed live in your browser, and each step uses exactly the formula written in the text. Press pause, and you can drag the slider to go through it one step at a time.

Start with a straight line

Suppose you run a small online shop and have just shipped 12 parcels. They weigh between 0.5 and 1.5 kg, and their shipping fees range from NT$92 to NT$150.

The fee looks roughly like “base fee + a surcharge per kg”, which we can write as a straight line:

y^=wx+b\hat y = w x + b

xx is the weight (kg), y^\hat y is the predicted shipping fee (in units of NT$100), ww is how much each kilogram adds, and bb is the base fee. From these 12 records, we want to find the best-fitting ww and bb.

Start with an arbitrary guess, NT$210 per kg plus a base fee of NT$20, that is, w=2.1w = 2.1, b=0.2b = 0.2. This line is too steep. It overestimates all 12 parcels, overcharging the lightest by NT$32 and the heaviest by NT$185.

How do we measure “how bad the guess is”? Square each parcel's error, then take the average:

L(w,b)=12n∑i=1n(wxi+b−yi)2L(w,b) = \frac{1}{2n}\sum_{i=1}^{n}\left(w x_i + b - y_i\right)^2

Squaring keeps positive and negative errors from canceling each other out, and it also penalizes parcels that are far off more heavily. The 12\frac{1}{2} in front is only there to cancel the 2 that the square brings down when we take the derivative in a moment.

This LL is called the loss function. Plugging in our guess gives L=0.71L = 0.71.

Next comes the most important idea in the whole article.

Every pair (w,b)(w, b) is a straight line, and each one has a loss value. Put ww on the horizontal axis and bb on the vertical axis, mark the loss at every point of the plane, and you get a terrain map. The lower the loss, the lower the ground. Finding the best line means finding the lowest point on this terrain.

Every line is one point on the landscapeLeft: twelve parcels and the current line. Right: the loss of every (w, b). Tap or drag on the right to choose a line yourself.
0.511.5012weight (kg)fee (×NT$100)
012-1012w (per kg)b (base fee)
NT$0 per kg + NT$120 base feeL = 0.016

The rings on the map are contour lines. Every point on the same ring has the same loss, and the closer you get to the star in the middle, the lower the loss. The star sits at w≈0.59w \approx 0.59, b≈0.61b \approx 0.61, that is, NT$59 per kg plus a base fee of NT$61, where L≈0.0005L \approx 0.0005.

Notice the shape of this terrain. It's a tilted valley, long and narrow. This shape will follow us through the whole article, and many of the methods later on are trying to deal with it.

Slope and gradient

Standing at some point on the terrain, how do you know which way is downhill?

Let's simplify first. Fix bb at its best value and move only ww, and the terrain reduces to a single curve. The slope of the tangent at any point on the curve is the derivative dLdw\frac{dL}{dw}:

The slope tells which way is downhillb is held at its best value and only w moves. The orange line is the tangent; its slope is the derivative dL/dw.
012012w (per kg)loss L
w = -0.40dL/dw = -1.08slope < 0: downhill is to the right

A positive slope means the loss grows as ww grows, so downhill is to the left; a negative slope means downhill is to the right; a slope of 0 means you've reached the bottom. Moving against the slope means moving downhill.

When both parameters can move, each has its own partial derivative, the slope when “only it moves and everything else stays put”. For our loss function:

∂L∂w=1n∑i=1n(wxi+b−yi) xi\frac{\partial L}{\partial w} = \frac{1}{n}\sum_{i=1}^{n} (w x_i + b - y_i)\,x_i ∂L∂b=1n∑i=1n(wxi+b−yi)\frac{\partial L}{\partial b} = \frac{1}{n}\sum_{i=1}^{n} (w x_i + b - y_i)

Stack them into a vector, and that's the gradient:

∇L=(∂L∂w, ∂L∂b)\nabla L = \left(\frac{\partial L}{\partial w},\ \frac{\partial L}{\partial b}\right)

These two expressions aren't hard to read. Take each parcel's error, multiply it by this parameter's influence on that parcel, and average. bb has an influence of 1 on every parcel; ww's influence is the parcel's weight xx, so the heavier the parcel, the more say it has about ww.

At the starting point (2.1, 0.2)(2.1,\ 0.2), all 12 parcels are overestimated, and the calculation gives ∇L≈(1.24, 1.10)\nabla L \approx (1.24,\ 1.10). Both components are positive, which means that increasing either ww or bb would make the loss larger.

The gradient is perpendicular to the contour and points uphillArrows show direction only. The dashed curve is the contour through the point; −∇L is the steepest way down.
012-1012w (per kg)b (base fee)∇L−∇L
∇L = (1.04, 0.95)‖∇L‖ = 1.41

The figure shows two properties of the gradient:

  1. It points in the direction of steepest ascent, so −∇L-\nabla L is the direction of steepest descent.
  2. It is perpendicular to the contour line through that point.

Why? Near a point θ=(w,b)\theta = (w, b), take a small step Δ\Delta, and the loss changes by roughly

L(θ+Δ)≈L(θ)+∇L⋅ΔL(\theta + \Delta) \approx L(\theta) + \nabla L \cdot \Delta

This is a first-order Taylor expansion. The dot product can be written as ∇L⋅Δ=∥∇L∥ ∥Δ∥cos⁡φ\nabla L \cdot \Delta = \lVert\nabla L\rVert\,\lVert\Delta\rVert \cos\varphi, where φ\varphi is the angle between the gradient and the step. For a fixed step length, the loss drops the most when φ=180∘\varphi = 180^\circ, that is, when you walk opposite to the gradient. Walking along a contour line leaves the loss unchanged, so ∇L⋅Δ=0\nabla L \cdot \Delta = 0 and the angle is 90∘90^\circ; that's why the gradient is perpendicular to the contour.

BGD: follow the negative gradient

Once you know which way is downhill, the rest is simple. Take a small step toward −∇L-\nabla L, then repeat:

θt+1=θt−η ∇L(θt)\theta_{t+1} = \theta_t - \eta\, \nabla L(\theta_t)

η\eta (eta) is the learning rate, which sets how far each step goes. This is gradient descent. Because every step computes the gradient from all 12 parcels, it's also called batch gradient descent (BGD).

The idea is older than computers. The earliest written record we can find is a three-page note by Cauchy from the session of the French Academy of Sciences on October 18, 1847, “Méthode générale pour la résolution des systèmes d'équations simultanées”, printed in Comptes Rendus, volume 25, pages 536 to 538. He wanted to solve the simultaneous equations that determine the orbit of a celestial body. His approach was to square each equation and add them up, then keep shrinking this sum by moving in the direction of the negative partial derivatives. There was no fixed learning rate yet. Cauchy gave several ways to choose the step, one of which was to keep searching along this direction and stop only at the point where the function is smallest.

With η=0.5\eta = 0.5, the first step is:

θ1=(2.1, 0.2)−0.5×(1.24, 1.10)=(1.48, −0.35)\begin{aligned} \theta_1 &= (2.1,\ 0.2) - 0.5 \times (1.24,\ 1.10) \\ &= (1.48,\ -0.35) \end{aligned}

The loss drops from 0.71 to 0.040, losing more than 90% in a single step.

Step by step along the negative gradientBatch gradient descent with η = 0.5, starting from NT$210 per kg plus a base fee of NT$20.
0.511.5012weight (kg)fee (×NT$100)
012-1012w (per kg)b (base fee)
NT$60 per kg + NT$60 base feeL = 0.0005

A beautiful first step. What comes next?

Once the point drops into the valley, it crawls slowly along the valley floor. In the parcel plot, the line has long since settled close to the data, and the hundred-odd steps that follow are only fine-tuning; it takes until step 168 to get within 0.03 of the minimum.

Why so slow? Would turning η\eta up a little help?

Learning rate: how far to step

Let's go back to the simplest case first, a one-dimensional bowl f(u)=12au2f(u) = \frac{1}{2} a u^2. aa is the bowl's curvature, that is, its second derivative; the larger it is, the steeper the bowl. The gradient is aua u, so each step is:

ut+1=ut−η a ut=(1−ηa) utu_{t+1} = u_t - \eta\, a\, u_t = (1 - \eta a)\, u_t

Every step multiplies uu by (1−ηa)(1 - \eta a) once more.

One bowl, four step sizesf(u) = ½u², whose curvature is 1. All four runs start at u = −2.4.
η = 0.1: crawls
η = 0.9: quick
η = 1.6: swings across
η = 2.1: diverges
out of view

In the figure, a=1a = 1, and the four runs differ only in η\eta:

  • η=0.1\eta = 0.1: each step shrinks it by only 10%, a slow crawl.
  • η=0.9\eta = 0.9: each step leaves only 10%, so it reaches the bottom quickly.
  • η=1.6\eta = 1.6: 1−ηa=−0.61 - \eta a = -0.6, so every step jumps to the other side, but lands closer each time; it converges while swinging back and forth.
  • η=2.1\eta = 2.1: 1−ηa=−1.11 - \eta a = -1.1, so every step jumps to the other side and lands even farther out; the swings keep growing, and it diverges.

So the condition for convergence is ∣1−ηa∣<1|1 - \eta a| < 1, that is, η<2/a\eta < 2/a. The steeper the bowl, the smaller the learning rate has to be.

A long, narrow valley

Our terrain is two-dimensional, and the trouble is that it has two curvatures.

Cut across the valley and the terrain is steep; follow the valley floor and it's very flat. In linear-algebra terms, these are the two eigenvalues of the loss's matrix of second derivatives (the Hessian), which in this example are 2.05 and 0.045, a factor of 46 apart. This ratio is called the condition number, written κ\kappa.

The same η\eta has to take care of both directions at once. To avoid diverging in the steep direction, η\eta must be smaller than 2/2.05≈0.982/2.05 \approx 0.98; but then in the flat direction, each step can shrink the remaining distance by at most 0.98×0.045≈4.4%0.98 \times 0.045 \approx 4.4\%, so it takes at least several dozen steps. Make η\eta a little larger and the steep direction bounces back and forth; make η\eta a little smaller and the flat direction barely moves.

A long, narrow valleySame parcels, same η = 0.9. Only the way the weight is written changes.
-1012-1012wb
η = 0.9Within 0.03 of the minimum at step 93

With η=0.9\eta = 0.9, the point bounces back and forth in the steep direction, tracing a zigzag while it slowly advances along the valley floor, and doesn't arrive until step 93.

Why is this valley tilted? The valley floor runs roughly along (1,−1)(1, -1), that is, ww up by 1 and bb down by 1. For a 1 kg parcel, the predicted fee w×1+bw \times 1 + b doesn't change at all. The average weight of these 12 parcels happens to be exactly 1 kg, so adjusting along this direction barely affects the predictions overall, and the loss hardly changes. The slope and the intercept cancel each other out, and that forms a tilted valley.

Click “minus the mean” on the figure to subtract 1 kg from every parcel's weight, and the model becomes y^=w(x−1)+b′\hat y = w(x - 1) + b'. It can draw exactly the same lines, but ww and the new intercept b′b' no longer pull against each other. The valley straightens out, the condition number drops to 10.9, and it arrives at step 46. Then divide by the standard deviation as well, which is “standardization”, and the slope becomes a new w′w' too. The curvatures in the two directions become equal, the valley turns into a round bowl with a condition number of 1, and it arrives in two steps.

This is why inputs are often standardized before training. But a deep network easily has hundreds of millions of parameters, and the inputs to every layer keep changing, so we have no way of reshaping the terrain by hand one piece at a time.

The methods that follow attack the problem from roughly three directions:

  1. Remember the direction you just came from, so the back-and-forth bouncing cancels out (momentum).
  2. Give each parameter its own step size (AdaGrad, RMSProp, Adam).
  3. Estimate the shape of the terrain, or measure a step with a different ruler (Newton's method, Shampoo, Muon).

Hitting a cliff

Too large a learning rate makes it diverge. But sometimes, even with a modest learning rate, a single step can fling you out.

The step that hits the cliffA gentle slope ends in a steep wall at u = 2.4. Both runs use η = 0.6; the right one caps the gradient's size at 1.
no clipping
-2-1012024uL
clip to 1
-2-1012024uL

This terrain is a gentle slope followed by a steep wall. The point follows the slope to the wall, and one step happens to land on the wall's face. The gradient on the wall is huge, and the next step bounces the point straight to u=−5.8u = -5.8, out of the frame. It spends nearly ten steps climbing back to the wall, then lands on the face again and gets bounced out again; in 40 steps it's thrown out three times, twice right out of the frame.

Real RNNs have walls like this too. In 2013, while analyzing why RNNs are hard to train, Pascanu, Mikolov, and Bengio put forward a hypothesis. In the directions where gradients explode, the curvature is usually large as well, so the loss surface forms a steep wall; step onto it, and you're flung out of the valley you were in. In On the difficulty of training recurrent neural networks, they drew such a wall using an RNN with a single hidden unit.

The fix is straightforward. When the gradient is too large, scale it down. The most common version is clipping by norm:

g←g⋅min⁡ ⁣(1, c∥g∥)g \leftarrow g \cdot \min\!\left(1,\ \frac{c}{\lVert g \rVert}\right)

If the gradient's length exceeds cc, it gets scaled down to cc with its direction unchanged. The “clip to 1” panel caps the gradient's size at 1, and the point keeps making small moves back and forth by the wall instead of flying off. In PyTorch, this is torch.nn.utils.clip_grad_norm_.

What should cc be? Pascanu et al. suggest first watching the average gradient norm over a stretch of training; anything from half of it to ten times it still converges, though the speed is affected.

SGD and MBGD: a few samples at a time

BGD has to go through all of the data for every single step. With 12 parcels, that's no problem; switch to ImageNet's 1.28 million training images, and every step means running all 1.28 million images through a forward and a backward pass first.

Stochastic gradient descent (SGD) draws just one random sample per step and uses its gradient in place of the gradient over all the data:

θt+1=θt−η ∇ℓit(θt)\theta_{t+1} = \theta_t - \eta\, \nabla \ell_{i_t}(\theta_t)

ℓi\ell_i is the loss of sample ii on its own, and iti_t is the sample drawn at step tt.

With only one sample, the gradient is of course inaccurate. But if every sample is equally likely to be drawn, its expected value is exactly the full gradient:

Ei[∇ℓi(θ)]=1n∑i=1n∇ℓi(θ)=∇L(θ)\mathbb{E}_{i}\big[\nabla \ell_i(\theta)\big] = \frac{1}{n}\sum_{i=1}^{n} \nabla \ell_i(\theta) = \nabla L(\theta)

On average the direction is right; each step just carries some noise.

The compromise is to draw a small batch each time, say 4 samples, and average their gradients. This is called mini-batch gradient descent (MBGD). The larger the batch, the smaller the noise; the variance of the gradient is roughly inversely proportional to the batch size.

Twelve parcels, four, or one per stepAll three use η = 0.5. The left plot shows the one-parcel run's current line and circles the parcel it drew this step.
0.511.5012weight (kg)fee (×NT$100)
012-1012w (per kg)b (base fee)
BGD (12): 2400 parcels read · Within 0.03 of the minimum at step 168MBGD (4): 800 parcels read · Within 0.03 of the minimum at step 169SGD (1): 200 parcels read · Within 0.03 of the minimum at step 178

All three routes use the same η=0.5\eta = 0.5. BGD is the smoothest; MBGD wobbles slightly all the way; SGD shakes the most, and the circled parcel in the parcel plot is the one it drew for this step.

But look at how much data each one has read. All three get within 0.03 of the minimum at around step 170. BGD reads 12 samples per step and has read 2,016 by the time it arrives; MBGD has read 676; SGD only 178. Measured by the amount of data read, the shakiest route is the cheapest.

The learning rate has to shrink

SGD has a problem that BGD doesn't. At the minimum, the full gradient is 0, but each parcel's own gradient isn't. The 12 parcels can't all lie exactly on the same line, so every one of them is still pulling the parameters toward itself. Whichever parcel gets drawn, the parameters drift a little its way.

A fixed learning rate never settlesSingle-parcel SGD started exactly at the minimum, zoomed to ±0.12 around it. Both panels draw the parcels in the same order.
fixed η = 0.5
decaying η = 0.5/(1+t/20)
fixed η = 0.500: last 50 steps average 4.2/100 from the minimumdecaying η = 0.031: last 50 steps average 1.5/100 from the minimum

This figure starts right at the minimum and looks at one sample at a time. With a fixed learning rate, the parameters keep circling around the minimum, and over the last 50 steps their average distance from it is 0.042; when the learning rate shrinks with the step count as ηt=0.5/(1+t/20)\eta_t = 0.5/(1 + t/20), the circling narrows along with it, and the average distance drops to 0.015.

So how should the learning rate shrink? Shrink it too fast and you stop before you get there; too slowly and the noise never goes away.

Finding the minimum means finding the point where ∇L=0\nabla L = 0, that is, finding the root of an equation. Robbins and Monro's 1951 “A Stochastic Approximation Method” studied exactly this kind of problem, except that every observation they could make was noisy. Later authors distilled their step-size conditions into two:

∑t=1∞ηt=∞,∑t=1∞ηt2<∞\sum_{t=1}^{\infty} \eta_t = \infty, \qquad \sum_{t=1}^{\infty} \eta_t^2 < \infty

The first makes the total distance long enough to get there however far away you start; the second keeps the total accumulated noise finite, so you can come to a stop in the end. ηt∝1/t\eta_t \propto 1/t satisfies both, and the decay in the figure above belongs to this family.

Noise has its uses

SGD's noise isn't all bad.

Stuck on a saddlef(u, v) = ½u² − ½v² + ¼v⁴, η = 0.2, starting at (1.8, 0). The orange run adds noise of about 0.0001 to every gradient.
012-101uvsaddleminimumminimum
gradient descent: f = 0.000, ‖∇f‖ = 0.000with a little noise: f = -0.250, ‖∇f‖ = 0.000

This terrain is f(u,v)=12u2−12v2+14v4f(u, v) = \frac{1}{2}u^2 - \frac{1}{2}v^2 + \frac{1}{4}v^4. The origin is a saddle point. Along the uu direction it's a valley floor, along the vv direction it's a hilltop, and the gradient there is 0. The two real valley floors are at v=±1v = \pm 1.

GD starts from (1.8, 0)(1.8,\ 0). Its gradient in the vv direction is 0 from the start and stays 0 forever, so it slides all the way into the saddle point and stops there. Add a tiny bit of noise (a standard deviation of just 0.0001), and when the point nears the saddle it gets gently nudged off the ridge, dropping into the valley floor at (0,−1)(0, -1) by step 64.

In a two-dimensional figure, a saddle point looks like a special case. Dauphin et al.'s 2014 “Identifying and attacking the saddle point problem in high-dimensional non-convex optimization” argues that the higher the dimension, the more saddle points there are relative to local minima, and that slow training is often due to getting stuck on the flat plateaus near saddle points, rather than falling into bad local minima.

Noise can help you escape, but in theory this comes with conditions. Ge et al. proved in 2015 that if every saddle point has a clear downhill direction (negative curvature), the function is smooth enough, and the noise has variance in every direction, then noisy SGD escapes saddle points within a polynomial number of steps. Our example starts right on top of the ridge, a deliberately chosen worst case; from a random starting point, GD almost never stops exactly on a saddle point, though it may move very slowly across a flat plateau.

SGDM: remember where you were going

Back to the long, narrow valley. The problem with the zigzag is that in the direction across the valley, the gradient flips sign at every step, pushing toward one side of the valley this step and back to the other side the next. In the direction along the valley floor, the gradient is small, but it always points the same way.

If each step looked not only at the current gradient but also at the gradients of the previous few steps, the parts that flip back and forth would cancel out, and the part that keeps pointing the same way would build up.

Exponential moving average

The most common way to “look at the previous few steps” is the exponential moving average (EMA):

mt=β mt−1+(1−β) gtm_t = \beta\, m_{t-1} + (1 - \beta)\, g_t

The new average is the old average times β\beta, plus the new gradient times 1−β1 - \beta. With β=0.9\beta = 0.9, the new gradient gets a 10% share; a gradient from ten steps ago still keeps 0.910≈35%0.9^{10} \approx 35\% of its influence, and the older a gradient is, the fainter it gets. There's no need to store past gradients one by one; a single number is enough.

Averaging the zigzagGradients of the η = 0.9 zigzag, split into the direction across the valley and the direction along it. The orange line is the moving average with β = 0.9.
across the valley: the sign flips every step
0102030-1.701.7step
along the valley: small, always the same sign
010203000.06step
gradient at each stepmoving average

We split the gradients from GD's path through the valley into two components, across the valley and along it. The across component is about 1.7 at the first step and alternates in sign after that, and the EMA quickly flattens it to near 0; the along-the-floor component is only about 0.06, but its direction never changes, and the EMA holds on to it steadily.

Momentum

Use this average to update the parameters, and you have momentum. SGD with momentum is often abbreviated SGDM:

vt=β vt−1+gt,θt+1=θt−η vtv_t = \beta\, v_{t-1} + g_t, \qquad \theta_{t+1} = \theta_t - \eta\, v_t

This vv drops the (1−β)(1 - \beta) factor, so it's 11−β\frac{1}{1-\beta} times the EMA above. When the gradient keeps pointing the same way, vv builds up to g1−β\frac{g}{1-\beta}, which is 10 times for β=0.9\beta = 0.9. The small gradient that couldn't get anywhere in the flat direction gets amplified this way.

The metaphor comes from physics. vv is like the velocity of a ball, the gradient is the push the slope gives it, and β\beta decides how much of the velocity is kept at each step.

This idea also predates deep learning by a long way. In 1964, Polyak wrote it as

xn+1=xn−α∇f(xn)+β (xn−xn−1)x^{n+1} = x^n - \alpha \nabla f(x^n) + \beta\,(x^n - x^{n-1})

that is, on top of the gradient step, add “the previous displacement” times β\beta. Eliminate vv from the two lines above, and you get exactly the same expression, with α\alpha being η\eta. Polyak himself called it the heavy ball method, viewing it as a discrete version of an object moving in a potential field, and recommended taking β\beta between 0.8 and 0.99 in most cases.

He also worked out how much it speeds things up. On a quadratic with condition number κ\kappa, the number of steps GD needs is roughly proportional to κ\kappa, while the heavy ball method needs a number proportional only to κ\sqrt{\kappa}. Our valley has κ≈46\kappa \approx 46, so κ≈6.8\sqrt{\kappa} \approx 6.8. But this result holds only on quadratics, or near a minimum where the function looks enough like one. In 2016, Lessard et al. constructed a one-dimensional function with condition number 25 on which the heavy ball method, run with the optimal parameters for quadratics, falls into oscillations that never die down.

Same η, plus momentumThe arrow on the orange run is its next move, −ηv.
012-1012w (per kg)b (base fee)
gradient descent (η = 0.9): Within 0.03 of the minimum at step 93momentum (η = 0.9, β = 0.7): Within 0.03 of the minimum at step 19

With the same η=0.9\eta = 0.9, GD doesn't arrive until step 93; add momentum with β=0.7\beta = 0.7, and it arrives at step 19.

Look closely at the momentum route. At first it's no better behaved; in fact, it swings widely across the valley several times. What momentum cancels is the kind of bouncing that flips sign every step; the velocity the ball has already built up still takes several steps to come to a stop.

What happens if β\beta is too large?

How much to rememberMomentum with η = 0.1 and three values of β. With a steady gradient, the step settles at η/(1 − β).
β = 0.5 · η/(1 − β) = 0.2
β = 0.9 · η/(1 − β) = 1.0
β = 0.99 · η/(1 − β) = 10.0
β = 0.5: Not within 0.03 of the minimum after 200 stepsβ = 0.9: Within 0.03 of the minimum at step 51β = 0.99: Not within 0.03 of the minimum after 200 steps

All three panels use η=0.1\eta = 0.1. β=0.5\beta = 0.5 remembers too little and helps only a bit, failing to arrive within 200 steps; β=0.9\beta = 0.9 arrives at step 51; β=0.99\beta = 0.99 has almost no friction, so the ball circles around and around near the valley floor and doesn't settle within 200 steps either.

NAG: look before you step

Momentum is slow to react. Near the bottom, the ball overshoots with the velocity it has built up, and only turns back after the gradient has pushed the other way for several steps.

Nesterov momentum (Nesterov Accelerated Gradient, NAG) changes just one thing. Since this step will move along the velocity anyway, go there first, check the slope there, and then decide how to correct:

vt=β vt−1+∇L(θt−ηβ vt−1)θt+1=θt−η vt\begin{aligned} v_t &= \beta\, v_{t-1} + \nabla L(\theta_t - \eta\beta\, v_{t-1}) \\ \theta_{t+1} &= \theta_t - \eta\, v_t \end{aligned}

The only difference is where the gradient is computed. Momentum computes it at the current position; NAG computes it at the look-ahead point, “where one step at the current velocity would take you”. If the ground ahead has already started to rise, NAG feels it a step early and taps the brakes a little in advance.

Look ahead before reading the slopeBlue reads the gradient where it stands; orange first moves by −ηβv to the hollow look-ahead point and reads it there. Arrows show direction only.
012-1012w (per kg)b (base fee)
momentum: Within 0.03 of the minimum at step 37Nesterov: Within 0.03 of the minimum at step 22η = 0.5, β = 0.8

The hollow circles are NAG's look-ahead points. With the same η=0.5\eta = 0.5 and β=0.8\beta = 0.8, momentum arrives at step 37 and NAG at step 22.

Why does it work especially well in a valley? Sutskever et al. showed in 2013 that on a quadratic, NAG amounts to using a different momentum β(1−ηλ)\beta(1 - \eta\lambda) in each eigendirection, where λ\lambda is the curvature in that direction. The larger the curvature, the smaller the momentum. Plugging in this figure's settings, the direction across the valley has λ=2.05\lambda = 2.05, leaving an effective momentum of only 0.8×(1−0.5×2.05)≈−0.020.8 \times (1 - 0.5 \times 2.05) \approx -0.02, with almost no drive left; the direction along the floor has λ=0.045\lambda = 0.045 and keeps 0.78. Directions prone to bouncing get damped, and directions that need speeding up still speed up.

The method comes from Nesterov's 1983 paper, where it was originally a theoretical result in convex optimization, guaranteeing that the error decreases at a rate of 1/k21/k^2. The “compute the gradient at the look-ahead point” formulation is a rewrite by Sutskever, Martens, Dahl, and Hinton in “On the importance of initialization and momentum in deep learning”. They had momentum rise slowly from a small value to 0.99 or even 0.999, paired it with carefully designed initialization, and got SGD with momentum to match the more complex second-order methods of the time on deep autoencoders.

AdaGrad: every parameter goes its own way

So far, all parameters have shared the same η\eta.

For many problems, that doesn't make sense. The AdaGrad authors, Duchi, Hazan, and Singer, use text data as their example. Common words show up in every sample, so their parameters are updated all the time; rare words appear only once in a while, but when they do, they're often highly predictive. The same learning rate may be too large for the former and too small for the latter.

AdaGrad has each parameter keep the sum of all its past squared gradients, and divides by its square root:

Gt=Gt−1+gt2θt+1=θt−ηGt+ϵ gt\begin{aligned} G_t &= G_{t-1} + g_t^2 \\ \theta_{t+1} &= \theta_t - \frac{\eta}{\sqrt{G_t} + \epsilon}\, g_t \end{aligned}

The squaring, square root, and division here are all done separately for each parameter. ϵ\epsilon is a very small number that prevents division by zero.

A parameter whose gradients are always large sees GG grow fast, so its steps get smaller; a parameter that rarely gets updated has a small GG, so it can take a big step as soon as a gradient comes along. The first step is even more interesting. G1=g12G_1 = g_1^2, so every parameter's first step is ±η\pm\eta, no matter how large the gradient itself is.

AdaGrad: each coordinate gets its own learning rateη = 0.5. G adds up every squared gradient so far, so η/√G can only shrink. Its first step moves w and b by exactly η each.
012-1012w (per kg)b (base fee)
effective learning rate
00.250.5η/√Gη = 0.5w: 0.392b: 0.445
G = (1.624, 1.261)η/√G = (0.392, 0.445)97% of G comes from the first 5 steps

With η=0.5\eta = 0.5, the first step moves ww and bb by exactly 0.5 each. After that, the effective learning rates of the two parameters sit at about 0.39 and 0.45 and practically never move again. 97% of GG comes from the first five steps. At the start, you're far from the minimum and the gradients are large, so each square adds a big chunk; later the gradients keep shrinking, and adding more of them barely makes a difference. This run doesn't get within 0.03 until step 203, slower than BGD's 168 steps.

GG only ever grows, so each parameter's effective learning rate η/Gt\eta/\sqrt{G_t} only ever shrinks, and it's set mainly by those first few steps. On convex problems, this is exactly the mechanism that makes it converge; but training a deep network takes hundreds of thousands of steps over terrain that keeps changing, and the learning rate often becomes too small to move before it gets anywhere good.

RMSProp and AdaDelta: learning to forget

Since the trouble comes from “adding up every past gradient”, just remember the recent ones.

Replace AdaGrad's running sum with an exponential moving average, and you get RMSProp:

vt=ρ vt−1+(1−ρ) gt2θt+1=θt−ηvt+ϵ gt\begin{aligned} v_t &= \rho\, v_{t-1} + (1 - \rho)\, g_t^2 \\ \theta_{t+1} &= \theta_t - \frac{\eta}{\sqrt{v_t} + \epsilon}\, g_t \end{aligned}

ρ\rho is usually 0.9. The vv here tracks the average of the squared gradients, which is a different thing from the velocity vv in the momentum section. Large gradients from long ago are gradually forgotten, so the effective learning rate can shrink and can also grow back.

RMSProp has an unusual history. It has no paper. It comes from the slides for Lecture 6, section 5 (Lecture 6e) of the Coursera course “Neural Networks for Machine Learning” that Hinton taught in 2012. The slide shows exactly 0.9 MeanSquare+0.1 g20.9\,\mathit{MeanSquare} + 0.1\,g^2, then divides the gradient by its square root, and notes that this is unpublished work by Tijmen Tieleman. Later papers that cited it could only cite these lecture slides.

From a steep slope onto a long, gentle oneSlope 1 for u < 0, slope 0.1 after that, bottom at u ≈ 10. AdaGrad η = 0.5; RMSProp η = 0.1, ρ = 0.9. Both start at u = −3.
where each one is
-20246810uLAdaGradRMSProp
step size |Δu| (log scale)
060120180240step|Δu|0.0010.010.11AdaGradRMSProp
AdaGrad u = 3.45RMSProp u = 10.10

The terrain in this figure starts with a steep stretch of slope 1, followed by a long, gentle stretch with a slope of only 0.1, and the bottom is at u≈10u \approx 10. Both start from u=−3u = -3; AdaGrad uses η=0.5\eta = 0.5, and RMSProp uses η=0.1\eta = 0.1, ρ=0.9\rho = 0.9.

On the steep stretch, both denominators get inflated by the large gradients. Once on the gentle stretch, AdaGrad's GG still remembers every step on the steep part, and its step size drops to about 0.01; RMSProp forgets quickly, its denominator shrinks along with the gradient, and its step size comes back to about 0.1. By step 100, AdaGrad has only reached u≈1.7u \approx 1.7, while RMSProp is already at 5.1; RMSProp reaches the bottom around step 150, while AdaGrad, by step 240, has only made it to 3.45. The one given the larger learning rate turns out to be the slower one.

That same year, Zeiler's AdaDelta used the same moving average and thought one step further. He noticed that the updates of SGD and AdaGrad have mismatched units. A gradient's units are “loss ÷ parameter”, so using it to update a parameter gets the units wrong; only after also dividing by a second derivative (we'll see this again with Newton's method) do the units come back to “parameter”. AdaDelta doesn't actually compute any second derivatives. Instead, it replaces the η\eta in the numerator with the root mean square of “past updates”:

Δθt=−RMS[Δθ]t−1RMS[g]t gt\Delta\theta_t = -\frac{\mathrm{RMS}[\Delta\theta]_{t-1}}{\mathrm{RMS}[g]_t}\, g_t

Once the units of the numerator and denominator cancel, the update has the same units as the parameter, and you don't even need to set a learning rate.

Adam: putting the two together

At this point, we have two tools in hand. Momentum uses a moving average of the gradient to deal with directions that keep flip-flopping; RMSProp uses a moving average of the squared gradient to adjust the step size for each parameter.

Adam, proposed by Kingma and Ba in 2014, puts the two together:

mt=β1 mt−1+(1−β1) gtvt=β2 vt−1+(1−β2) gt2m^t=mt1−β1t,v^t=vt1−β2tθt=θt−1−α m^tv^t+ϵ\begin{aligned} m_t &= \beta_1\, m_{t-1} + (1 - \beta_1)\, g_t \\ v_t &= \beta_2\, v_{t-1} + (1 - \beta_2)\, g_t^2 \\ \hat m_t &= \frac{m_t}{1 - \beta_1^t}, \qquad \hat v_t = \frac{v_t}{1 - \beta_2^t} \\ \theta_t &= \theta_{t-1} - \alpha\, \frac{\hat m_t}{\sqrt{\hat v_t} + \epsilon} \end{aligned}

mm is the first-moment estimate of the gradient, that is, its mean; vv is the second-moment estimate, that is, the mean of its square. In the Adam paper, the learning rate is written α\alpha, playing the same role as η\eta earlier. The paper's defaults are α=0.001\alpha = 0.001, β1=0.9\beta_1 = 0.9, β2=0.999\beta_2 = 0.999, ϵ=10−8\epsilon = 10^{-8}, and these numbers are still PyTorch's defaults today.

We've seen the first two lines before; the third is new.

Bias correction

Both mm and vv start from 0. At the first step, m1=0.1 g1m_1 = 0.1\, g_1, only a tenth of the true gradient; v1=0.001 g12v_1 = 0.001\, g_1^2, only a thousandth. Both estimates are dragged down by the zero they started from.

Expand mtm_t and assume the gradient's expected value stays at E[g]\mathbb{E}[g], and you can work out that E[mt]=(1−β1t) E[g]\mathbb{E}[m_t] = (1 - \beta_1^t)\,\mathbb{E}[g]. So dividing by 1−β1t1 - \beta_1^t corrects the bias; the same goes for vv.

Why is the correction a must? The two estimates are biased by different amounts. β2=0.999\beta_2 = 0.999 drags vv down much more than mm, so without correction the denominator is too small and the steps are too big.

Adam's moving averages start from zeroConstant gradient g = 1, β₁ = 0.9, β₂ = 0.999. Dashed: m and v as stored; solid: after dividing by 1 − βᵗ. The time axis is logarithmic.
m and v
110100100000.51tmvm̂ = v̂
step multiplier
110100100001357tuncorrected m/√vhighest: 6.6× at t = 12corrected m̂/√v̂
mt = 1.000vt = 0.6323uncorrected m/√v = 1.26corrected m̂/√v̂ = 1.00

Suppose the gradient is a constant 1. The corrected m^t/v^t\hat m_t / \sqrt{\hat v_t} is 1 from the very first step, so the step size is exactly α\alpha. Without correction, the first step is 0.1/0.001≈3.160.1/\sqrt{0.001} \approx 3.16 times as large, shoots up to 6.6 times around step 12, and is still 1.26 times at step 1,000. In their variational autoencoder experiments, Kingma and Ba saw that the closer β2\beta_2 is to 1, the more unstable training without bias correction is at the start.

Each step is about α\alpha

After correction, m^t/v^t\hat m_t / \sqrt{\hat v_t} falls roughly around ±1\pm 1. Multiply the whole loss by 100, and the gradient is multiplied by 100 too; m^\hat m and v^\sqrt{\hat v} scale up together, and apart from the tiny ϵ\epsilon, the update barely changes. Each parameter moves about α\alpha per step, regardless of how large the gradient itself is.

This is the main reason Adam is so easy to use. SGD's learning rate has to be tuned to the scale of the gradient; Adam's α\alpha is closer to “how much you want the parameters to move per step”, and when you switch to a different model, the default often still works.

Four update rules, one valleyAll four use the full 12-parcel gradient, so only the update rule differs; each learning rate was picked to work well here, and Adam keeps the default β₁ = 0.9, β₂ = 0.999. The count is the first step within 0.03 of the minimum.
GD · η = 0.9
Momentum · η = 0.3, β = 0.9
RMSProp · η = 0.02, ρ = 0.9
Adam · η = 0.2
GD: arrives at step 93Momentum: arrives at step 47RMSProp: arrives at step 130Adam: arrives at step 87

Four methods in the same valley, each with its learning rate tuned to a value that works well here; the numbers are the steps at which each first gets within 0.03 of the minimum. All four use the gradient over all 12 parcels, so the first two are GD without sampling and GD with momentum. GD (η=0.9\eta = 0.9) arrives at step 93; momentum (η=0.3\eta = 0.3, β=0.9\beta = 0.9) at step 47; RMSProp (η=0.02\eta = 0.02) at step 130; Adam (α=0.2\alpha = 0.2) at step 87.

Adam isn't much faster. We'll save the reason for the section on “making the valley round”; it has to do with the valley being tilted.

Adam can also use Nesterov momentum. In 2016, Dozat worked NAG's “look first” into Adam and called it Nadam. NAG first takes a step along the momentum and then measures the gradient; Nadam doesn't compute an extra gradient, but uses “the next step's momentum” β1m^t\beta_1 \hat m_t directly in the update, plus this step's gradient. The NAdamW baseline later used by AlgoPerf (a public benchmark that compares how fast training algorithms are; more on it in the “Which one is fastest?” section) is Nadam plus the AdamW weight decay from the next section.

Adam gets things wrong too

Adam quickly became the default choice. But over the next few years, people pointed out its problems one after another.

Forgetting the big gradients that matter

Reddi, Kale, and Kumar constructed a tiny example. The parameter xx is restricted to [−1,1][-1, 1], and every three steps form one cycle; the loss is CxCx at the first step and −x-x at the other two. With C=3C = 3, the total gradient over one cycle is 3−1−1=1>03 - 1 - 1 = 1 > 0, so xx should be pushed toward −1-1.

Adam's denominator is what goes wrong here. As soon as the rare large gradient 3 appears, vv jumps up right away, and this step's g/vg/\sqrt{v} is squashed to about 1.05; for the next two steps the gradient is only −1-1, vv quickly shrinks again, and these two steps are inflated to −0.76-0.76 and −0.97-0.97. Over one cycle, measured in units of αt\alpha_t, it ends up 0.68 further in the wrong direction.

Adam heads the wrong wayReddi et al.'s example: on x ∈ [−1, 1] the gradient is +3 every third step and −1 otherwise, so the minimum is at x = −1. β₁ = 0, β₂ = 0.1, α_t = 0.5/√t, both start at x = 1.
0100200300-101txminimum x = −1AdamAMSGrad
gradient: +3 once every 3 steps, −1 otherwise
t = 300, gt = -1Adam: x = 1.00, g/√v = -0.97AMSGrad: x = -0.98, g/√v = -0.35

Both start from x=1x = 1, with a learning rate of 0.5/t0.5/\sqrt{t}. Adam gets pushed back to +1+1 every three steps and stays stuck at the wrong end; AMSGrad reaches about −0.40-0.40 by step 100 and −0.98-0.98 by step 300.

Here β1=0\beta_1 = 0 and β2=1/(1+C2)=0.1\beta_2 = 1/(1 + C^2) = 0.1, settings the paper chose for the proof. The root cause is that Adam's effective learning rate α/vt\alpha/\sqrt{v_t} can grow. Once the recent gradients are small, the large gradients of the past are forgotten.

Their fix, AMSGrad, changes only one thing. The denominator uses the largest vv seen so far:

v^t=max⁡(v^t−1, vt)\hat v_t = \max(\hat v_{t-1},\ v_t)

From then on, the effective learning rate can only shrink, never grow. In the example above, all three steps use the same v^\hat v, and the net effect over one cycle becomes a move of 0.35 in the right direction.

Worse generalization?

Another challenge came from Wilson et al.'s 2017 “The Marginal Value of Adaptive Gradient Methods in Machine Learning”. They constructed a linear classification problem on which the solution SGD finds has a test error of 0, while the solutions found by AdaGrad, RMSProp, and Adam put every new sample into the same class.

On CIFAR-10, tuning only the learning rate and its decay scheme, they got a test error of 7.65% for SGD, 9.60% for RMSProp, and 9.78% for Adam. The training loss went down fast, but the test results were worse. This paper kept a fair amount of computer vision research on SGD with momentum.

Two years later, Choi et al. reran the experiments with Wilson et al.'s public code, tuning momentum and ϵ\epsilon as well, and the test errors of the four methods came out nearly the same. How large the gap is depends to a great extent on which hyperparameters get tuned.

Weight decay gets shrunk

The third problem is the hardest to spot, and it has had the biggest impact.

Weight decay pulls the parameters a little toward 0 at every step, keeping them from growing too large. In SGD, it's equivalent to L2 regularization. Add λ2∥θ∥2\frac{\lambda}{2}\lVert\theta\rVert^2 to the loss, and the gradient gains an extra term λθ\lambda\theta, whose effect is to shrink the parameters a little each step.

So early Adam implementations simply added λθ\lambda\theta to the gradient. But inside Adam, this term also gets divided by v^\sqrt{\hat v}. For a parameter whose gradients are always large, v^\sqrt{\hat v} is large, and the decay is shrunk to very little; in Loshchilov and Hutter's words, weights with larger past gradients get regularized less.

Their fix is to take the decay out so it doesn't go through Adam's denominator:

θt=θt−1−ηt(α m^tv^t+ϵ+λ θt−1)\theta_t = \theta_{t-1} - \eta_t \left( \alpha\, \frac{\hat m_t}{\sqrt{\hat v_t} + \epsilon} + \lambda\, \theta_{t-1} \right)

This is AdamW. In the formula, α\alpha is the learning rate, and ηt\eta_t is the multiplier the learning-rate schedule gives at step tt, which is 1 when the learning rate is fixed.

Weight decay inside Adam, or outside itTwo weights start at 1 with zero-mean noisy gradients: w₁'s noise has size 1, w₂'s size 0.001. α = 0.002, λ = 0.004. Only the regularizer pulls them toward 0.
L2 inside Adam
025050000.51stepww₁ (large gradients)w₂ (small gradients)
AdamW (decoupled)
025050000.51stepww₁ (large gradients)w₂ (small gradients)
L2 inside Adam: w₁ = 0.96, w₂ = 0.28AdamW (decoupled): w₁ = 0.13, w₂ = 0.14

Both weights start from 1, their gradients are pure noise, and the sizes of the two noises differ by a factor of a thousand. α=0.002\alpha = 0.002, λ=0.004\lambda = 0.004. In the version that adds L2 to the gradient, w1w_1, with the large gradients, is hardly decayed at all and is still at 0.96 after 500 steps; w2w_2, with the small gradients, is instead pulled down by about 0.002 per step and falls to 0.28 by step 500. AdamW treats the two weights the same, multiplying each by 0.996 at every step, and by step 500 both are down to around 0.13.

In the CIFAR-10 experiments, the good results for Adam with L2 lie along a diagonal line in learning rate and λ\lambda, so tuning one means retuning the other; AdamW's good region runs roughly parallel to the two axes, so the two hyperparameters can be tuned separately, and its best result also caught up with SGD. Today, AdamW is practically the default for training Transformers and large language models.

Learning-rate schedules

We said earlier that SGD's learning rate has to shrink gradually. But shrink how?

From steps to cosine

The earliest and most common is the step schedule, which divides by 10 every so often. The classic recipe for training ResNet-50 is 90 epochs (one complete pass over the whole training set is called an epoch), dividing by 10 at epochs 30, 60, and 80. Its drawback is that you only get a good model after the last division, and you have to decide for yourself when to divide.

In 2016, SGDR switched to a cosine curve that decreases smoothly from the maximum to the minimum:

ηt=ηmin⁡+12(ηmax⁡−ηmin⁡)(1+cos⁡TcurTiπ)\eta_t = \eta_{\min} + \frac{1}{2}\left(\eta_{\max} - \eta_{\min}\right)\left(1 + \cos\frac{T_{\text{cur}}}{T_i}\pi\right)

TcurT_{\text{cur}} is how long the current cycle has been running, and TiT_i is the length of this cycle. At the end of a cycle, the learning rate jumps straight back to the maximum and the next cycle begins, which is called a warm restart; the next cycle's length can be multiplied by TmultT_{\text{mult}}, so the cycles get longer and longer. It was proposed by none other than Loshchilov and Hutter, who later wrote AdamW.

Five learning-rate schedulesHeight is η divided by its peak. Step: Goyal et al.'s 90-epoch recipe. SGDR: T₀ = 10, T_mult = 2 over 150 epochs. Warmup and WSD warm up over the first 5%; WSD decays over the last 10%.
Step
100%100%
Cosine
100%100%
SGDR
100%100%
Warmup + cosine
100%100%
WSD
100%100%
Step 0Cosine 0SGDR 0Warmup + cosine 0WSD 0

The figure shows five schedules, with the cursor moving in sync. The SGDR panel uses T0=10T_0 = 10 and Tmult=2T_{\text{mult}} = 2, restarting at epochs 10, 30, and 70.

Restarts never really caught on. In SGDR's own experiments, the best single model for WRN-28-10 on CIFAR-10, at 3.86%, came from a cosine with no restarts at all, just one cycle. The training setups of GPT-3 and LLaMA later used a single-cycle cosine as well. What stuck was the smooth curve, at the cost of having to know the total number of steps in advance once again; the experiments in Chinchilla (DeepMind's paper on how model size and the amount of training data should be matched) found that if the cosine cycle is longer than the actual number of training steps by 25% or more, the results get noticeably worse.

Without fixing the finish line in advance

A cosine schedule needs to know the total training length up front. But when training large language models, people often decide halfway through whether to train a bit longer.

In 2024, MiniCPM used a three-phase schedule called WSD (Warmup-Stable-Decay). It starts with warmup (covered in the next section), then holds the learning rate constant for a long time, and lowers it only over roughly the last tenth of training. Any checkpoint (a model saved partway through training) from before the decay can be picked up and trained further at any time.

In 2024, Defazio et al. went even further and did away with the schedule altogether. Schedule-Free maintains three sets of parameters at once:

yt=(1−β) zt+β xtzt+1=zt−γ ∇f(yt)xt+1=(1−1t+1)xt+1t+1 zt+1\begin{aligned} y_t &= (1-\beta)\, z_t + \beta\, x_t \\ z_{t+1} &= z_t - \gamma\, \nabla f(y_t) \\ x_{t+1} &= \left(1 - \tfrac{1}{t+1}\right) x_t + \tfrac{1}{t+1}\, z_{t+1} \end{aligned}

Here xx, yy, and zz are three sets of parameters, unrelated to the parcel weight xx. zz is plain SGD with a fixed step size γ\gamma; xx is the average of all the zz's, and it's also the model you end up using; the gradient is computed at yy, which sits between the two. Averaging the past zz's works like a linear decay whose endpoint keeps moving along with the current step count.

Schedule-Free: three points, one averageMini-batches of 4, γ = 0.3, β = 0.9. z takes the SGD steps, x is the running average of every z so far, and the gradient is taken at y = 0.1·z + 0.9·x. Early on x trails behind, because it still averages in the first, far-off z.
012-1012w (per kg)b (base fee)zxy
x (average): 0.007 from the minimumz (SGD steps): 0.461 from the minimumy (where the gradient is taken)

Each step draws 4 parcels, with γ=0.3\gamma = 0.3 and β=0.9\beta = 0.9. For the first hundred-odd steps, xx clearly lags behind zz, because it gives every zz equal weight, including the earliest zz's, far from the minimum; the longer it runs, the steadier xx gets.

In the 2024 AlgoPerf training-algorithm competition, Schedule-Free AdamW won the track that forbids external hyperparameter tuning, averaging about 8% faster than the baseline, and it was also the only submission in that track to beat the baseline. It still needs warmup, though. Most of the paper's experiments set warmup to 5% of total training, so the total length still comes in through the warmup.

Warmup

These days, almost every training config specifies that over the first few thousand steps, the learning rate should climb slowly from something very small. This is called warmup. Why can't you just use a large learning rate from the start?

Linear scaling and warmup

Warmup was first written up as a recipe in Goyal et al.'s 2017 “Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour”. They wanted to grow the batch from 256 to 8,192 and train ResNet-50 in one hour on 256 GPUs.

The first rule is linear scaling. When the batch grows kk-fold, the learning rate grows kk-fold too, so the learning rate for batch 8,192 is 3.2. But the network changes very fast at the start of training, and using such a large learning rate right away wrecks the model. So over the first 5 epochs, the learning rate rises linearly from the original 0.1 to 3.2.

Without warmup, the top-1 error (the fraction of images where the first guess is wrong) is 24.84%; warming up with a fixed small learning rate is actually worse, at 25.88%; gradual warmup gets 23.74%, catching up with the 23.60% of batch 256. Up to a batch of 8,192, the error barely changes; it starts getting worse at 16k, reaching 27.55% at 32k and 33.96% at 64k.

Why Adam needs warmup too

Adam's step size is roughly α\alpha, so in principle it shouldn't be afraid of large gradients. Why, then, does Adam without warmup still often break right at the start when training Transformers?

RAdam's authors, Liu et al., offered an explanation in 2019. At the start of training, vv has seen only a few gradients, so its estimate of “how large the gradient is” is very unreliable. The first step is even more extreme. v^1=g12\hat v_1 = g_1^2, so the adaptive learning rate ψ=1/v^1=1/∣g1∣\psi = 1/\sqrt{\hat v_1} = 1/|g_1| is estimated from a single sample, and its variance actually diverges.

The first few adaptive learning rates are unreliable2,000 runs with gradients drawn from N(0, 1), β₂ = 0.999. Bands: middle 50% and 90% of ψ = 1/√v̂ at each step; line: median; ▲: at least one run above 100. Both axes are logarithmic.
12481632641282560.3110100tψ = 1/√v̂first 4 steps: momentum only1/σ = 1▲
90% of runs: 0.93 to 1.1largest: 1.2ρt = 245.1RAdam rectification rₜ = 0.346

The figure shows 2,000 simulations, with gradients drawn from a standard normal distribution and β2=0.999\beta_2 = 0.999. In the simulations, 90% of the values of 1/v^1/\sqrt{\hat v} at step 1 fall between 0.50 and 14.4, a spread of nearly thirty times; by step 10, they've already narrowed to between 0.73 and 1.6.

RAdam estimates how large this variance is and multiplies by a rectification term rtr_t, skipping the adaptive learning rate altogether in the first few steps, while the variance is too large. With β2=0.999\beta_2 = 0.999, the first 4 steps use momentum only, and after that rtr_t climbs slowly from very small to 1, which works like a built-in warmup.

In October of the same year, Ma and Yarats pointed out that rtr_t depends only on the step count tt and on β2\beta_2, not on the data. So RAdam is roughly a fixed-length warmup, and their rule of thumb is 2/(1−β2)2/(1-\beta_2) steps, which comes to 2,000 steps for β2=0.999\beta_2 = 0.999. Comparing the two with ResNet-50 on ImageNet, they found that a linear warmup of exactly this length comes within half a percentage point of RAdam.

When batches get bigger: LARS and LAMB

Goyal et al.'s recipe is very stable up to batch 8,192, but beyond that it can't hold up. The error rate is 24.79% at 16k, 27.55% at 32k, and 33.96% at 64k. What goes wrong as the batch keeps growing?

One explanation is that large batches find sharper minima. Keskar et al. observed in 2017 that models trained with a batch of 10% of the training set end up in sharper minima and have lower test accuracy. This explanation was later challenged. Dinh et al. proved that the sharpness of a ReLU network can be inflated arbitrarily without changing the function it computes; Hoffer et al. found that once the learning rate is scaled up with the square root of the batch size and the number of training steps is increased to compensate, the gap disappears.

The other problem is more direct. LARS's authors, You, Gitman, and Ginsburg, measured the weight norm and gradient norm of each layer of AlexNet-BN at the first step. The ratio of the two, ∥w∥/∥∇L∥\lVert w\rVert/\lVert\nabla L\rVert, is 5.76 in the first convolutional layer and 1,345 in the first fully connected layer, more than two hundred times apart.

One learning rate, layers that move by very different amountsBars show how much one step changes each layer, ‖Δw‖/‖w‖, on a log scale. ‖w‖ and ‖∇L‖ are AlexNet-BN's first iteration (LARS paper, Table 2). The animation raises the global learning rate γ; LARS uses trust coefficient η = 0.001.
One global rate (SGD)
10⁻⁵10⁻⁴10⁻³10⁻²10⁻¹110¹update = weightconv1.w174%fc6.w0.74%
Per-layer trust ratio (LARS)
10⁻⁵10⁻⁴10⁻³10⁻²10⁻¹110¹update = weightconv1.w1.00%fc6.w1.00%
‖w‖/‖∇L‖: conv1.w 5.76, fc6.w 1345With one global rate, conv1.w moves 234× more than fc6.w

Under the same global learning rate, the relative change per step in the first convolutional layer is 234 times that of the fully connected layer. Once the learning rate passes a certain point, the first layer's update becomes larger than its weights, and training falls apart; yet for the fully connected layer, that learning rate is still tiny.

LARS computes a learning rate λl\lambda^{l} of its own for each layer, making the update proportional to the size of that layer's weights:

λl=τ ∥wl∥∥∇L(wl)∥\lambda^{l} = \tau\, \frac{\lVert w^{l}\rVert}{\lVert\nabla L(w^{l})\rVert}

τ\tau is a small trust coefficient, which the paper writes as η\eta and sets to 0.001; we use a different letter here so it isn't confused with the learning rate. This way every layer's relative change is equally large, and the most sensitive layer no longer holds the rest back. For AlexNet at batch 8k, without LARS and with warmup alone, the learning rate can't be scaled up no matter how it's tuned, and the best top-1 accuracy is 44.8%; with LARS it's back to 58.2%, close to the 58.7% of batch 512.

LAMB applies the same layer-wise scaling to Adam. It first computes this layer's update rr the Adam way, adds the weight decay λx\lambda x, and then rescales the length of the whole update to be proportional to the length of this layer's weights:

x(l)←x(l)−η ϕ(∥x(l)∥)∥r(l)+λx(l)∥ (r(l)+λx(l))x^{(l)} \leftarrow x^{(l)} - \eta\, \frac{\phi\big(\lVert x^{(l)}\rVert\big)}{\lVert r^{(l)} + \lambda x^{(l)}\rVert}\,\big(r^{(l)} + \lambda x^{(l)}\big)

x(l)x^{(l)} is the weight of layer ll, and ϕ\phi is a function that keeps the weight norm between a lower and an upper bound. In 2019, You et al. used it to cut BERT-Large pre-training from 81.4 hours on 16 TPUv3 chips to 76 minutes on 1,024 TPUv3 chips; the batch was 64k in the first phase and 32k in the second.

The story has a sequel. In 2021, Nado et al. retuned Adam and fixed two problems in BERT's open-source code, the missing bias correction and a cliff in the learning-rate schedule; with the same batch size and number of steps, Adam matched LAMB or even did slightly better. Well-tuned Nesterov momentum also matched LARS on the ResNet-50 setup of MLPerf (an industry benchmark for training speed).

Finding a flat valley floor: SAM

Back to Keskar et al.'s observation. Are sharp minima really worse?

All of the methods so far answer “how to get to the valley floor faster”. In 2020, Foret et al. asked a different question. Which valley floor is better?

Picture two valleys, one deep and narrow, the other shallow and wide. On the training loss, the narrow one is lower; but test data always differs a bit from training data, and when the parameters shift slightly, the narrow valley's loss shoots up while the wide valley's barely changes.

SAM (Sharpness-Aware Minimization) changes the objective to “the worst case nearby”:

min⁡w max⁡∥ϵ∥≤ρ L(w+ϵ)\min_{w}\ \max_{\lVert\epsilon\rVert \le \rho}\ L(w + \epsilon)

Here ww stands for all of the model's parameters; ϵ\epsilon is a perturbation of the parameters, not the ϵ\epsilon from earlier that prevents division by zero; ρ\rho is the radius of the perturbation, also unrelated to RMSProp's ρ\rho. Within radius ρ\rho, find the point with the highest loss, then make the loss at that point smaller. The inner max uses a first-order approximation, under which the worst direction is the gradient direction, ϵ^=ρ ∇L/∥∇L∥\hat\epsilon = \rho\, \nabla L / \lVert\nabla L\rVert. So each step first pushes uphill by ϵ^\hat\epsilon, computes the gradient there, and then brings that gradient back to the original position to do the update.

Compare valleys by their worst loss within ρSolid: the loss L. Dashed: the worst loss within distance ρ, which SAM minimizes. The animation widens ρ; the arrow at each minimum points to its worst neighbor.
0123400.511.5Lnarrow valleywide valley
L at the bottom: narrow 0.33, wide 0.58Worst within ρ: narrow 1.36, wide 0.80By this measure the wide valley wins

The solid line is the original loss, and the dashed line is the worst loss within ρ\rho, which is what SAM actually pushes down. At ρ=0\rho = 0 the two lines coincide, and the bottom of the narrow valley is at 0.33, lower than the wide valley's 0.58. Raise ρ\rho to 0.1, and the worst loss around the narrow valley rises to 0.63, while the wide valley's is only 0.60, so the ranking flips; at ρ=0.45\rho = 0.45, the gap widens to 1.36 versus 0.80. What SAM changes is the standard by which valley floors are compared.

The cost is two gradient computations per step. On ImageNet, the best top-1 error for ResNet-152 with standard training is 20.3%, while SAM gets 18.4%; the effect is even more pronounced under heavy label noise. Later, Andriushchenko and Flammarion found that it only works when the perturbation is computed separately on each mini-batch, and the full-data version that the paper's generalization bound describes barely helps. The relationship between sharpness and generalization is still being debated today.

Not enough memory: Adafactor and GaLore

Adam has to store two extra numbers for every parameter, mm and vv. With a lot of parameters, these two copies of state get very large.

Training memory of a 7B model, before activationsComputed from LLaMA-7B shapes (32 layers, d = 4096, FFN 11008, vocabulary 32000), everything in fp32, 1 GB = 10⁹ bytes. Adafactor keeps no first moment; GaLore projects only the attention and MLP matrices (r = 1024) and uses Adam for the rest; gradients are counted at full size.
04080120160GBAdamWAdafactorGaLore (r = 1024)107.8 GB53.9 GB72.7 GB
parametersgradientsoptimizer stateOptimizer state: AdamW 53.9 GB, Adafactor 0.011 GB, GaLore 18.8 GB

Taking LLaMA-7B's shapes, with all 6.7 billion parameters stored in fp32, the parameters themselves take 27 GB, the gradients 27 GB, and AdamW's two states another 54 GB, for a total of about 108 GB, and that's before counting the intermediate results the forward pass leaves behind (the activations).

Keep only the row and column sums

Adafactor's authors, Shazeer and Stern, set their sights on vv. For an n×mn \times m weight matrix, vv is also n×mn \times m. They store only the sum of each row, RR, and the sum of each column, CC, and piece the matrix back together when it's needed:

V^=R CΣR\hat V = \frac{R\, C}{\Sigma R}

ΣR\Sigma R adds all the row sums together, which gives the total of the whole matrix. This amounts to approximating the original vv with a rank-1 matrix; rank 1 means every row is just a multiple of the same vector. Storage drops from nmnm to n+mn + m.

Keep the row and column sums, rebuild the restA 6×8 second-moment matrix V. Adafactor stores R = V·1 (row sums) and C = 1ᵀ·V (column sums) and uses V̂ = R·C / ΣR in place of V.
V (full)RCV̂ = R·C / ΣR (rebuilt)
Stored numbers: 48 → 14Mean relative error of V̂: 21%

A 6×8 matrix originally takes 48 numbers to store; now it takes only 14. The rebuilt matrix is of course not exactly the same, and in this example the mean relative error is 21%, but vv only decides roughly how large each parameter's step is, so errors of this size don't matter much. On WMT'14 English-to-German translation, the BLEU (a translation-quality score, higher is better) before and after factorization is 25.6 versus 25.4.

Adafactor also drops momentum, not even storing mm, and that's where the trouble started. Without momentum and without warmup, BLEU crashes to 0.1, a complete training failure. Tracking it down, they found that with β2=0.999\beta_2 = 0.999, vv can't keep up with changes in the gradient, and around step 800 the updates spike to more than twice their normal size. The fix is update clipping, which caps the root mean square of the whole update at 1, together with letting β2\beta_2 rise slowly with the step count. T5 used Adafactor; PaLM used it too, but kept momentum and dropped even the factorization.

Running Adam in a small space

GaLore, from 2024, came at it from a different angle. Its authors, Zhao et al., noticed that the gradients of weight matrices tend to concentrate in a few directions, that is, they're low-rank. So project the gradient into a small space first, run Adam in that small space, and project back:

Rt=Pt⊤Gt,G~t=α Pt NtR_t = P_t^\top G_t, \qquad \tilde G_t = \alpha\, P_t\, N_t

PtP_t comes from the singular value decomposition of the gradient (splitting a matrix into a few directions, each with its own amount of stretching), keeping the top rr most important directions; NtN_t is the update Adam computes on the small matrix RtR_t, and α\alpha is GaLore's own scaling factor. Adam's state then only needs to store r×nr \times n, instead of m×nm \times n. Every TT steps, a new singular value decomposition is computed to switch to a new subspace.

Run Adam in a small subspace, then project backA 6×10 gradient G whose main directions drift. Every 10 steps P is recomputed from the top 2 left singular vectors of G; Adam's two moments live on R = PᵀG. The line chart shows how much of ‖G‖² the subspace keeps.
GPR = PᵀGAdam updates hereα·P·N back to 6×100%50%100%010203040share of ‖G‖² kept
Optimizer state: full Adam 120 numbers, GaLore 52 (P plus two r×n moments)Kept now: 99%

Unlike LoRA (a fine-tuning method that trains only a pair of small low-rank matrices), GaLore keeps the weights themselves full-rank; only the optimizer state is constrained. Pre-training LLaMA 1B on C4, GaLore reaches a perplexity (how uncertain a language model is about the next token, lower is better; a token is the basic unit in which a model reads text) of 15.64, against 15.56 for full-rank training and 19.21 for LoRA.

The paper's abstract says the 8-bit version can pre-train a 7B model on a 24 GB consumer GPU like the RTX 4090. That memory figure, about 22 GB, was measured with only 256 tokens per batch; the 7B experiments that were actually run to completion used 64 A100s. The cost is a singular value decomposition every time the subspace switches. On the 1B model, the paper estimates it's about 17% slower than 8-bit Adam; for a 7B model, a single singular value decomposition takes about 10 minutes in LLaMA-Factory's measurements, and up to 20 minutes according to the GaLore team's later GaLore 2 paper.

Making the valley round: Newton's method, Shampoo, SOAP, Sophia

Back to the question of why Adam wasn't much faster. In the same valley, GD arrives at step 93 and Adam at step 87, only a tiny bit faster.

Newton's method

If you know the shape of the terrain, you can make the valley round directly. Newton's method corrects the gradient with the inverse of the Hessian:

θt+1=θt−H−1∇L(θt)\theta_{t+1} = \theta_t - H^{-1} \nabla L(\theta_t)

On a quadratic, this one step goes straight to the minimum. Seen another way, multiplying by H−1H^{-1} amounts to switching to a new set of coordinates in which the valley is round, so the gradient points straight at the bottom. This approach of “making the terrain round first, then walking” is called preconditioning.

Make the valley round by changing coordinatesThe parcel valley, centered on its minimum. Gradient descent (η = 0.9) runs in each coordinate system, then the picture morphs to the next: original, each axis rescaled by its own curvature (diagonal), and rescaled by the full matrix H^½.
full matrix: straight in
κ = 1.0original: κ = 45.7diagonal scaling: κ = 45.6full matrix: κ = 1.0

This figure morphs between three coordinate systems. The original coordinates have a condition number of 45.7. Scaling only along each coordinate axis separately, which is diagonal preconditioning, gives a condition number of 45.6, almost unchanged, because the Hessian's two diagonal entries were nearly equal to begin with, 1.09 and 1. Only when the full Hessian is used to change coordinates does the condition number become 1.

This is why Adam wasn't much faster in our valley. Adam, RMSProp, and AdaGrad all decide the step size for each parameter separately, which amounts to scaling only along the coordinate axes; our valley's whole problem is that it's “tilted”, and scaling along the axes can't fix that.

So why not just use Newton's method? For a model with nn parameters, the Hessian is n×nn \times n. A billion parameters means storing 101810^{18} numbers, and then inverting the matrix. And when the terrain isn't convex (a convex function has just one valley, with no hilltops or saddle points), the Hessian can have negative eigenvalues, and Newton's method will head straight for a hilltop.

Shampoo and SOAP

Shampoo, by Gupta, Koren, and Singer, found a compromise. Most of a neural network's parameters are matrices, and for an m×nm \times n weight matrix, a full preconditioner needs mn×mnmn \times mn. Shampoo instead uses two small matrices, one on the left and one on the right, which accumulate GG⊤GG^\top and G⊤GG^\top G respectively:

Lt=Lt−1+GtGt⊤Rt=Rt−1+Gt⊤GtWt+1=Wt−η Lt−1/4 Gt Rt−1/4\begin{aligned} L_t &= L_{t-1} + G_t G_t^\top \\ R_t &= R_{t-1} + G_t^\top G_t \\ W_{t+1} &= W_t - \eta\, L_t^{-1/4}\, G_t\, R_t^{-1/4} \end{aligned}

Lt−1/4L_t^{-1/4} is the matrix raised to the −1/4-1/4 power, computed by first doing an eigendecomposition of LtL_t and then replacing each eigenvalue with its −1/4-1/4 power. For a 1,024×1,024 layer, the full preconditioner has about 1.1×10121.1 \times 10^{12} entries, while Shampoo needs only about 2.1×1062.1 \times 10^{6}. Meta's later Distributed Shampoo added many engineering changes and won the track that allows external hyperparameter tuning at AlgoPerf 2024, averaging about 28% faster than the NAdamW baseline; still, it was the fastest on only two of the eight workloads, and on ResNet it didn't reach the target within the time budget.

Shampoo recomputes the matrix roots only every so often, and the preconditioner stays completely frozen in between. SOAP's authors, Vyas et al., found that Shampoo with an exponent of 1/2 is equivalent to running Adafactor in Shampoo's eigenbasis. So SOAP first rotates the gradient into Shampoo's eigenbasis, runs Adam in that coordinate system, and then rotates the update back to the original coordinates. The basis is still updated only once in a while, but the second moment is updated at every step. On a 360-million-parameter language model, relaxing the recomputation from every step to every 100 steps raises Shampoo's final loss from 2.824 to 2.932, already close to AdamW's 2.943; SOAP only goes from 2.820 to 2.870.

Sophia

Sophia takes the diagonal route, but it looks at the diagonal of the Hessian rather than at squared gradients. Every 10 steps it estimates the diagonal Hessian hh, then clips coordinate by coordinate:

θt+1=θt−η clip ⁣(mtmax⁡(γht, ϵ), 1)\theta_{t+1} = \theta_t - \eta\, \mathrm{clip}\!\left(\frac{m_t}{\max(\gamma h_t,\ \epsilon)},\ 1\right)

When the curvature is positive and large enough, it steps in Newton's proportions; when the curvature is negative or too small, ϵ\epsilon props up the denominator, and the clipping keeps the step to at most η\eta.

Where the curvature turns negativef(x) = x⁴/4 − x² + 0.6x + 2, starting at x = 0.1 where f″ < 0. Newton's step heads for the hilltop; Sophia clips m / max(γh, ε) to [−1, 1]. Toy settings: β₁ = β₂ = 0.9, the exact f″ at every step, γ = 1, η = 0.15 (the paper uses β₁ = 0.96, β₂ = 0.99 and a Hessian estimate every 10 steps).
GD (η = 0.1)
f″ < 0
Newton's method
f″ < 0
Sophia-style clipping
f″ < 0
GD (η = 0.1): x = -1.55Newton's method: x = 0.32Sophia-style clipping: x = -1.49

This one-dimensional function has negative curvature near the starting point. Newton's method converges to the local maximum at x=0.316x = 0.316, heading for the hilltop; Sophia moves a fixed 0.15 per step in the negative-curvature region, walking downhill to near the valley floor. GD, in the leftmost panel, looks only at the slope and also keeps walking downhill, stopping near −1.55-1.55; it never uses curvature at all, so negative curvature can't fool it, but its step size doesn't adapt to the curvature either.

The paper's authors, Liu et al., claimed on GPT-2 that reaching the same loss takes only half as many steps as AdamW. None of the later independent evaluations saw a gap this large. Defazio pointed out on GitHub that once AdamW's learning rate was raised, the two tied on a 125-million-parameter model; the 2025 “Fantastic Pretraining Optimizers”, whose authors include three of Sophia's original authors, also measured no significant speedup below 500 million parameters.

Only the sign: Lion

Back to memory. Adam has to store two states, mm and vv, for every parameter. Could we store one fewer?

Lion was found by Chen et al. at Google through program search. They had a computer evolve programs in a space built from basic operations, and the rule that came out of it is this:

ct=β1 mt−1+(1−β1) gtθt=θt−1−η (sign(ct)+λ θt−1)mt=β2 mt−1+(1−β2) gt\begin{aligned} c_t &= \beta_1\, m_{t-1} + (1 - \beta_1)\, g_t \\ \theta_t &= \theta_{t-1} - \eta\,\big(\mathrm{sign}(c_t) + \lambda\, \theta_{t-1}\big) \\ m_t &= \beta_2\, m_{t-1} + (1 - \beta_2)\, g_t \end{aligned}

The update direction keeps only the sign, so every parameter moves exactly η\eta at every step. It stores only one momentum, mm, one state fewer than Adam. β1=0.9\beta_1 = 0.9 takes care of this step's direction, and β2=0.99\beta_2 = 0.99 takes care of remembering the long-term momentum, so the two roles are split apart. Because the size of each step is fixed, Lion's learning rate has to be 3 to 10 times smaller than AdamW's, and its weight decay 3 to 10 times larger.

Lion keeps only the signBoth start from NT$210 per kg plus a base fee of NT$20, with a cosine schedule over 200 steps (peak rate: Adam 0.2, Lion 0.05). Every Lion step moves each coordinate by exactly ±η.
012-1012w (per kg)b (base fee)
First 40 steps, close up
Adam: 0.022 from the minimumLion: 0.023 from the minimumRates now: Adam 0.000, Lion 0.0000

Adam and Lion both run 200 steps with a cosine schedule, and both end up about 0.02 from the minimum. Look at the “First 40 steps, close up” panel. At first, the two gradient components have the same sign, so Lion moves the same distance in ww and bb at every step, advancing along a 45° diagonal.

For ViT pre-trained on JFT, Lion catches up with AdamW using about a third of the compute; but on language models from 1.1 billion to 7.5 billion parameters, its perplexity was no different from Adafactor's from start to finish. Most of the later independent evaluations found it roughly on par with AdamW.

A different ruler

The sign operation looks crude, but there's a clean explanation behind it.

“The steepest direction of descent” depends on how you measure length. The length we normally use is the L2 norm, and on the unit circle, the direction that lowers the loss the most is −g/∥g∥-g/\lVert g\rVert. Switch to the L∞ norm, which looks at the largest coordinate, and the unit “circle” becomes a square; the steepest direction then lands on a corner, which is exactly −sign(g)-\mathrm{sign}(g).

The steepest step depends on how you measure lengthAmong steps of length at most 1, take the one that lowers the loss fastest, ⟨g, d⟩ smallest. With the usual length (a circle) it is −g/‖g‖. Measured by the largest coordinate (a square) it is always a corner, −sign(g).
L2 norm: circle
g
L∞ norm: square
g
g = (0.94, 0.34)L2: (-0.94, -0.34)L∞: (-1, -1)

Bernstein and Newhouse generalized this view in their 2024 “Old Optimizer, New Norm”. With its exponential moving averages switched off, Adam is steepest descent under the L∞ norm; Shampoo, with its accumulation removed, is exactly steepest descent under the spectral norm. The spectral norm measures the most a matrix can stretch a vector, and that's the starting point for Muon in the next section.

Treating weights as matrices: Muon

Apart from the Shampoo line, most methods decide step sizes coordinate by coordinate, which amounts to flattening the parameters into one long list of numbers and treating each one separately. But most of a neural network's weights are matrices.

The gradient of a weight matrix is often dominated by a few directions, with singular values of wildly different sizes. Updating straight from the gradient changes those few directions a lot, while the others barely move. Muon's approach is to “orthogonalize” the momentum matrix. Take the singular value decomposition of the momentum matrix MM, M=UΣV⊤M = U\Sigma V^\top, replace all the singular values with 1, and keep only UV⊤UV^\top. Every direction then moves by the same amount.

Starting in October 2024, Keller Jordan used Muon in the NanoGPT training speedrun and set several records, and only wrote it up as a blog post in December. It's applied only to the two-dimensional weight matrices in the middle of the network; the embedding layer, the output layer, and the vector parameters still use AdamW.

An exact singular value decomposition is too expensive, so Muon approximates it with Newton–Schulz iteration. First divide the matrix by its Frobenius norm so that all the singular values fall between 0 and 1, then repeat five times:

X←aX+b (XX⊤)X+c (XX⊤)2XX \leftarrow aX + b\,(XX^\top)X + c\,(XX^\top)^2 X

The coefficients are (a,b,c)=(3.4445, −4.7750, 2.0315)(a, b, c) = (3.4445,\ -4.7750,\ 2.0315).

This polynomial keeps pushing small singular values up and pulling large ones down.

Five Newton–Schulz iterationsA 2×2 update matrix, first divided by its Frobenius norm, then iterated with X ← aX + b(XXᵀ)X + c(XXᵀ)²X, (a, b, c) = (3.4445, −4.7750, 2.0315). Left: what the matrix does to the unit circle. Right: its two singular values.
Unit circle after X
Singular values
≈0.68–1.2 after five steps (our calculation)00.511.5σ₁σ₂
Original matrix: σ₁ = 2.05, σ₂ = 0.083σ₁ = 0.940, σ₂ = 0.697

This 2×2 matrix originally squashes the unit circle into a flat ellipse, with singular values of 2.05 and 0.083. After dividing by the norm, they're 0.999 and 0.040. After five steps they become 0.94 and 0.70, and the ellipse is close to a circle, though not exactly one. Jordan states in a code comment that these coefficients only aim to push the singular values to roughly between 0.5 and 1.5, trading this imprecision for fewer iterations; by our own calculation, after five steps they actually land at about 0.68 to 1.2.

In 2025, Moonshot AI's “Muon is Scalable for LLM Training” took it to large models. They added weight decay and multiplied each matrix's update by 0.2max⁡(r,c)0.2\sqrt{\max(r, c)} (where rr and cc are the matrix's numbers of rows and columns), which fixes the update's root mean square at around 0.2 to match the typical size of AdamW's, so AdamW's learning rate and weight decay can be reused directly. In their scaling law experiments, Muon reached the same loss with only about 52% of AdamW's compute. Later, Moonshot's Kimi K2 and DeepSeek's V4 both used Muon for most of their parameters.

Which one is fastest?

Every new optimizer paper includes a plot showing that it's faster than AdamW. Which of these hold up?

AlgoPerf tries to answer this with rules. What it measures is “how much time it takes to reach a fixed target on fixed hardware”. There are 8 workloads in total, always run on 8 V100s, split into two tracks; one allows limited external hyperparameter tuning, and the other doesn't, but gets three times the time budget. In the first round, in 2024, Distributed Shampoo and Schedule-Free AdamW each won one track, as mentioned earlier; in the track that allows external tuning, the same team's Schedule-Free entry also took second place.

On the language model pre-training side, Wen, Hall, Ma, and Liang carefully tuned the hyperparameters of AdamW and ten newer optimizers in their 2025 “Fantastic Pretraining Optimizers and Where to Find Them”. The first thing they found is that just raising AdamW's learning rate in the GPT-3 recipe from 6e-4 to 8e-3 made a 130-million-parameter model about 2 times faster. Many of the 2× speedups claimed in papers were measured against a poorly tuned AdamW.

With a well-tuned baseline, matrix-based optimizers like Muon and SOAP are about 1.3 to 1.4 times faster at 130 million to 520 million parameters, dropping to about 1.1 times at 1.2 billion; coordinate-wise optimizers like Lion and Sophia come out about even with AdamW. They also warn that the loss partway through training can be misleading. Muon often leads by a wide margin early on, but once the learning rate comes down, it ends up level with AdamW.

Larger scales are another story. NVIDIA's July 2026 “SOAP, Muon, and Beyond” compared models with billions of parameters trained on trillions of tokens, and still measured SOAP and Muon beating AdamW. Their baseline batch was about 25 million tokens per step; scale it up two or three times, and AdamW gets worse while Muon stays stable. The original SOAP also runs into sudden loss spikes at large batch sizes, and only holds up after switching to a version that re-orthogonalizes the basis at every step. The two studies differ in model size, amount of data, and batch size, so both results can hold at the same time.

Wrapping up

Ever since Cauchy's three pages in 1847, the core of gradient descent has been to walk along the negative gradient. Today it's usually written as this one line:

θt+1=θt−η ∇L(θt)\theta_{t+1} = \theta_t - \eta\, \nabla L(\theta_t)

Every later method patches a shortcoming somewhere in it:

  1. SGD, MBGD: too much data, so look at only a small batch at a time; the price is noise, and the learning rate has to shrink gradually.
  2. SGDM, NAG: bouncing back and forth in the valley, so remember the recent direction and let the bounces cancel out; NAG also brakes a step early.
  3. AdaGrad, RMSProp, Adam: one step size shared by every parameter doesn't make sense, so let each parameter go its own way; RMSProp learns to forget, and Adam brings in momentum and bias correction as well.
  4. AMSGrad, AdamW: patch Adam's habit of forgetting large gradients, and the shrinking of weight decay.
  5. Cosine, WSD, Schedule-Free, warmup: how the learning rate should change over time, whether you can do without knowing the finish line in advance, and why you should speed up slowly at the start.
  6. LARS, LAMB: once batches get large, each layer's step size needs adjusting on its own.
  7. SAM: look not just for a low valley floor, but for a flat one.
  8. Adafactor, GaLore, Lion: too much state and not enough memory, so store less.
  9. Shampoo, SOAP, Sophia, Muon: estimate the shape of the terrain, or use a different ruler to measure how long a step is.

If you're training a model today, AdamW with warmup and a cosine or WSD schedule is still the safest place to start. If the model is big enough and has enough weight matrices, Muon and SOAP are worth a try. Whichever one you pick, tune the learning rate first.

The next time you see a new optimizer that claims to be twice as fast, start by asking, “Did you tune AdamW's learning rate?”