Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training
- Publication
- ICLR 2024Conference
- First submitted to arXiv
- arXiv:2305.14342
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{optimizerssophia2023,
title = {{Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training}},
author = {Hong Liu and Zhiyuan Li and David Hall and Percy Liang and Tengyu Ma},
year = {2023},
eprint = {2305.14342},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2305.14342}
}
When someone says an optimizer is “twice as fast as Adam,” how exactly is that measured?
Hong Liu, Zhiyuan Li, David Hall, Percy Liang and Tengyu Ma at Stanford spend a short section of this paper answering exactly that. The baseline gets its learning rate and schedule tuned to the best for a preset total of steps; the new method runs only steps, and its final loss must be less than or equal to the baseline's, with “about the same” not good enough. They also warn that a cosine schedule stretches with the total step count, so comparing against intermediate checkpoints from the -step run biases the conclusion.
By this standard, they claim Sophia reaches AdamW's validation loss on GPT-2 in half the steps, with nearly unchanged time and memory per step.
This note first looks at how Sophia works, then at what others later measured that “twice” to be.
Defining the problem
Adam only looks at gradient size
Adam's denominator is the square root of a moving average of squared gradients. When the gradient direction is steady, is roughly , so every coordinate takes a step close to , whether the landscape in that direction is steep or flat.
In a language model's loss landscape, though, curvature varies a lot from direction to direction. The authors measured the diagonal Hessian of a 125M-parameter GPT-2, and the magnitudes of its positive entries are spread very widely.
A two-dimensional example
The authors illustrate with a separable two-dimensional function, , where is steep and is flat.

Every line starts at in the upper left, heading for in the lower right.
GD's learning rate is capped by the steepest direction at roughly the inverse of the curvature there, beyond which it diverges. That learning rate is too small for the flat direction, so the dark red line crawls along .
A simplified Adam is SignGD, which moves in both directions every step. In the flat direction, a step that size lowers the loss only a little; in the steep direction, it reaches the valley floor in three steps and then starts bouncing back and forth. To make it settle, has to decay to 0, which slows the flat direction down even more. In the figure, red Adam and orange SignGD nearly overlap, both zigzagging along the valley.
By the convex-optimization textbook, the ideal preconditioner is the Hessian. In this separable example, Newton's method divides each direction's gradient by that direction's curvature, so steps shrink automatically in the steep direction and grow in the flat one. The trouble is that has a stretch of negative curvature, where Newton's method heads toward a maximum or a saddle point; the blue line converges to a saddle point. In real models, curvature also changes quickly along the training trajectory, so second-order information isn't always reliable.
Second-order methods are too expensive
Existing second-order methods also have a cost problem. Block-diagonal approximations like K-FAC are too expensive for large language models. AdaHessian also uses the diagonal Hessian as a preconditioner, but estimates it almost every step, at a per-step cost of more than two gradient computations. The authors suspect these methods need frequent estimates because they lack a safeguard against bad estimates.
So the preconditioner this paper is after has to do three things at once: scale by curvature as Newton does, stay safe under negative curvature and bad estimates, and add only a little compute per step on average.
Solving the problem
The update
Sophia stands for Second-order Clipped Stochastic Optimization. Algorithm 3 reads:
The numerator is a moving average of the gradient, as in Adam. The denominator becomes an estimate of the diagonal Hessian, computed only every steps and smoothed with a moving average to reduce noise. is decoupled weight decay. Everything is per coordinate, and clips each entry to .
The inside the parentheses is a discounted Newton step. The authors rewrite the expression as
The right-hand side is easier to read. Compute the Newton step , clip each coordinate to , then multiply by . So only decides what fraction of coordinates gets clipped, and every clipped coordinate ends up with a step of .
Walking through the clipping
Take Sophia-H's default , and suppose a few coordinates all have momentum and differ only in their curvature estimate :
| Curvature | Actual step | |
|---|---|---|
| 20 | 0.01 | |
| 1 | 0.2 | |
| 0.2 | 1 | |
| 0.02 | 10 | (clipped) |
| −3 | (clipped) |
Computed for this note from Eq. (6) of the paper.
Coordinates with large curvature automatically shrink to small Newton-style steps; coordinates with small curvature would get huge Newton steps and are clipped to . The last row is a negative value, which the Hutchinson estimator often produces. There picks , the ratio blows up, and it is clipped to , which is SignSGD with momentum. The same happens when curvature is underestimated.
The authors describe this behavior as Sophia using “stochastic momentum SignSGD as a backup.” However far off the Hessian estimate is, each coordinate moves at most per step, and that is what lets the whole method get away with estimating the Hessian only every 10 steps.
So how do you pick ? The authors suggest tuning it on a small model so that the fraction of unclipped coordinates falls between 10% and 50%; if the fraction is too high or too low, multiply by 0.5 or 2 and try again. Models of different sizes with the same architecture and data can share the value. In actual training, the clipped fraction climbs to around 60%, which means most coordinates are in fact taking sign steps. The paper ends up with (Sophia-H) and (Sophia-G).
Two diagonal Hessian estimators
The Hutchinson estimator (Sophia-H). Draw a standard normal vector and compute
Its expectation is exactly the diagonal of the Hessian. It needs just one Hessian-vector product, that is, one more backward pass through , without ever forming the Hessian. The estimate is unbiased, but any single estimate can be negative.
The Gauss-Newton-Bartlett estimator (Sophia-G). The Hessian of the cross-entropy loss splits into a Gauss-Newton term and a remainder that is usually smaller, and the authors estimate only the former. Here is the Jacobian of the logits with respect to the parameters, and is the second derivative of the loss with respect to the logits. For cross-entropy, depends only on the model's output probabilities , not on the true labels. Then, by the Bartlett identity from statistics, can be written as the expectation of “sample a label from the model's own distribution and take the outer product of the gradient.”
The implementation takes four steps:
- Compute logits on a batch of inputs;
- Sample labels from the model's own softmax;
- Compute the gradient of the mini-batch loss on these sampled labels;
- Return , where is the batch size.
The factor is there because automatic differentiation only gives the average gradient. Bartlett's first identity says that under sampled labels, each example's gradient has expectation 0, so the expected value of the squared average gradient times equals exactly the average of the per-example squared gradients.
This estimate is always non-negative, so the preconditioned direction is always a descent direction. It looks a lot like Adam's second moment, since both are squared gradients, but there are two differences. Adam uses gradients for the true labels, while Sophia-G uses labels sampled from the model; Adam takes the square root of the denominator, while Sophia divides directly. Squared gradients with true labels are known in the literature as the empirical Fisher, and the authors' ablation shows they do worse than GNB.
The official code implements only Sophia-G. The authors have also replied on GitHub that Sophia-G's estimate can't be negative, and the negative-curvature safeguard is there for Sophia-H.
Estimating every 10 steps
, and the Hessian is estimated on only part of the batch. GPT-2's batch has 480 examples; Sophia-H uses 32 of them and Sophia-G 240. Per-step times measured on A100s are:
| Method | 355M | 770M |
|---|---|---|
| AdamW | 1.77 s | 3.25 s |
| Sophia-H | 1.88 s | 3.40 s |
| Sophia-G | 1.86 s | 3.42 s |
Compiled from Table 1 of the paper; the per-step times include the Hessian estimate every 10 steps, amortized.
For memory, Sophia stores only two states, and , the same as AdamW.
Where the theory applies
Section 4 proves a convergence rate for a simplified version. The function must be strictly convex, the Hessian can change only so much within a certain radius, and the updates are deterministic; moreover, the analysis uses the full Hessian, rotating into its eigenbasis, clipping per coordinate there, and rotating back. Under these conditions, the number of steps to reach error is independent of the condition number (the largest curvature divided by the smallest); SignGD's rate depends on the condition number even on a simple quadratic.
Why doesn't the theory use the diagonal Hessian? The authors explain that for general functions, the directions of curvature need not line up with the coordinate axes, so a diagonal preconditioner may not be effective. That sentence names exactly the weakness of the practical version of Sophia. A diagonal Hessian can't straighten out a slanted valley in a real model either; that takes a matrix preconditioner like Shampoo or SOAP.
Discussion
Experimental setup
GPT-2 comes in three sizes, 125M, 355M and 770M, trained with the nanoGPT code on OpenWebText (about 9 billion training tokens), with context 1024 and batch 480; a 30M-parameter model is used for tuning. The larger models are 1.5B and 6.6B with the GPT NeoX architecture, trained on the Pile with Levanter, with context 2048.
The baselines are AdamW (, weight decay 0.1), Lion and AdaHessian. Except for the peak learning rate, all hyperparameters are grid-searched on the 30M-parameter model and then fixed. The peak learning rates for 125M and 355M get their own grid search; for larger models, the authors search upward for the learning rate at which going 1.25 times higher would blow up. The shared settings are cosine decay to 0.05 times the peak, 2,000 warmup steps, gradient-norm clipping at 1.0 and bf16. Sophia uses , and weight decay 0.2.
One detail is asymmetric. AdamW and Lion are unstable on models of 355M and up, so the authors give them a common trick, dividing the attention scores by the layer index; Sophia doesn't use it.
Where the 2× comes from

The first three plots read the same way. The black line is AdamW on the full budget, and the blue and green lines are Sophia on half of it; a gray arrow runs horizontally from Sophia's endpoint to AdamW's, labeled “2x Speedup.” Plot (d) looks at model size instead. Trained for the same 100k steps, Sophia's 540M model has a lower loss than AdamW's 770M model; in other words, AdamW needs about 40% more parameters to catch up.
Final losses at the same number of steps:
| Model | AdamW | Sophia-H | Sophia-G |
|---|---|---|---|
| GPT-2 125M | 2.921 | 2.901 | 2.875 |
| GPT-2 355M | 2.691 | 2.645 | 2.627 |
| GPT-2 770M | 2.613 | 2.554 | 2.524 |
| NeoX 1.5B | 2.250 | – | 2.218 |
| NeoX 6.6B | 1.992 | – | 1.969 |
Compiled from the caption of Figure 5 in the paper; 100k steps for GPT-2, 200k steps for NeoX 1.5B.
The paper says that in this range, a validation loss 0.05 lower is worth roughly half the steps. The gap at 770M is 0.06 to 0.09, which fits; at 125M it's only 0.02 to 0.05. At 6.6B parameters there is only a comparison at equal steps, with no half-step run. Downstream evaluation covers only four SuperGLUE subtasks (2-shot), where Sophia at 200k steps and AdamW at 400k steps reach similar accuracy.
Stability and hyperparameters
On the 125M-parameter GPT-2, all three use gradient-norm clipping at 1.0; AdamW and Lion trigger clipping on more than 10% of steps, while Sophia-H rarely does. Without the attention-scaling trick, AdamW on the 355M model can use a learning rate of at most 1.5e-4 and blows up at 3e-4, and Lion's ceiling drops from 6e-5 to 3e-5; Sophia-H never needed the trick in the first place. Across nine combinations of and swept on the 30M-parameter model, the losses all fall between 3.341 and 3.356.
Ablations

In all three plots, lower left is better, meaning lower loss for the same compute.
(a) Estimating the Hessian every step () gives the lowest loss at the same number of steps, but costs 50% more, so measured by compute it actually loses to ; still beats AdamW but falls short of .
(b) With clipping added to all of them, GNB beats the empirical Fisher, and Sophia-H beats AdaHessian. The authors suggest this is because Sophia-H takes a moving average of the Hessian estimates while AdaHessian takes a moving average of their squares, and the former reduces noise better.
(c) is the one most worth a close look. The red “Clip” line uses no preconditioner at all, only elementwise clipping of the momentum; the authors themselves say this is “essentially the same as sign momentum, or Lion with a single β.” That line already beats black AdamW. Without clipping, AdaHessian diverges at and GNB at , so they have to estimate every step or two. Blue Sophia-G sits another notch lower.
In other words, part of Sophia's gain comes from sign momentum itself, with the Hessian adding a layer on top; clipping is the safeguard that lets the Hessian be estimated less often.
What others measured later
Within a few months of the paper appearing on arXiv, issues about failed reproductions started showing up in the official repo.
Reproductions on GitHub. On June 28, 2023, Aaron Defazio opened an issue in the official repo. He trained GPT-2 small in fp32 on 16 V100s and ran AdamW through Sophia's own training script as well, ruling out script differences; each optimizer got its own learning-rate sweep, and both came out best at 1e-3. Sophia-G reached 2.876 and AdamW 2.869, and he leaned toward the view that the two end at about the same loss. The first author replied that fp32 by itself gives language models a lower loss than bf16, offered recommended settings, and said his own fp32 Sophia run reached 2.846; Defazio reran with them, still couldn't reproduce the result, and closed the issue. In November, another issue reproduced 125M by following the README: AdamW at 200k steps got 2.863, lower than Sophia-G's 2.897 at 100k steps. The authors updated the config files in January 2024.
No Train No Gain (Kaddour et al., 2023). This paper puts Sophia, Lion and several training-efficiency methods through BERT and T5 pretraining under fixed budgets of 6, 12 and 24 hours, against an AdamW baseline whose learning rate decays fully. Dropping Sophia straight into an FP16 training pipeline produces NaNs, and it only becomes stable in BF16; T5 also needs RMS scaling to converge. For BERT at 24 hours, the GLUE average is 77.9 for AdamW and 72.3 for Sophia; for T5 on SNI it's 39.5 vs 34.1. Even leaving out the time spent on Hessian estimation and running the same number of steps, the authors “still do not observe any speedup.”
Zhao et al. (2024). Comparing SGD, Adafactor, Adam, Lion and Sophia on autoregressive language models, they conclude that all but SGD perform about the same. In the appendix they compare Sophia with Signum (sign updates with momentum, equivalent to Sophia with set to 0), and Sophia doesn't win.
Fantastic Pretraining Optimizers (Wen et al., 2025). David Hall, Tengyu Ma and Percy Liang on this author list are also authors of Sophia. Retuning and comparing from 130M to 520M parameters, they find Sophia loses to AdamW at small compute and pulls slightly ahead only as the model or data grows, with no significant speedup below 0.5B. They also explain why. The Sophia paper “largely follows the correct procedure,” and the baseline's peak learning rate really was tuned; but the data loader in that codebase didn't fully shuffle the data, and in that code AdamW's best learning rate was pushed very low. Once the data is fully shuffled, AdamW can use a much larger learning rate.
Conclusion
Sophia uses a diagonal Hessian estimated every 10 steps as its denominator, then caps each step's movement at with per-coordinate clipping. Coordinates with bad estimates, negative curvature or too little curvature automatically fall back to a sign update with momentum. In the paper's setup, GPT-2 and the 1.5B-parameter NeoX both catch up with AdamW in half the steps, at only about 5% more time per step.
The most useful thing this paper leaves practitioners turns out to be that measurement rule. Next time you see “twice as fast,” first check that the baseline's learning rate was tuned to its best with the same total step count and the same fully shuffled data, then check that the new method is compared only on final checkpoints. If you use Sophia, tune by watching the fraction of clipped coordinates, and choose bf16 for mixed precision.
(In the paper's own ablation, dropping the Hessian and keeping only the clipping already beats AdamW; and in Zhao et al.'s later comparison, Sophia indeed didn't beat Signum.)