Symbolic Discovery of Optimization Algorithms
- Publication
- NeurIPS 2023Conference
- First submitted to arXiv
- arXiv:2302.06675
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{optimizerslion2023,
title = {{Symbolic Discovery of Optimization Algorithms}},
author = {Xiangning Chen and Chen Liang and Da Huang and Esteban Real and Kaiyuan Wang and Yao Liu and Hieu Pham and Xuanyi Dong and Thang Luong and Cho-Jui Hsieh and Yifeng Lu and Quoc V. Le},
year = {2023},
eprint = {2302.06675},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2302.06675}
}
Here is a program. It clips the gradient and takes its arcsin, then does two interpolations, one of them with a coefficient of 1.109, greater than 1; after computing the update, it sets the momentum variable to the cosh of that update.
No human wrote it. A Google Brain team started from AdamW and had a computer randomly edit the program one statement at a time, generating over a million candidates and spending about 3,000 TPU v2 days to sift out this one. Delete the statements that do nothing and simplify, and only four lines remain, under the name Lion (EvoLved Sign Momentum).
Those four lines keep a single momentum buffer, and each parameter's update looks only at the sign. The paper reports that it gives ViT up to 2% higher accuracy on ImageNet, cuts pretraining compute on JFT by up to 5 times, and pushes BASIC-L's zero-shot ImageNet accuracy to 88.3%.
The same paper also lists where Lion doesn't win, and does so quite honestly. This note reads in two halves. The first looks at how the computer found these four lines; the second at what the sign-only design buys, what it costs, and how it later fared in language-model pretraining.
Defining the problem
By 2023 there were plenty of hand-designed optimizers, mostly adaptive methods. Yet for training large language, vision and multimodal models, people were still using AdamW, or Adafactor when memory had to be saved.
There had also been many attempts to have computers find optimizers, each stuck somewhere.
- Learning to optimize: train a neural network to output the parameter updates directly. These black-box optimizers had been learned on only a few small tasks and failed to generalize to large models and long training runs.
- Searching over expression trees: for example, Bello et al. used reinforcement learning in 2017 to find PowerSign and AddSign. To keep the search tractable, the operands were fixed and the tree size limited, so how momentum is tracked and how it enters the update couldn't be changed. The paper points out that this is precisely the most crucial part of Lion.
- AutoML-Zero: hands the entire machine-learning pipeline to the search, but evaluates only on toy tasks.
The authors split the problem in two. First, the space of programs is infinite and good programs are extremely sparse; they randomly sampled 2 million programs on a cheap proxy task, and the best one still clearly lost to AdamW. Second, the search can only evaluate on small tasks, while the tasks that actually matter are more than 10,000 times larger, and programs that win on small tasks often fall apart on large ones.
Solving the problem
Writing an optimizer as a program
The search operates on a train function. Its inputs are the weights w, the gradient g, the current step's learning rate lr, and two variables m and v, initialized to zero, for recording history; its output is the update to subtract this step. Written in this form, AdamW looks like this (the paper's Program 3, omitting bias correction and ):
def train(w, g, m, v, lr):
g2 = square(g)
m = interp(g, m, 0.9)
v = interp(g2, v, 0.999)
sqrt_v = sqrt(v)
update = m / sqrt_v
wd = w * 0.01
update = update + wd
lr = lr * 0.001
update = update * lr
return update, m, v
interp(x, y, a) equals (1 - a) * x + a * y, an exponential moving average. The signature deliberately keeps only two extra variables, as AdamW does, so nothing the search finds can use more memory than AdamW.
The function body is a sequence of assignment statements, with no limit on its length or on local variables; each statement calls one of 40-odd common math functions, most of which correspond to NumPy functions or linear-algebra operations. Conditionals, loops and user-defined functions were also tried, brought no improvement, and were left out.
There are three kinds of mutation: insert a random statement, delete a statement, or change one argument of a statement. An argument can be replaced by an existing variable or by a new constant, and existing constants are multiplied by , with drawn from a standard normal distribution. The constants are the algorithm's hyperparameters; in the program above, for example, 0.001 is the peak learning rate and 0.01 the weight decay.
Programs may contain redundant statements that have no effect on the output. Each mutation changes only one statement, so when a larger change needs two or three statements working together, redundant statements can serve as stepping stones.
Making the search tractable
The search uses regularized evolution. The population holds 1,000 programs; each round, two are drawn at random, the better one is copied and mutated, the new program joins the population, and the oldest one leaves.
The initial population is all AdamW, combined with two kinds of restart. One restarts from AdamW, running several times to explore more; the other restarts from the best program so far, to dig deeper. Results vary a lot from one search to the next, so multiple runs are necessary. Against AdamW hyperparameter tuning with four times the compute, and against random search with four times the compute, the evolutionary search clearly wins.
Another key to keeping costs down is abstract execution. Before a program actually runs, three things happen:
- Infer each variable's type and shape; programs with mismatched types or shapes are discarded right away, and mutation continues until a valid child program comes out;
- Compute a functional hash from inputs to outputs, so programs that are semantically identical but written differently are looked up instead of rerun;
- Trace which statements the output depends on, picking out the redundant ones.
Across five searches of 300k programs each, about 70% of the statements are redundant by the late stage of the search, and deleting them makes the programs 3 times shorter on average; the cache hit rate is about 89%, cutting the search cost to roughly a tenth.
The proxy tasks are all small. For vision, a 3-layer, 96-dimensional ViT is trained on 10% of ImageNet with images downsized to 64×64, batch size 64, for 30k steps; for language, a 2-layer, 128-dimensional Transformer is trained on LM1B for 20K steps. Each program is evaluated on a single TPU v2 within 20 minutes, with validation accuracy or perplexity as the fitness.
Each search runs on 100 TPU v2 chips for about 72 hours and generates 200k to 300k programs, of which only 20k to 30k are actually executed thanks to caching. Five searches plus one more round restarting from the best program come to about 3,000 TPU v2 days in total.
From small tasks to large ones
The search fitness keeps rising while validation on a somewhat larger task starts to fall; the paper calls this meta-overfitting, overfitting to the proxy task. The authors ran 50 searches and recorded at what point each one began to overfit. Half the searches overfit early, there is a long tail that overfits only much later, and the later a search overfits, the better its programs do on large tasks.
So on top of the search they added funnel selection. Starting from proxy task A, they enlarge the model and the step count to make a task B that is 10 times larger; only programs that beat the baseline on B go on to task C, 100 times larger.
Programs that make it through are then simplified in three passes. First, abstract execution removes redundant statements; next, statements that do something but whose removal makes little difference are deleted (this can also be done by evolution with insertions disabled); finally, a human reorders statements and renames variables into a simple, mathematically equivalent form.
From Program 4 to four-line Lion

This is the strange program from the start of the note. Let's simplify it line by line.
- The three red lines:
m2 = m * m,abs_m = sqrt(m2)andupdate = m / abs_mcompute , which issign(m). clipandarcsin: removing them didn't hurt quality, so they were deleted. This step isn't an exact equivalence.clipshrinks the whole gradient according to the current step's gradient norm, by a factor that differs from step to step, and once the gradient enters the momentum interpolation, that still changes the relative weight of the current step against history; deleting them rests on the experimental results.- The final
cosh: it writes tom, butmis reassigned right at the start of the next round, so the line has no effect. - The two green lines:
minterpolates the current gradient andvwith 0.899, andvin turn interpolates the gradient and this newmwith 1.109. Substituting the first line into the second,vbecomes a moving average with a coefficient close to 1, just written another way; the paper tidies it up as .m, meanwhile, is the interpolation of the current gradient and history at about 0.9, before it goes into the sign. - Constants:
0.4602is the weight decay and0.0002the learning-rate scale; after simplification they become two hyperparameters.
Also, Adam's moving averages start at 0, so the first few estimates are too small, and bias correction is needed to keep the update size from going off. The paper points out that bias correction doesn't change the direction, and the sign looks only at the direction, so this step goes too.
The tidied-up Lion (the paper's Algorithm 2):
The defaults are and . Compared with AdamW, it drops the second moment , and bias correction. One step of the official PyTorch implementation is just these lines (google/automl):
p.data.mul_(1 - lr * weight_decay) # decoupled weight decay
update = exp_avg * beta1 + grad * (1 - beta1) # c_t: interpolate the current gradient and momentum
p.add_(update.sign_(), alpha=-lr) # keep only the sign
exp_avg.mul_(beta2).add_(grad, alpha=1 - beta2) # m_t: update momentum with beta2
Note the order of the last two lines. The weights are updated with , while what's stored for the next step is , and the two use different coefficients.
What each β does
Why two βs? Expanding shows how much weight each past gradient carries in the update direction. The current gradient gets ; the gradient from steps back gets , which with the defaults is about . The figure below plots the summed weight of the most recent steps.

The orange line is momentum with a single . The current gradient gets 10%, but it only remembers the last twenty or thirty steps. The green line uses only ; its memory is about 10 times longer, but the current gradient gets just 1%. Lion's blue line has it both ways. In the update, the current gradient gets 10%, as on the orange line, while the older part follows the green line's long tail.
The paper's ablation tests exactly the orange and green lines. On ViT-B/16 (ImageNet, with RandAug and Mixup), the two versions that take the sign of a single momentum reach 79.54% () and 79.90% (), both below AdamW's 80.12%, while Lion gets 80.77%. Language models on PG-19 lead to the same conclusion.
This is also where Lion differs from 2018's signSGD (and its momentum version). Both take the sign, but they accumulate momentum differently; signSGD was designed to cut communication in distributed training, and it performs worse on convolutional networks for image classification.
The cost of taking the sign
The sign gives every coordinate an update of the same size. Ignoring weight decay, every parameter moves either or at each step.
Adam's update varies in size from coordinate to coordinate, and its overall root mean square (RMS) is usually below 1; Moonshot AI measured 0.2 to 0.4 in their Muon paper. The sign's output is all ±1, so its RMS is exactly 1, and the norm of the whole update vector is larger as a result. The paper recommends a learning rate 3 to 10 times smaller for Lion than for AdamW, with the initial, peak and final learning rates all scaled down by the same factor; other settings, such as the learning-rate schedule and gradient clipping, need no change.
Once the learning rate shrinks, weight decay has to be adjusted too. The effective strength of decoupled weight decay on the weights is , so if is 3 to 10 times smaller, has to be 3 to 10 times larger to keep a similar strength. The paper lists some of the values actually used:
| Task | Lion (, ) | Baseline (, ) |
|---|---|---|
| ViT-B/16, strong augmentation | 1e-4, 10.0 | AdamW 1e-3, 1.0 |
| Diffusion model | 3e-5, 0.1 | AdamW 3e-4, 0.01 |
| 7.5B language model | 1e-4, 0.01 | Adafactor 1e-3, 0.001 |
Compiled from §5 of the paper.
As a rule, the values can stay at their defaults; only the language-model experiments tuned them, to and . The authors' experience is that a smaller , which remembers a shorter history, makes training more stable.
The upside is memory and speed. Lion stores one momentum buffer, while AdamW stores two. For ViT-B/16 at resolution 224 with a batch of 4,096, AdamW needs at least 16 TPU v4 chips to fit, while Lion needs only 8 (both store momentum in bfloat16). Because the computation is simple, steps per second in the experiments are also 2% to 15% higher than with AdamW and Adafactor, depending on the task, codebase and hardware.
Discussion
All comparisons are against AdamW, or against Adafactor for large models where memory is tight; momentum SGD is compared only on ResNet. Every optimizer's peak learning rate and weight decay are tuned on a log scale. The default is cosine decay with 10K warmup steps, with momentum stored in bfloat16.
Image classification: the best-looking results
Training from scratch on ImageNet (averaged over three runs; ResNet-50 for 90 epochs at batch 1,024, the other models for 300 epochs at batch 4,096), selected Top-1 results:
| Model | Augmentation | AdamW | Lion |
|---|---|---|---|
| ResNet-50 | No | 76.34 | 76.45 |
| Mixer-B/16 | No | 68.12 | 70.11 |
| ViT-B/16 | No | 75.48 | 77.44 |
| ViT-B/16 | Yes | 80.12 | 80.77 |
| CoAtNet-3 | Yes | 84.45 | 84.87 |
Compiled from Table 2 of the paper; “Yes” means RandAug and Mixup. ResNet-50 with momentum SGD gets 76.22.
The trend is consistent. The fewer inductive biases an architecture has, as with ViT and MLP-Mixer, the bigger the gap; strong data augmentation narrows it; and on the convolutional ResNet-50, the three optimizers are nearly tied.
The authors' explanation for ViT is that the sign introduces a kind of noise with a regularizing effect. The Lion-trained ViT-B/16 actually has a higher training loss (0.75 vs 0.61) yet 2% higher validation accuracy; with Gaussian noise added to the weights, the Lion model's loss (1.37) is also far below the AdamW model's (3.74), indicating it converged to a flatter region. So far, this explanation rests only on such indirect evidence.
On large-scale data like JFT, the gap becomes even clearer.

In the left plot, the horizontal axis is pretraining compute and the vertical axis is ImageNet accuracy after fine-tuning; the points on the curves cover ViT-B/16, ViT-L/16, ViT-H/14 and different pretraining budgets. The arrow marks a Lion-trained ViT-L/16 catching up with an AdamW-trained ViT-H/14, cutting pretraining compute by 3 times on ImageNet and ImageNet V2 and by 5 times on ImageNet ReaL. Zhai et al. trained ViT-L/16 with AdamW for 4 million steps and still fell short of Lion at 1 million steps.
After fine-tuning, the Lion-trained ViT-L/16 leads by even more on the harder distribution-shift test sets, by 6.08 on ImageNet-A and 5.54 on ImageNet-R. With JFT-3B as the data, a Lion-trained ViT-g/14 surpasses the earlier ViT-G/14 with 1.8 times fewer parameters, and Lion's ViT-G/14 reaches 90.71%.
Image-text contrastive learning improves too. LiT freezes the image tower and trains only the text tower, and its zero-shot ImageNet accuracy is about 0.7 to 1.1 higher at all three scales. BASIC-L switches both stages, image-tower pretraining and contrastive learning, to Lion, and its zero-shot ImageNet accuracy rises from Adafactor's 85.7% to 88.3%, 2.0 above the best result at the time; after fine-tuning it reaches 91.1%, 0.1 above the previous record.
Diffusion models: the higher the resolution, the bigger the gap
The right plot is unconditional generation on ImageNet 256×256 (an improved U-Net, batch 2,048, a constant learning rate, 1,000 DDPM sampling steps, no guidance). Lion reaches AdamW's final FID at 440k steps, 2.3 times fewer steps, and its final FID is 4.1 vs 4.7. 64×64 and 128×128 improve too, but the higher the resolution, the bigger the gap.
Imagen-style text-to-image generation is mixed. The 64×64 base model shows no clear improvement; on the 64→256 super-resolution model, Lion gets higher CLIP scores, and its FID curve is less noisy.
Language models: a tie at scale
The small-scale experiments use Wiki-40B and PG-19, with 110M, 336M and 731M models, tokens per batch and 125k training steps. Lion's validation perplexity is consistently lower; the medium model gets speedups of 1.6× and 1.5× on the two datasets, and the large model reaches 2× on PG-19.
Scale up, though, and things look different. On a 1.6-trillion-token dataset, the authors train three models of 1.1B, 2.1B and 7.5B following GPT-3's recipe, each on 300B tokens with 3M-token batches, against Adafactor. Throughout training, the two show no difference in perplexity. Lion leads only slightly on average one-shot in-context evaluation scores; for the 7.5B model, for example, the average over generation tasks goes from 24.1 to 24.7 and over understanding tasks from 61.3 to 61.7.
For masked language modeling (C4), perplexity for Lion vs AdamW is 4.18 vs 4.25, 3.42 vs 3.54 and 3.18 vs 3.25; fine-tuning T5 on GLUE, the average scores at the three scales are each 0.4 to 0.5 higher.
Batch size and tolerance to tuning
The sign amplifies gradient noise, so intuitively Lion should need larger batches to estimate the direction accurately. The authors measured it directly.

The left plot fixes training at 300 epochs and varies the batch size. AdamW's best batch is 256 and Lion's is 4,096, so Lion does prefer large batches; still, at a batch of 64, Lion doesn't collapse either. At 32K (only 11k steps), Lion gets 77.9% and AdamW 75.4%.
The middle and right heat maps each sweep three learning rates and three weight decays. AdamW drops to around 67.5% in the corner where both are on the large side; of Lion's nine cells, only the one with the smallest learning rate and weight decay is noticeably lighter. In this setup, Lion is less sensitive to hyperparameters. Note that each map's grid was picked from its own recommended range, and Lion's learning rates are 3 to 10 times smaller to begin with.
The paper also compares RAdam, NAdam, AdaBelief, AMSGrad and the AutoML-discovered PowerSign and AddSign (each with its own tuned learning rate and weight decay, Optax defaults otherwise). On ViT-B/16, these baselines land between 78.50% and 80.32% with no clear winner among them, while Lion gets 80.77%. PowerSign and AddSign are the worst two, in line with the old experience that automatically discovered optimizers rarely generalize to real tasks.
Limitations the paper states itself
In Section 6, the authors list the limitations clearly.
- Search bias: the design of the search space is still inspired by first-order optimizers, so what it finds tends to resemble existing algorithms, and it lacks the functions second-order methods need; the search is expensive, and the simplification still takes manual work.
- Tasks with a small gap or none: on ResNet the three optimizers differ little; strong augmentation eats up the gain; and on the Imagen base model, language-model perplexity on the large internal dataset, and masked language modeling on C4, Lion is about the same as AdamW. The authors think what these have in common is large, high-quality data.
- Batches that are too small: with batches smaller than 64, Lion may not beat AdamW.
- Memory: it still stores a bfloat16 momentum buffer, which remains expensive for very large models.
Alongside point 2, the authors add that perplexity can be considered a more reliable metric than in-context evaluation. In other words, on the very metric the authors themselves consider more reliable, the large-scale language models show no difference.
Later evaluations
Language-model pretraining later became the main battleground for optimizers, and several independent studies retested Lion; all of them found smaller gains than the original paper.
- Kaddour et al. (NeurIPS 2023): pretraining BERT and T5 under fixed compute budgets, the gains of Lion, Sophia and similar methods vanish against a baseline whose learning rate decays all the way down.
- Zhao et al. (ICLR 2025): on autoregressive language models from 150M to 1.2B, Adam, Adafactor, Lion and Sophia, but not SGD, have similar best performance and similar stability across learning rates. They also point out that Lion with is Signum (signSGD with momentum).
- Semenov et al. (September 2025): Lion and Signum do poorly with small batches and improve greatly with larger ones, but at 720M with 1 million tokens per batch, they fall behind the other methods again.
- Wen et al. (September 2025): after a full hyperparameter sweep for each optimizer, Lion is at most 5% slower than AdamW, occasionally slightly faster, and falls further behind as models grow. The best values they found happen to be a good check of the advice above. In the 130M, 1× Chinchilla setting, AdamW's best learning rate is 0.008 with weight decay 0.1; the best combination in Lion's table is 0.002 and 0.7, and changing only the learning rate to 0.001, or only the weight decay to 0.6, gives a loss 0.003 lower still. A learning rate 4 to 8 times smaller and a weight decay 6 to 7 times larger both fall within the Lion paper's advice of “3 to 10 times.”
Community implementations report similar experience. The README of lucidrains' lion-pytorch records a string of user reports. Keeping Adam's learning rate, Lion did much worse than Adam; with the learning rate divided by 3, it actually won early on; following the “10 times smaller” rule directly produced the worst run of all. The README ends up recommending it only for batch sizes of 64 and above, and notes that negative results mostly came from settings the paper didn't evaluate, such as reinforcement learning.
Conclusion
Lion is an optimizer found by program search and then simplified by hand. Its update is , with momentum accumulated separately at ; every parameter moves by a fixed at every step, and only one momentum buffer is stored. Its gains are clearest on ViT with JFT, image-text contrastive learning and high-resolution diffusion models, and these gains all come with conditions: large batches, Transformer-style architectures and data augmentation that isn't too strong.
To swap AdamW for Lion, divide the learning rate by 3 to 10, multiply the weight decay by the same factor, and make sure the batch isn't too small. Then start the sweep from 3×, rather than jumping straight to 10×. For language models, try and first.
The search started from AdamW, evolved for 3,000 TPU days, and in language-model pretraining ended up with an optimizer about as fast as AdamW. At least it cut the optimizer state in half.