Back to paper library
arXiv 2017ICLR 2019OptimizersPaper notes

AdamW

Not that kind of decay

Original paper

Decoupled Weight Decay Regularization

Paper authors

Ilya Loshchilov · Frank Hutter

Publication
ICLR 2019Conference
First submitted to arXiv
arXiv:1711.05101
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{optimizersadamw2017,
  title = {{Decoupled Weight Decay Regularization}},
  author = {Ilya Loshchilov and Frank Hutter},
  year = {2017},
  eprint = {1711.05101},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1711.05101}
}
On this page

The two lines of PyTorch below have the same parameter name and the same numbers:

python
torch.optim.Adam(model.parameters(), lr=1e-3, weight_decay=1e-2)
torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-2)

The first line treats 1e-2 as the coefficient of L2 regularization, adds it into the gradient, and hands the result to Adam; the second multiplies the parameters directly by 1−10−3×10−21-10^{-3}\times10^{-2} at every step, independently of the gradient. The two train different models.

When Ilya Loshchilov and Frank Hutter of the University of Freiburg pointed out this difference in November 2017, the common framework implementations all used the first version, and they all called it weight decay.

Their claim is that this explains a large part of why Adam generalizes worse than SGD on image classification.

Defining the problem

Two kinds of "decay"

Weight decay, as Hanson and Pratt defined it in 1988, multiplies the weights by a number slightly less than 1 at every step before applying the gradient update (the paper's Eq. 1):

θt+1=(1−λ) θt−α ∇ft(θt)\theta_{t+1}=(1-\lambda)\,\theta_t-\alpha\,\nabla f_t(\theta_t)

L2 regularization instead adds a term λ′2∥θ∥22\frac{\lambda'}{2}\|\theta\|_2^2 to the loss function, whose gradient is λ′θ\lambda'\theta. With an SGD update:

θt+1=θt−α(∇ft(θt)+λ′θt)=(1−αλ′) θt−α ∇ft(θt)\theta_{t+1}=\theta_t-\alpha\big(\nabla f_t(\theta_t)+\lambda'\theta_t\big)=(1-\alpha\lambda')\,\theta_t-\alpha\,\nabla f_t(\theta_t)

Compare this with the previous equation, and as long as λ′=λ/α\lambda'=\lambda/\alpha, the two are exactly the same. This is the paper's Proposition 1, and it is why people have long used the two terms interchangeably.

But this equality hides something that is often overlooked. If a problem has a best amount of decay λ\lambda, then the best L2 coefficient λ′=λ/α\lambda'=\lambda/\alpha is tied to the learning rate; change the learning rate, and the L2 coefficient has to change with it. Even in SGD, these two hyperparameters are coupled.

What about Adam?

Adam divides the gradient by v^t\sqrt{\hat v_t}. If L2 is written into the gradient, the λ′θ\lambda'\theta term gets divided along with it.

In May 2017, Wilson et al. found that Adam generalized worse than SGD on several tasks, and the best image classification results at the time were all still trained with SGD plus momentum. Several explanations for this gap had been proposed, such as sharp minima or problems inherent to adaptive methods. This paper offers another, more concrete hypothesis, that the problem may lie in the "weight decay" implemented in frameworks, which inside Adam isn't weight decay at all.

Solving the problem

Not equivalent in Adam

Write Adam in a general form, θt+1←θt−αMt∇ft(θt)\theta_{t+1}\leftarrow\theta_t-\alpha M_t\nabla f_t(\theta_t), where MtM_t is the preconditioning matrix; for SGD MtM_t is the identity matrix, while for Adam it is diag⁡(1/(v^t+ϵ))\operatorname{diag}(1/(\sqrt{\hat v_t}+\epsilon)) plus momentum. One step under each approach is:

L2:θt+1=θt−αλ′Mtθt−αMt∇ft(θt)weight decay:θt+1=(1−λ) θt−αMt∇ft(θt)\begin{aligned} \text{L2:}\quad &\theta_{t+1}=\theta_t-\alpha\lambda' M_t\theta_t-\alpha M_t\nabla f_t(\theta_t)\\ \text{weight decay:}\quad &\theta_{t+1}=(1-\lambda)\,\theta_t-\alpha M_t\nabla f_t(\theta_t) \end{aligned}

For the two to be equal, we would need λθt=αλ′Mtθt\lambda\theta_t=\alpha\lambda' M_t\theta_t for every θt\theta_t, which is only possible when MtM_t is a constant multiple of the identity matrix. Adam's MtM_t differs from coordinate to coordinate, so no λ′\lambda' can make L2 equal to weight decay. This is Proposition 2.

Where exactly is the difference? Let's work it out with two parameters. Suppose both currently have value 1, their typical gradient sizes (v^t\sqrt{\hat v_t}) are 1 and 0.01, and the L2 coefficient is λ′=10−4\lambda'=10^{-4}. Ignore momentum for now, and assume λ′θ\lambda'\theta is small enough not to affect v^t\hat v_t:

Parameterv^t\sqrt{\hat v_t}Decay under L2Decay under AdamW
A110−4 α10^{-4}\,\alpha10−4 α10^{-4}\,\alpha
B0.0110−2 α10^{-2}\,\alpha10−4 α10^{-4}\,\alpha

Computed for this note. The AdamW column follows PyTorch's formulation, where the decay is αλθ\alpha\lambda\theta, with λ=10−4\lambda=10^{-4}.

Under L2, the decay is first divided by v^t\sqrt{\hat v_t}, so parameter A, with its large gradients, is decayed by only one hundredth as much as parameter B, with its small gradients. As the paper puts it, under L2, weights with large historical gradients are regularized less. Weight decay, by contrast, shrinks every weight by the same proportion.

Proposition 3 captures this difference in a formula. If the preconditioning matrix is fixed at Mt=diag⁡(s)−1M_t=\operatorname{diag}(s)^{-1}, then weight decay is equivalent to an L2 term weighted by the gradient scale:

ftsreg(θ)=ft(θ)+λ2α∥θ⊙s∥22f_t^{\text{sreg}}(\theta)=f_t(\theta)+\frac{\lambda}{2\alpha}\big\|\theta\odot\sqrt{s}\big\|_2^2

Parameters with large sis_i (large past gradients) are regularized more than under plain L2. The penalty term is ∥θ⊙s∥2\lVert\theta\odot\sqrt{s}\rVert^2, so each coordinate's penalty weight is proportional to sis_i. Real Adam changes its preconditioner at every step, so this proposition can't be applied directly, but it shows what kind of objective each of the two approaches is optimizing.

Two versions in Algorithm 2

The fix is to take the decay out of the gradient and apply it directly to the parameters. The paper puts the two versions side by side in a single algorithm, marking the term each one adds in a different color; here we mark them with labels underneath instead:

gt←∇ft(θt−1)  + λθt−1⏟Adam+L2mt←β1mt−1+(1−β1) gtvt←β2vt−1+(1−β2) gt2m^t←mt/(1−β1t),v^t←vt/(1−β2t)ηt←SetScheduleMultiplier(t)θt←θt−1−ηt(α m^t/(v^t+ϵ)  + λθt−1⏟AdamW)\begin{aligned} g_t &\leftarrow \nabla f_t(\theta_{t-1}) \;\underbrace{+\,\lambda\theta_{t-1}}_{\text{Adam+L2}}\\ m_t &\leftarrow \beta_1 m_{t-1}+(1-\beta_1)\,g_t\\ v_t &\leftarrow \beta_2 v_{t-1}+(1-\beta_2)\,g_t^2\\ \hat m_t &\leftarrow m_t/(1-\beta_1^t),\qquad \hat v_t\leftarrow v_t/(1-\beta_2^t)\\ \eta_t &\leftarrow \text{SetScheduleMultiplier}(t)\\ \theta_t &\leftarrow \theta_{t-1}-\eta_t\Big(\alpha\,\hat m_t/\big(\sqrt{\hat v_t}+\epsilon\big)\;\underbrace{+\,\lambda\theta_{t-1}}_{\text{AdamW}}\Big) \end{aligned}

Keep only the term labeled Adam+L2 on the first line, and you have Adam with L2; keep only the term labeled AdamW on the last line, and you have AdamW. The defaults are the same as Adam's, α=0.001\alpha=0.001, β1=0.9\beta_1=0.9, β2=0.999\beta_2=0.999, ϵ=10−8\epsilon=10^{-8}.

ηt\eta_t is the schedule multiplier for the learning rate; it can be fixed, decaying, or periodically restarted as in AdamWR later on. Note that in the last line, the decay term λθt−1\lambda\theta_{t-1} is multiplied by ηt\eta_t but not by the base learning rate α\alpha. This is what the paper means by "decoupling," since λ\lambda no longer changes meaning when α\alpha changes. Framework implementations handle this detail differently, and we'll come back to it in the discussion.

The SGD version (SGDW, Algorithm 1) works the same way. With L2 written into the gradient, mt←β1mt−1+ηtαgtm_t\leftarrow\beta_1 m_{t-1}+\eta_t\alpha g_t; SGDW instead changes the last line to θt←θt−1−mt−ηtλθt−1\theta_t\leftarrow\theta_{t-1}-m_t-\eta_t\lambda\theta_{t-1}. In SGD the two were interchangeable to begin with, and the point of SGDW is that λ\lambda doesn't have to be retuned whenever α\alpha changes; with momentum the two are no longer exactly the same either, since the L2 term gets accumulated by the momentum.

In code, the difference comes down to two lines. Below is a single-tensor version following the algorithm listed in the PyTorch 2.15 documentation:

python
def adam_step(p, grad, m, v, t, lr=1e-3, betas=(0.9, 0.999), eps=1e-8,
              weight_decay=0.0, decoupled=False):
    b1, b2 = betas
    if weight_decay and not decoupled:
        grad = grad + weight_decay * p        # Adam + L2: the decay term is mixed into the gradient and later divided by sqrt(v_hat)
    if weight_decay and decoupled:
        p.mul_(1 - lr * weight_decay)         # AdamW: shrink the parameter directly; nothing goes into m or v
    m.mul_(b1).add_(grad, alpha=1 - b1)
    v.mul_(b2).addcmul_(grad, grad, value=1 - b2)
    m_hat = m / (1 - b1 ** t)
    v_hat = v / (1 - b2 ** t)
    p.add_(-lr * m_hat / (v_hat.sqrt() + eps))

The decoupled=True line uses lr * weight_decay, whereas the paper uses ηtλ\eta_t\lambda. Keep this difference in mind for later.

Why weight decay

Section 3 borrows Aitchison's 2018 Bayesian filtering view to give weight decay a justification. Treat stochastic optimization as tracking each parameter's posterior distribution, with the state transition assumed to be P(θt+1∣θt)=N((I−A)θt,Q)P(\theta_{t+1}\mid\theta_t)=\mathcal N\big((I-A)\theta_t,Q\big); with A=λIA=\lambda I, each step multiplies the current estimate by 1−λ1-\lambda, which is exactly decoupled weight decay. This decay acts on the prior and has nothing to do with each parameter's uncertainty; to get the effect of L2, the amount of decay would instead have to vary with the uncertainty. The authors also make clear that this theory is entirely Aitchison's, and they are only summarizing it.

Normalized weight decay

In their experiments, the authors found that the best weight decay changes with training length: the longer the training, the smaller the best λ\lambda. Appendix B.1 proposes a normalized form:

λ=λnormbBT\lambda=\lambda_{\text{norm}}\sqrt{\frac{b}{BT}}

bb is the batch size, BB is the number of training samples, and TT is the total number of epochs. λnorm\lambda_{\text{norm}} can be understood as the amount of decay "if the whole of training were a single batch."

Let's work it out for CIFAR-10, with B=50,000B=50{,}000 and b=128b=128. Training for 100 epochs gives b/BT≈0.0051\sqrt{b/BT}\approx0.0051; training for 1800 epochs gives about 0.0012, so the actual λ\lambda is 18≈4.2\sqrt{18}\approx4.2 times smaller. Move to ImageNet32x32, where one epoch is 24 times as long as CIFAR-10's, and for the same number of epochs λ\lambda has to be another 24≈4.9\sqrt{24}\approx4.9 times or so smaller; the authors say that reusing CIFAR-10's raw λ\lambda directly on ImageNet32x32 makes it about 5 times too large, with much worse results. With the normalized form, the best λnorm\lambda_{\text{norm}} is quite close across training lengths, across datasets, and even between SGDW and AdamW, for example 0.025 and 0.05.

The authors also stress that the square root is just one choice based on a small number of experiments; the reliable conclusion is that doing some kind of normalization in itself improves results substantially.

AdamWR

The last component is the learning-rate schedule. SGDR, which the authors proposed earlier, uses cosine annealing with periodic restarts:

ηt=ηmin⁡(i)+12(ηmax⁡(i)−ηmin⁡(i))(1+cos⁡(πTcur/Ti))\eta_t=\eta_{\min}^{(i)}+\tfrac12\big(\eta_{\max}^{(i)}-\eta_{\min}^{(i)}\big)\big(1+\cos(\pi T_{\text{cur}}/T_i)\big)

TcurT_{\text{cur}} is the number of epochs since the last restart, updated at every batch, so it need not be an integer; TiT_i is the length of the current run. With ηmax⁡=1\eta_{\max}=1 and ηmin⁡=0\eta_{\min}=0, the learning-rate multiplier falls along a cosine from 1 to 0 in each run and then jumps back to 1. With a first run of T0=100T_0=100 and each run's length multiplied by Tmult=2T_{\text{mult}}=2, restarts happen at epochs 100, 300, 700, and 1500, and the model just before each restart is the recommended solution.

They had tried applying restarts to Adam long before, but at the time it couldn't beat SGDR, precisely because L2 doesn't work as well in Adam as it does in SGD. With weight decay fixed and normalization added (so that runs of different lengths can share the same settings), AdamW with restarts becomes AdamWR.

Discussion

Experimental setup

The experiments reuse the settings and code from the Shake-Shake regularization paper. The main model is a ResNet of depth 26 with two residual branches, whose first residual block has width 64 (2x64d, 11.6 million parameters) or 96 (2x96d, 25.6 million parameters); the batch size is fixed at 128, with standard CIFAR-10 data augmentation. They also use ImageNet32x32, which shrinks ImageNet's 1.2 million images to 32×3232\times32.

The first set of experiments compares only Adam and AdamW, with three learning-rate schedules (fixed, reduced at epochs 30, 60, and 80, and cosine annealing), each swept over a grid of learning rates and decay amounts and trained for 100 epochs. Under every schedule, decoupled weight decay beats L2, and the better the schedule, the larger the gap; cosine annealing is clearly the best, and all later experiments use it. This result also shows that even though Adam already adapts the learning rate for each parameter, adding a global learning-rate schedule on top still helps. Appendix D also trains traditional Adam (L2 with a fixed learning rate) for 1800 epochs, and on a coarse 4×4 grid its best result is only about as good as AdamW with one-eighteenth the epochs and a smaller network.

Heatmaps of learning rate and weight decay

Four heatmaps, where color shows top-1 test error on CIFAR-10 after 100 epochs, from about 3% in dark blue to over 7% in dark red, with black circles marking the 10 best settings. The vertical axis is the initial learning rate (multiplied by 0.1 for Adam and AdamW), and the horizontal axis is the L2 coefficient or weight decay (multiplied by 0.001). Top left, SGD with L2: the good region is a diagonal band from upper left to lower right. Top right, SGDW: the good region is roughly a vertical band, concentrated around weight decay of 1/4 to 1/2. Bottom left, Adam with L2: the good region is small and near the bottom left corner, the best settings include ones with L2 set to 0, and the colors lean green, meaning higher error. Bottom right, AdamW: the good region is an upright ellipse, concentrated at weight decay of 1/4 to 1/2, as dark as SGDW's.
Original Figure 2, Loshchilov and Hutter, from PDF page 6, arXiv license, scholarly quotation.

This figure is the core evidence of the whole paper. All four panels show the test error of a 26 2x64d ResNet trained on CIFAR-10 for 100 epochs; the vertical axis is the initial learning rate, the horizontal axis is the L2 coefficient or weight decay, bluer is better, and the black circles are the 10 best settings.

Top left, SGD with L2. The good region is a diagonal band. For example, the black circle in the top-left corner sits at α=1/2\alpha=1/2 and λ=1/8×0.001\lambda=1/8\times0.001; changing either one alone makes things worse, while changing both together might even make them better. This is exactly the λ′=λ/α\lambda'=\lambda/\alpha of Proposition 1; when the learning rate changes, the best L2 coefficient changes with it. The authors suggest that SGD's reputation for being "sensitive to hyperparameters" may come partly from this coupling.

Top right, SGDW. The good region becomes a vertical strip, with most black circles at decay values between 1/4 and 1/2 (times 0.001), spanning five different learning rates. Even if the learning rate isn't tuned well, say stuck at 1/1024, tuning just the decay can still find a decent value.

Bottom left, Adam with L2. The best result is clearly worse than SGD's. Now look at where the black circles are. Settings with an L2 coefficient of 0 also make the top ten, so the best result with L2 is about the same as with none at all. In other words, L2 has almost no effect here, and the good region is diagonal as well.

Bottom right, AdamW. The good region is vertical, like SGDW's, with the black circles concentrated at decay values of 1/4 to 1/2; the best result is clearly better than Adam with L2, and on par with SGD and SGDW.

So this figure supports two things at once. After decoupling, the learning rate and weight decay can be tuned more or less separately; and once Adam switches to decoupled weight decay, it catches up with SGD on this task.

Better generalization, or just better training?

Four plots of a 26 2x96d ResNet trained on CIFAR-10 for 1800 epochs with an initial learning rate of 0.001. The top row shows training loss and test error curves for Adam (red) and AdamW (blue) under different decay settings; in the first half the two groups of curves cross each other, and in the second half the blue lines are generally lower. The bottom left plots final test error against the decay setting: Adam with L2 bottoms out at about 3.55%, AdamW at about 3.1%. The bottom right is a scatter plot of test error against training loss: at the same training loss, AdamW has lower test error than Adam; Adam's points bottom out around 3.5%, while AdamW's extend down to around 3.1%.
Original Figure 3, Loshchilov and Hutter, from PDF page 7, arXiv license, scholarly quotation.

The heatmaps cover only 100 epochs of training. The authors then train the larger 2x96d for 1800 epochs, with the initial learning rate fixed at Adam's default of 0.001 (the schedule is still cosine annealing), trying 12 coefficients for Adam with L2 and 7 normalized decay values for AdamW.

In the learning curves on the top row, the two groups often overlap in the first half, while in the second half AdamW's training loss and test error are usually lower. The bottom left shows the final test error, with Adam with L2 at best about 3.55% and AdamW at best about 3.1%.

The bottom right plot answers the question in this section's title. The horizontal axis is training loss and the vertical axis is test error; if AdamW were simply driving the training loss lower, both groups of points should fall on the same curve. In fact, at the same training loss, the blue AdamW points have lower test error than the red Adam ones. From this the authors argue that AdamW not only trains better but also generalizes better. The results on ImageNet32x32 are the same.

Warm restarts and anytime results

Two plots of test error curves. The left is CIFAR-10, trained for 1800 epochs: Adam (magenta) ends at about 3.6%, AdamW (blue) at about 3.1%, and SGDW (green) at about 2.9%; AdamWR (black) and SGDWR (red) each reach a low point just before the restarts at epochs 100, 300, 700, and 1500, about 4.0, 3.4, 3.1, and 3.2% for AdamWR and about 3.65, 3.15, 2.95, and 2.95% for SGDWR, with the error jumping up after each restart before falling again. The right is top-5 error on ImageNet32x32, trained for 150 epochs: Adam ends at about 17.7%, and the other four methods are between 15.3% and 16%, with an inset on the right showing the last 20 epochs in detail.
Original Figure 4, Loshchilov and Hutter, from PDF page 8, arXiv license, scholarly quotation.

The last comparison adds warm restarts. The left panel is CIFAR-10. The two lines with an R jump up at every restart and then quickly fall to a low point; before the first restart (epoch 100), AdamWR has already reached about 4.0%, and the authors say this makes AdamW up to 10 times faster. This is what the paper calls "anytime" performance; stop halfway through training, and you still have a decent model.

The paper reports that at the default learning rate of 0.001, AdamW's test error is 15% lower than Adam's in relative terms, on both CIFAR-10 and ImageNet32x32, and that AdamWR achieves the same results while being usable at any time. Reading off the left panel, Adam on CIFAR-10 ends at about 3.6% and AdamW at about 3.1%. AdamWR closes most of the gap between Adam and SGDWR, and on ImageNet32x32 the two are on par.

Other tasks and limitations

Section 4.5 collects reports from other teams that used it. A team working on EEG classification provided a direct comparison under the same hyperparameters, with Deep4Net going from 71.37% to 73.68% and ResNet from 61.34% to 72.04%; OpenAI's 2018 GPT trained its Transformer with AdamW; and Zhang et al. compared SGD, Adam, and K-FAC on CIFAR and found that whenever L2 and weight decay are not equivalent, decoupled weight decay does better.

The limitations are also clear. The main experiments are all image classification, and the authors write in their conclusion that these results still need to be verified on more tasks, especially ones where regularization matters a lot. The square-root rule for normalized weight decay also rests on only a few experiments.

The learning rate gets multiplied back in

Back to that lr * weight_decay from earlier.

In the paper's AdamW, the decay is ηtλ\eta_t\lambda; it follows only the schedule multiplier and is not multiplied by the base learning rate α\alpha. PyTorch's AdamW does θt←θt−1−γλθt−1\theta_t\leftarrow\theta_{t-1}-\gamma\lambda\theta_{t-1}, where γ\gamma is the actual learning rate including the schedule, so the decay is αηtλ\alpha\eta_t\lambda. Optax's adamw does the same; its documentation states explicitly that the weight decay here is multiplied by the learning rate, consistent with PyTorch but different from the paper, which multiplies only by the schedule multiplier. Keras's AdamW likewise subtracts variable * wd * lr from the parameters.

Converting between them, the paper's λ\lambda equals PyTorch's lr × weight_decay. PyTorch's defaults are lr=1e-3 and weight_decay=1e-2, so each step decays by 10−510^{-5}, equivalent to λ=10−5\lambda=10^{-5} in the paper's formulation. The default values also differ from one library to the next, 0.01 in PyTorch, 10−410^{-4} in Optax, and 0.004 in Keras, so when you move hyperparameters over, first check which formulation and which default you're dealing with.

This difference has a practical consequence. With PyTorch's formulation, changing the learning rate also changes the amount of decay, so the property from the heatmaps above, that the two hyperparameters can be tuned separately, no longer fully holds. In a February 2024 blog post, Fabian Schaipp demonstrated this on a synthetic ridge regression problem and recommended keeping lr × weight_decay constant when sweeping the learning rate with PyTorch's AdamW, so that doubling the learning rate halves the decay; he also mentioned that his experience training ViTs with timm was consistent with this, though he didn't release experiments for that part. Wang and Aitchison in 2024 looked at it from another angle, viewing the weights AdamW learns as an exponential moving average of recent updates, with a timescale of roughly 1/(lr×weight_decay)1/(\text{lr}\times\text{weight\_decay}) steps. On ResNet, ViT, and NanoGPT, they found that the optimal timescale measured in epochs stays roughly fixed, so as the dataset grows, weight decay should shrink. This points in the same direction as the paper's normalized weight decay.

Some frameworks do keep the paper's formulation, for example MosaicML Composer's DecoupledAdamW, where the decay is multiplied only by "the current learning rate divided by the initial learning rate," that is, the schedule multiplier. Also, PyTorch's Adam has its own weight_decay, which is L2; to get AdamW's behavior inside Adam, you have to set decoupled_weight_decay=True separately.

What came after

AdamW went on to become standard equipment for training Transformers. LLaMA uses AdamW with β2=0.95\beta_2=0.95 and weight decay 0.1; OLMo 2 even excludes the embedding layer from weight decay; DeepSeek-V4 hands most of its parameters to Muon, but still uses AdamW for the weights of the embedding layer, the prediction head, and the normalization layers.

Conclusion

This paper proves that L2 regularization and weight decay are equivalent only in SGD, and even then only with the coefficient converted according to the learning rate; in Adam, the L2 gradient gets divided by v^t\sqrt{\hat v_t}, so weights with large past gradients are regularized less. Move the decay out of the gradient and apply it directly to the parameters, and you have AdamW. The CIFAR-10 heatmaps show that this lets the learning rate and weight decay be tuned more or less separately, and that Adam's best results catch up with SGD's; the 1800-epoch comparison shows that at the same training loss, AdamW has lower test error. The evidence comes mainly from ResNets on image classification, and the normalization rule rests on only a few experiments.

When you use it in practice, first check which AdamW you're using. In PyTorch, Optax, and Keras, the decay is the learning rate times weight_decay, while the paper's and Composer's versions multiply only by the schedule multiplier; when moving settings over from the paper or another framework, remember to convert. When sweeping the learning rate in PyTorch, you can hold lr × weight_decay fixed, and keep in mind that torch.optim.Adam(weight_decay=...) is L2, through and through.

The first version of the paper was titled "Fixing Weight Decay Regularization in Adam," and the fixed version went on to become standard equipment for large language models. It's just that the most widespread implementations multiplied the learning rate back into the decay term.