Back to paper library
arXiv 2016ICLR 2017OptimizersPaper notes

SGDR

Have you tried restarting it?

Original paper

SGDR: Stochastic Gradient Descent with Warm Restarts

Paper authors

Ilya Loshchilov · Frank Hutter

Publication
ICLR 2017Conference
First submitted to arXiv
arXiv:1608.03983
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{optimizerssgdr2016,
  title = {{SGDR: Stochastic Gradient Descent with Warm Restarts}},
  author = {Ilya Loshchilov and Frank Hutter},
  year = {2016},
  eprint = {1608.03983},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1608.03983}
}
On this page

Open PyTorch's torch.optim.lr_scheduler, and the documentation of two schedulers points to the same paper.

One is CosineAnnealingWarmRestarts, whose name already spells out where it comes from. The other is CosineAnnealingLR, whose documentation cites the same paper but makes a point of adding that, although SGDR has periodic restarts, this implementation does cosine annealing without restarts.

For a paper with "restarts" in its title, the part that survived into the training setups of large models is mostly the half without restarts. Let's go back to the beginning and see what Ilya Loshchilov and Frank Hutter actually proposed in 2016, and what problem restarts were originally meant to solve.

If you haven't heard of these two authors, here is a bit of background. They were at the University of Freiburg in Germany at the time. Hutter's main research focus is hyperparameter optimization and AutoML; Loshchilov had done a good deal of earlier work on the evolution strategy CMA-ES, and when Section 2 of the paper reviews restart strategies, his own 2012 work is among those cited. In gradient-free optimization, "run for a while, then start over completely" is an old trick for dealing with multimodal functions, and a common practice is to double the population size at every restart. This paper brought the habit of restarting into SGD.

Defining the problem

In 2016, the mainstream way to train networks like ResNet was SGD with momentum, which is the paper's Eqs. (3) and (4):

vt+1=μtvt−ηt∇ft(xt),xt+1=xt+vt+1v_{t+1} = \mu_t v_t - \eta_t \nabla f_t(x_t), \qquad x_{t+1} = x_t + v_{t+1}

The authors say in the introduction that the hardest remaining parts of training this way are how to schedule the learning rate and how much weight decay to use. The baseline used in this paper, Wide ResNet (WRN), starts the learning rate at 0.1 and multiplies it by 0.2 at epochs 60, 120, and 160, training for 200 epochs in total.

This kind of step schedule has two drawbacks. First, where to drop and by how much are both hyperparameters to tune. Second, the model only gets good after the learning rate has come down; if you have to stop at epoch 100, the model you're holding is still stuck in the high-learning-rate phase, with a large error. The authors describe the latter with the term anytime performance, meaning that whenever you stop, you have a good enough model in hand.

The goal of this paper is to improve SGD's anytime performance. How should the learning rate be scheduled so that training can be stopped at any time, and still end up no worse than a well-tuned step schedule?

Solving the problem

Warm restarts

The authors' approach is to simulate a fresh start of SGD every TiT_i epochs. This restart doesn't retrain from scratch. The parameters xtx_t are kept exactly as they are and used as the new starting point; only the learning rate is pulled back up and then allowed to come down again. The paper also notes that how far the learning rate is raised determines how much of the previously accumulated information (such as momentum) is kept.

Cosine annealing

Within the ii-th run, the learning rate decreases batch by batch according to Eq. (5):

ηt=ηmini+12(ηmaxi−ηmini)(1+cos⁡(TcurTiπ))\eta_t = \eta^i_{min} + \frac{1}{2}\left(\eta^i_{max} - \eta^i_{min}\right)\left(1 + \cos\left(\frac{T_{cur}}{T_i}\pi\right)\right)

TcurT_{cur} is the number of epochs run since the last restart. It is updated at every batch, so it can take fractional values like 0.1 or 0.2. Let's plug in a few points:

  • Tcur=0T_{cur} = 0, cos⁡0=1\cos 0 = 1, and the learning rate is ηmax\eta_{max}.
  • Tcur=Ti/2T_{cur} = T_i/2, cos⁡(π/2)=0\cos(\pi/2) = 0, and the learning rate is exactly halfway between the two ends.
  • Tcur=TiT_{cur} = T_i, cos⁡π=−1\cos \pi = -1, and the learning rate drops to ηmin\eta_{min}.

So this curve falls slowly at first, is close to a straight line in the middle, and slows down again near the end, lingering at a low learning rate for a while. In the CIFAR experiments, ηmax\eta_{max} and ηmin\eta_{min} are fixed at 0.05 and 0 for every run. The authors mention that lowering these two values at each restart might help, but they didn't do it, to keep the number of hyperparameters down.

T0T_0 and TmultT_{mult}

The length of each period is set by two numbers. T0T_0 is the length of the first run, and after each restart the period is multiplied by TmultT_{mult}. Here is the calculation for the setting that appears most often in the paper, T0=10T_0 = 10 and Tmult=2T_{mult} = 2:

RunPeriod lengthEnds at epoch
11010
22030
34070
480150
5160310

Within a budget of 200 epochs, only the first four runs finish; the fifth ends halfway through. With T0=1T_0 = 1 and Tmult=2T_{mult} = 2 instead, the restarts fall at epochs 1, 3, 7, 15, 31, 63, and 127.

The point of doubling the period is to get a usable model quickly with very short periods first, then trade increasingly long periods for better results, which is the same idea as the evolution strategies mentioned earlier doubling the population at every restart. Another setting is Tmult=1T_{mult} = 1, restarting every T0T_0 epochs; with T0=200T_0 = 200, the whole of training is a single cosine with no restarts at all. This line will become very important later on.

A learning-rate chart drawn for this site, with epochs 0 to 200 on the horizontal axis and the learning rate from 0 to 0.06 on the vertical axis. The black dotted line is the default step schedule, staying at 0.05 until epoch 60 and then dropping in turn to 0.01, 0.002, and 0.0004. The blue solid line is SGDR with T0 = 10 and T_mult = 2, dropping to 0 at epochs 10, 30, 70, and 150 and immediately jumping back to 0.05, with each period longer than the last; the low points at epochs 30, 70, and 150 are marked with blue dots. The orange dashed line is a single-period cosine, falling smoothly from 0.05 to 0 at epoch 200.
Drawn for this note from Eq. (5) and the settings in Section 4.1 of the paper.

The paper plots these schedules on a log axis, and the authors themselves acknowledge that the log axis hides the cosine shape, so here we redraw them on a linear axis. The blue line jumps right back to 0.05 every time it hits 0, while the orange dashed line comes down slowly from 0.05 all the way to 0.

Which model do you hand over?

When the learning rate is suddenly raised, the model temporarily gets worse, so SGDR doesn't necessarily hand over the parameters from the last step. Before the first restart, the recommended solution is simply the latest xtx_t; after that, it is the end of the most recently completed period, the model at the point where the learning rate has just reached ηmin\eta_{min}. In the table above, at epoch 200 SGDR hands over the model from epoch 150. The authors stress that this rule doesn't need a separate validation split to pick the model.

Why restart

Section 2 reviews two lines of work. Gradient-free optimization uses restarts to deal with multimodal functions; gradient-based optimization mostly uses restarts to speed up convergence. For example, O'Donoghue and Candès (2012) found that Nesterov's accelerated method oscillates periodically when momentum overshoots, so they restart whenever the objective increases, or when the momentum direction forms an obtuse angle with the negative gradient.

The stochastic-gradient case is different. The loss and gradient of a single batch are too noisy for these criteria to carry over directly; they would first have to be averaged (once per epoch, for example). The authors chose the simplest version, restarting on a fixed schedule. In the discussion they are also quite cautious, saying the main purpose of SGDR is to improve anytime performance, and that they do not claim to have observed any effect related to multimodality.

Discussion

Experimental setup

The main experiments train WRN-28-10 (28 layers, with 10 times as many filters per layer as the original ResNet, about 36.5 million parameters) on CIFAR-10 and CIFAR-100. The authors first reproduce the settings of the original WRN paper: momentum 0.9, weight decay 0.0005, batch size 128, 200 epochs, with data augmentation by horizontal flips and random crops after 4-pixel reflection padding. It differs from the original setup in only two places, subtracting only the per-pixel mean without ZCA whitening, and using the plain momentum of Eqs. (3) and (4) instead of Nesterov momentum.

Single models

The top two rows of the paper's Figure 2. The top row is the test error of WRN-28-10 on CIFAR-10 (left) and CIFAR-100 (right), with epochs 0 to 200 on the horizontal axis; the blue and red lines are the default step schedule, and before the learning-rate drop at epoch 60, CIFAR-10 error is between 12% and 25% and CIFAR-100 error is above 35%; the SGDR lines in other colors look like staircases, and the period-doubling green stars and magenta triangles drop within a few dozen epochs to about 5% on CIFAR-10 and about 20% on CIFAR-100. The bottom row zooms the same results into the low-error range, 3% to 5% for CIFAR-10 and 18% to 21% for CIFAR-100.
Original Figure 2, Loshchilov and Hutter, from PDF page 5, arXiv license, scholarly quotation. Only the top two rows, for WRN-28-10, are excerpted.

Start with the top row. The blue and red lines are the default schedule. Before the learning-rate drop at epoch 60, their CIFAR-10 test error stays above 12%, and even after one drop it is still between 7% and 10%; on CIFAR-100, the red line even climbs between the two learning-rate drops. The SGDR lines plot the recommended solutions, which is why they look like staircases; the period-doubling green stars (T0=1T_0 = 1) and magenta triangles (T0=10T_0 = 10) fall into the low-error region within a few dozen epochs. The bottom row zooms in on the same results, showing which lines end higher or lower over the last few epochs.

The authors conclude that SGDR reaches test errors of about 4% on CIFAR-10 and about 20% on CIFAR-100 with 2 to 4 times fewer epochs. The final numbers are summarized below:

WRN-28-10 scheduleCIFAR-10CIFAR-100
Default, η₀ = 0.14.2420.33
Default, η₀ = 0.054.1320.21
T₀ = 50, T_mult = 14.1719.99
T₀ = 100, T_mult = 14.0719.87
T₀ = 200, T_mult = 13.8619.98
T₀ = 1, T_mult = 24.0919.74
T₀ = 10, T_mult = 24.0319.58

Compiled from Table 1 of the paper; test error (%), median of five runs.

Two things to watch for when reading this table. First, most of the differences are only a few tenths of a percentage point, each setting was run only five times, and the authors note in the table caption that the bold numbers just mark the better results, with no statistical test. Second, the best single model on CIFAR-10, at 3.86%, comes from T0=200T_0 = 200, Tmult=1T_{mult} = 1, a single complete cosine with no restarts at all. The authors take this as a sign that WRN's original step schedule may not be the best choice; but this setting also has the worst anytime performance, catching up only in the last few epochs.

The authors spent the time saved on training the wider WRN-28-20 (145.8 million parameters, about 3 to 4 times the compute). CIFAR-10 improved by about 0.25 percentage points and CIFAR-100 by 0.5 to 1 percentage point, with T0=10T_0 = 10, Tmult=2T_{mult} = 2 reaching 3.74% and 18.70%. Looking at just the first 50 epochs, WRN-28-20 paired with T0=50T_0 = 50, before any restart, was already below 19% on CIFAR-100, while WRN-28-10 never did better than 19.5% with any schedule.

The training curves in the appendix add one more observation. Up to about epoch 120, SGDR's training loss falls faster than the default schedule's; after that, the default schedule starts to overfit, with the test loss climbing, while SGDR overfits only slightly.

Free snapshot ensembles

After the first arXiv version of SGDR came out, Huang et al.'s Snapshot Ensembles used its periods to build ensembles. Just before each restart, the learning rate has just bottomed out and the model has settled at a good position; save these snapshots, and you can assemble a set of models at no extra training cost. The final version of this paper goes back and checks whether the idea holds up on WRN.

The procedure is to train WRN-28-10 with SGDR at T0=10T_0 = 10, Tmult=2T_{mult} = 2 and save snapshots at epochs 30, 70, and 150, the three blue dots on the schedule chart. At inference time, the softmax output probabilities of the snapshots are averaged with equal weights.

The two heatmaps of the paper's Figure 3, CIFAR-10 on the left and CIFAR-100 on the right, with the median test error of the ensemble in each cell. The horizontal axis is the number of independent training runs N (1, 2, 3, 4, 8, 16), and the vertical axis is the number of snapshots taken per run M (1, 2, 3). For CIFAR-10, the bottom-left cell, N = 1 and M = 1, is 4.03%; going up to M = 3 gives 3.51%; the top-right cell, N = 16 and M = 3, is 3.14%. The corresponding values for CIFAR-100 are 19.57%, 17.75%, and 16.21%. Darker colors mean lower error.
Original Figure 3, Loshchilov and Hutter, from PDF page 7, arXiv license, scholarly quotation.

The horizontal axis is how many independent training runs (NN), and the vertical axis is how many snapshots are taken from each run (MM). The bottom-left corner is a single model, 4.03% on CIFAR-10 and 19.57% on CIFAR-100 (the median of 16 runs). Going up to M=3M = 3, the three snapshots from a single run already bring this down to 3.51% and 17.75%. Getting a similar effect from independent training takes N=3N = 3 runs (3.51% and 17.58%), so in this setting the snapshots cut the training needed roughly threefold.

Where the snapshots come from matters too. Ensembling only the models from the last three epochs (148, 149, 150) gives the same result as using only the model from epoch 150, with no improvement. The snapshots taken before restarts help because they come from different periods, so their predictions differ from one another.

The two approaches can be stacked. Three training runs with 3 snapshots each bring the error down to 3.25% and 16.64%; 16 runs with 3 snapshots each give the 3.14% and 16.21% from the abstract, the paper's claimed state of the art on CIFAR at the time, though that is an ensemble of 48 models. The two datasets don't follow quite the same trend either. On CIFAR-10, 9 snapshots (N=3N = 3, M=3M = 3) at 3.25% are already on par with 16 independent models at 3.27%; on CIFAR-100, 16 independent models at 16.50% are still slightly better than 9 snapshots at 16.64%.

EEG and downsampled ImageNet

To show that the method isn't limited to CIFAR, the authors ran two more sets of experiments.

The first uses electroencephalography (EEG) data, from 14 subjects with about 1,000 trials each, where the task is to classify actual movements of the left hand, right hand, and feet, using the convolutional network designed by Schirrmeister et al. The baseline schedule has to fix the total number of epochs in advance (from 30 to 480) and then cut the learning rate by a factor of 10 at fixed proportional points; the larger the budget, the better the result. SGDR doesn't need the budget decided in advance; its final performance is about the same as the baseline, and its performance along the way is better. Ensembling snapshots also works. Against a median reference error of about 9%, taking snapshots from a single training run lowers it by another 1 to 2 percentage points, and combining snapshots from two learning-rate settings lowers it by 2 to 3 percentage points.

The second uses a downsampled ImageNet the authors made themselves, with the same 1,000 classes and the same images as the original, only shrunk to 32 × 32 with box downsampling (this dataset was later published separately in 2017). Each image has two orders of magnitude fewer pixels on average, so it is harder than the original ImageNet; with more classes, and objects that often occupy only a small part of the frame, it is also harder than CIFAR. On WRN-28-10, they compare the default schedule with two SGDR settings, each with 4 initial learning rates (0.05, 0.025, 0.01, 0.005). SGDR with T0=10T_0 = 10, Tmult=2T_{mult} = 2, ηmax=0.01\eta_{max} = 0.01 reaches top-1 and top-5 errors of 39.24% and 17.17%, comparable to AlexNet's 40.7% and 18.2% on full-resolution ImageNet. This experiment ran for only about 40 epochs, and the authors themselves call it a preliminary result; the comparison with AlexNet crosses resolutions and models, so it only serves as a rough point of reference.

The authors' reading of these results is that the choice of initial learning rate matters a lot, and since every SGDR period sweeps all the way from the initial value down to 0, it effectively tries out a whole range of learning rates, softening the damage from picking the wrong initial value.

Are restarts really the point?

The best single models on CIFAR-10, WRN-28-10 at 3.86% and WRN-28-20 at 3.66%, both come from T0=200T_0 = 200 without restarts. The authors also write in the discussion that even without any restarts, the aggressive schedule of Eq. (5) is quite competitive with the default schedule, and it needs only two hyperparameters, the initial learning rate and the total number of epochs.

Later research made this clearer. Gotmare et al. (ICLR 2019) used mode connectivity to analyze SGDR's trajectory, training VGG-16 on CIFAR-10 with the same T0=10T_0 = 10, Tmult=2T_{mult} = 2 schedule. Between the models at epochs 30 and 150, and at epochs 70 and 150, the training loss along the connecting path is indeed higher than at the two ends, meaning the parameters crossed a barrier after the restart; but the models at epochs 70 and 150 are neither local minima nor, by the look of it, in different basins, so the "escaping local minima" story may be oversimplified. In a footnote they add that what consistently worked well in their experiments was a single cosine (or linear) decay over the whole budget, and they speculate that the decay is what really does the work, with restarts contributing less.

That is also how people use it in practice. fast.ai promoted SGDR in its 2017 course through cycle_len and cycle_mult; by fastai v1 at the end of 2018, a maintainer replied on the forums that this API no longer worked and that the default had changed to 1cycle. Later large-scale training mostly runs just a single cosine period:

  • Bag of Tricks (He et al., 2018) uses a "simplified version" on ImageNet, with the learning rate following a cosine from its initial value all the way down to 0 (warmup counted separately).
  • GPT-3 uses a cosine to bring the learning rate down to 10% of its original value over 260 billion tokens, and then holds it at 10%; the first 375 million tokens are a linear warmup.
  • LLaMA's cosine schedule sets the final learning rate to 10% of the maximum, with 2,000 warmup steps.

Run only a single period, though, and you need to know the total number of steps in advance, which brings back the very problem SGDR set out to address. The Chinchilla appendix quantified the cost, finding that setting the cosine period more than 25% longer than the actual number of training steps makes the model noticeably worse. The later WSD (a three-phase schedule of warmup, stable, and decay), WSM, and Schedule-Free all deal with exactly this.

Conclusion

SGDR writes the learning-rate schedule as a single formula. Each period descends along a cosine from ηmax\eta_{max} to ηmin\eta_{min}, the period lengths are set by T0T_0 and TmultT_{mult}, and when a period ends, the learning rate is pulled back up to start again. On CIFAR, it reaches errors comparable to a step schedule with 2 to 4 times fewer epochs, and the snapshots taken before restarts can form an ensemble for free; meanwhile, the best single model on CIFAR-10 comes from a single cosine without restarts.

If you know how long you'll be training in total, the simplest choice is one complete cosine period, with T0T_0 equal to the total number of steps and warmup added in front if needed; CosineAnnealingLR is exactly that version. Only when you want snapshot ensembles, or genuinely don't know your budget, is it worth bringing out CosineAnnealingWarmRestarts; and in that case, remember to follow the paper's rule and hand over the model from the end of a period, not from the last step.

The names of these two PyTorch schedulers have already made the choice for you. If you're not planning to restart, just pick the one with the shorter name.