Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Publication
- ICML 2015Conference
- First submitted to arXiv
- arXiv:1502.03167
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{normalizationbatchnorm2015,
title = {{Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift}},
author = {Sergey Ioffe and Christian Szegedy},
year = {2015},
eprint = {1502.03167},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/1502.03167}
}
Deep learning has made significant progress in various fields such as vision and speech. Stochastic Gradient Descent (SGD) and its variants like Momentum and Adagrad have proven to be highly effective methods for training deep neural networks. These optimization algorithms adjust the network parameters to minimize loss by processing small batches of data during training.
In the SGD process, instead of updating parameters with each individual training example, we use mini-batches. This approach has two main benefits: first, the gradient of the loss over a mini-batch is a good estimate of the gradient over the entire training set, and as the batch size increases, this estimate improves; second, modern computing platforms support parallel computation, making it more efficient to process a whole batch of data at once rather than multiple individual data points.
However, SGD requires precise tuning of hyperparameters such as the learning rate. As the network depth increases, small changes can be amplified, leading to the problem known as covariate shift.
Defining the Problem
The authors define internal covariate shift as:
- Changes in the distribution of network activations due to the updating of network parameters during training.
To improve training, we aim to reduce internal covariate shift. By stabilizing the distribution of layer inputs , we expect to accelerate training.
Past research indicates that network training converges faster if its inputs are whitened, i.e., transformed to have zero mean and unit variance, and are uncorrelated. Since the input to each layer comes from the output of the previous layer, it would be beneficial to whiten the input to each layer.
However, alternating these modifications with optimization steps might cause the gradient descent steps to update parameters in a way that necessitates re-normalization, reducing the effectiveness of gradient steps.
For instance, consider a layer with input plus a learned bias , normalized by subtracting the activation mean: , where , is the set of training values of , and . If the gradient descent step ignores the dependence of on , it updates , where .
Then . Thus, the combination of updating and subsequent normalization changes results in no change in the layer's output or loss. As training continues, grows indefinitely while the loss remains unchanged. If normalization involves both centering and scaling activations, the problem worsens.
The issue with the above approach is that gradient descent optimization does not account for the presence of normalization.
To solve this, the authors aim to ensure that the network always produces activations with the desired distribution for any parameter values. This allows the gradient of the loss with respect to model parameters to account for normalization and its dependence on model parameters .
Let be the layer input, treated as a vector, and the set of these inputs over the training dataset. Normalization can then be written as the transformation:
which depends not only on a given training example but also on all examples .
For backpropagation, we need to compute
and
Neglecting the latter leads to the issue mentioned above.
In this framework, calculating the covariance matrix and its inverse square root to produce whitened activations , and the derivatives of these transformations for backpropagation is very expensive. This motivated the authors to seek an alternative method to normalize inputs in a differentiable way without needing to analyze the entire training set after each parameter update.
Solving the Problem
Due to the high computational cost and potential differentiability issues of full whitening of each layer’s input, the authors made two necessary simplifications.
First, instead of jointly whitening the input and output features of a layer, they normalized each scalar feature independently to have zero mean and unit variance. For a layer with -dimensional input , each dimension is normalized as:
where the expectation and variance are computed based on the training dataset.
Note that simply normalizing each input of the layer might alter the layer's representation power. For example, normalizing inputs to a sigmoid function restricts it to its non-linear region. To address this, the authors introduced transformations within the network that can represent the identity transformation. To achieve this, they introduced parameters and for each activation value to scale and shift the normalized values:
These parameters are learned along with the original model parameters, restoring the network's representation power.
In fact, by setting and , the original activations can be recovered for fixed statistics with omitted. A fixed parameter pair need not undo the different statistics of every batch.
In the batch setting using the entire training set, activations are typically normalized using the entire dataset. However, this is impractical for stochastic optimization.
Therefore, the authors made a second simplification: using mini-batches during stochastic gradient training, each mini-batch produces an estimate of the mean and variance for each activation value. Thus, the statistics used for normalization can fully participate in gradient backpropagation. For convolutional layers, values in the same channel are pooled across the batch and spatial positions to estimate one channel's statistics. It is important to note that using mini-batches means computing variances per feature rather than a joint covariance matrix; in the joint case, regularization might be necessary, as the mini-batch size could be smaller than the number of activations to whiten, leading to a singular covariance matrix.
Consider a mini-batch of size . Since normalization is applied independently to each activation, let's focus on a specific activation and omit for simplicity. We have values of this activation in the mini-batch, .
Let be the normalized values, and the linear transformations of them. We call this transformation the BatchNorm transform , described as follows. In the algorithm, is a constant added to the mini-batch variance for numerical stability.
In the representation , the parameters and are learned, but note that the BN transform does not independently process activations in each training example. Instead, depends on both the training sample and the other samples in the mini-batch. The scaled and shifted values are propagated to other network layers.
BN includes its statistics in the computation graph, allowing differentiation through normalization, and uses a learnable affine transformation to adjust the output scale. These are the concrete operations; reducing internal covariate shift is the original paper’s explanation, not a proven exclusive mechanism.
Increasing Learning Rates
In the original experiments, BN enabled higher learning rates and improved training of sigmoid networks.
BN has a useful positive-scaling property. The following derivation assumes a fixed training batch, , and omission of the numerical stability term . With included, scale invariance is generally approximate.
It can be shown that
Under these assumptions, the input Jacobian is unchanged while the weight gradient scales by .
The authors further hypothesize that BatchNorm may cause the singular values of the Jacobian matrix of a layer to be close to 1, which is beneficial for training.
No Need for Dropout
When training with BatchNorm, a training sample is observed together with other samples in the mini-batch, so the network no longer produces deterministic values for a given training sample. In the authors' experiments, this effect benefitted network generalization. This allowed Dropout to be removed in some experiments, but it is not a universal rule; some ensemble models in the paper still used a small amount of Dropout.
Discussion
MNIST Experiment
To verify the impact of internal covariate shift on training and BatchNorm's ability to counteract it, the authors tested on the MNIST dataset for digit classification.
A very simple network was used here, with input being 28×28 grayscale digit images and three hidden layers, each with 100 activations. Each hidden layer computes , where is a sigmoid nonlinearity, and weights are initialized as small random Gaussian values.
After the last hidden layer, a fully connected layer with 10 activations (one per class) follows, using cross-entropy loss. The network was trained for 50,000 steps with mini-batches of 60 samples each. BatchNorm was added to each hidden layer.
The above figure shows the proportion of correct predictions on holdout test data as training progresses for both networks. The BatchNorm network achieved higher test accuracy. Panels (b) and (c) show the 15th, 50th and 85th percentiles of one typical sigmoid input in the last hidden layer over training. The original network's distribution changes substantially in location and spread; the BatchNorm network's percentiles remain more stable. This supports the authors' training explanation in this experiment.
ImageNet Experiment
The authors then applied BatchNorm to a new variant of the Inception network and trained it on the ImageNet classification task.
Experiments evaluated several modified versions of Inception with BatchNorm. In all cases, BatchNorm was applied to the input of each nonlinearity, leaving the rest of the architecture unchanged.
Simply adding BatchNorm to the network did not fully utilize this method. Therefore, the authors further modified the network and its training parameters:
- Increased Learning Rate: With BatchNorm, higher learning rates were achieved, speeding up training without adverse effects in these experiments.
- Removed Dropout: As BatchNorm shared some regularization goals with Dropout, removing Dropout from the modified BatchNorm Inception accelerated training without increasing overfitting.
- Reduced L2 Weight Regularization: In Inception, L2 loss on model parameters controlled overfitting, but in the modified BatchNorm Inception, this loss weight was reduced by 5 times.
- Accelerated Learning Rate Decay: As BatchNorm networks train faster than Inception, the learning rate decay was accelerated by 6 times.
- Removed Local Response Normalization: While Inception and other networks benefit from this, it was not needed with BatchNorm.
- More Thorough Shuffling of Training Samples: Shuffling training data within shards prevented the same samples from appearing in the same mini-batches. This led to about a 1 percentage-point improvement in validation accuracy, consistent with BatchNorm acting as a regularizer.
- Reduced Photometric Distortions: Since BatchNorm networks train faster and each training sample is observed fewer times, the training focused on more “real” images, reducing their distortions.
The authors evaluated the following networks, all trained on LSVRC2012 training data and tested on validation data:
- Inception: The architecture described initially, with an initial learning rate of 0.0015.
- BN-Baseline: Same as Inception but with BatchNorm applied before each nonlinearity.
- BN-x5: Inception with BatchNorm. The initial learning rate increased five times to 0.0075. The same learning rate increase pushed the original Inception model to machine limits.
- BN-x30: Similar to BN-x5 but with an initial learning rate of 0.045 (30 times that of Inception).
- BN-x5-Sigmoid: Similar to BN-x5 but using sigmoid nonlinearities instead of ReLU. The authors also tried training the original Inception with sigmoid, but the model accuracy remained random.
The above figure shows network validation accuracy as training progresses. Inception reached 72.2% accuracy after 31 million training steps. The table below shows the number of training steps each network took to reach the same 72.2% accuracy, the highest validation accuracy each network achieved, and the steps required to reach it.
Using only BatchNorm (BN-Baseline), the same accuracy as Inception was achieved in less than half the training steps. BN-x5 required about 1/14 as many steps to reach 72.2% accuracy than Inception. Interestingly, further increasing the learning rate (BN-x30) initially slowed down training but allowed for higher final accuracy. After 6 million steps, it reached 74.8%, about 1/5 as many steps as Inception needed to reach 72.2%.
Additionally, the authors observed that deep networks with BatchNorm could train even using sigmoid nonlinearities, achieving 69.8% accuracy with BN-x5-Sigmoid.
Conclusion
BatchNorm standardizes features using batch statistics, then applies a learned scale and shift. In the original experiments, it allowed higher learning rates, accelerated training and provided some regularization.
The authors explained this through reduced internal covariate shift. Later research (2018) offered smoother optimization geometry as an important alternative explanation. BN does not fix the entire input distribution, so its name should not be treated as proof of its mechanism.
Small batches can produce noisy statistics. Convolutional BN typically computes statistics per channel over both the batch and spatial positions, so effective sample size depends on more than batch size. RepVGG quantization difficulties also involve the weight and activation ranges after merging branches and cannot simply be attributed to small batches:
Finally, training and inference use different statistics—an easy engineering pitfall. Training normally uses the current batch; inference usually uses moving-average estimates accumulated during training, possibly followed by calibration, rather than recomputing statistics over the full training set. This can produce different behavior between the two stages. In practice, switch to evaluation mode, preserve the estimates and check the deployment data distribution so the model can work reliably in its intended setting.