{"notes":[{"id":"normalization/batchnorm","category":"normalization","legacyPath":"normalization/1502-batchnorm","legacySlug":"1502-batchnorm","legacyPaths":[],"title":"BatchNorm","subtitle":"Batch Normalization","author":"Z. Yuan","paperDate":"2015-02-11","publishedOn":"2024-08-01","paperTitle":"Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift","paperUrl":"https://arxiv.org/abs/1502.03167","bibliography":{"title":"Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift","authors":["Sergey Ioffe","Christian Szegedy"],"url":"https://arxiv.org/abs/1502.03167","pdfUrl":"https://arxiv.org/pdf/1502.03167","codeUrl":null,"preprint":{"id":"1502.03167","submittedOn":"2015-02-11","url":"https://arxiv.org/abs/1502.03167"},"publication":{"venue":"ICML","year":2015,"kind":"conference","url":"https://proceedings.mlr.press/v37/ioffe15.html"},"citation":"@misc{normalizationbatchnorm2015,\n  title = {{Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift}},\n  author = {Sergey Ioffe and Christian Szegedy},\n  year = {2015},\n  eprint = {1502.03167},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1502.03167}\n}\n","citationPath":"/content/papers/citations/normalization/batchnorm.bfa8abb33f32.bib"},"thumbnail":{"src":"/content/papers/assets/48b236d4b897164b93764eccfe22db29e5b9bce290d864336b8f12ec127c1e4b.jpg","width":1016,"height":824},"locale":"en","path":"/content/papers/en/normalization/batchnorm.149073a22595.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"increasing-learning-rates","text":"Increasing Learning Rates","depth":3},{"id":"no-need-for-dropout","text":"No Need for Dropout","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"mnist-experiment","text":"MNIST Experiment","depth":3},{"id":"imagenet-experiment","text":"ImageNet Experiment","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"normalization/layernorm","relation":"next","topic":"Choosing the normalization axes"}]},{"id":"normalization/layernorm","category":"normalization","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"LayerNorm","subtitle":"Compute Along a Different Axis","author":"Z. Yuan","paperDate":"2016-07-21","publishedOn":"2025-02-08","paperTitle":"Layer Normalization","paperUrl":"https://arxiv.org/abs/1607.06450","bibliography":{"title":"Layer Normalization","authors":["Jimmy Lei Ba","Jamie Ryan Kiros","Geoffrey E. Hinton"],"url":"https://arxiv.org/abs/1607.06450","pdfUrl":"https://arxiv.org/pdf/1607.06450v1","codeUrl":"https://github.com/ryankiros/layer-norm","preprint":{"id":"1607.06450","submittedOn":"2016-07-21","url":"https://arxiv.org/abs/1607.06450"},"publication":null,"citation":"@misc{normalizationlayernorm2016,\n  title = {{Layer Normalization}},\n  author = {Jimmy Lei Ba and Jamie Ryan Kiros and Geoffrey E. Hinton},\n  year = {2016},\n  eprint = {1607.06450},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1607.06450}\n}\n","citationPath":"/content/papers/citations/normalization/layernorm.ad656f4b892f.bib"},"thumbnail":{"src":"/content/papers/assets/6071a1dfaed86ed9aa6c64eb8a8585277c16a70a65d1cdd32b9157be5a99c1da.png","width":1500,"height":625},"locale":"en","path":"/content/papers/en/normalization/layernorm.08513521ce61.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"work-through-the-first-row","text":"Work Through the First Row","depth":3},{"id":"lets-write-some-code","text":"Let's Write Some Code!","depth":3},{"id":"remove-the-scale-what-remains","text":"Remove the Scale: What Remains?","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"who-is-affected-by-smaller-batches","text":"Who Is Affected by Smaller Batches?","depth":3},{"id":"rnns-already-have-batchnorm-does-layernorm-still-help","text":"RNNs Already Have BatchNorm. Does LayerNorm Still Help?","depth":3},{"id":"faster-training-how-does-retrieval-fare","text":"Faster Training: How Does Retrieval Fare?","depth":3},{"id":"what-about-convolutional-networks","text":"What About Convolutional Networks?","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"normalization/batchnorm","relation":"background","topic":"Choosing the normalization axes"}]}],"text":"Deep learning has made significant progress in various fields such as vision and speech. Stochastic Gradient Descent (SGD) and its variants like Momentum and Adagrad have proven to be highly effective methods for training deep neural networks. These optimization algorithms adjust the network parameters to minimize loss by processing small batches of data during training.\n\nIn the SGD process, instead of updating parameters with each individual training example, we use mini-batches. This approach has two main benefits: first, the gradient of the loss over a mini-batch is a good estimate of the gradient over the entire training set, and as the batch size increases, this estimate improves; second, modern computing platforms support parallel computation, making it more efficient to process a whole batch of data at once rather than multiple individual data points.\n\nHowever, SGD requires precise tuning of hyperparameters such as the learning rate. As the network depth increases, small changes can be amplified, leading to the problem known as **covariate shift**.\n\n\n> [!TIP]\n> \n> This means that \"current layers\" need to continuously adapt to the changing input distribution caused by parameter updates in \"previous layers\".\n> \n> Consider a layer with a sigmoid activation function $g(x) = \\frac{1}{1+\\exp(-x)}$:\n> \n> $$z = g(Wu + b)$$\n> \n> where $u$ is the layer input, and the weight matrix $W$ and bias vector $b$ are learnable parameters.\n> \n> As $|x|$ increases, $g'(x)$ tends to zero, meaning that except for dimensions where $x = Wu + b$ has small absolute values, the gradient vanishes when flowing towards $u$, slowing down model training. However, since $x$ is influenced by $W$, $b$, and parameters of all preceding layers, parameter changes during training might push multiple dimensions of $x$ into the non-linear saturation regions, decelerating convergence.\n\n\n## Defining the Problem\n\nThe authors define internal covariate shift as:\n\n- **Changes in the distribution of network activations due to the updating of network parameters during training.**\n\nTo improve training, we aim to reduce internal covariate shift. By stabilizing the distribution of layer inputs $x$, we expect to accelerate training.\n\nPast research indicates that network training converges faster if its inputs are whitened, i.e., transformed to have zero mean and unit variance, and are uncorrelated. Since the input to each layer comes from the output of the previous layer, it would be beneficial to whiten the input to each layer.\n\nHowever, alternating these modifications with optimization steps might cause the gradient descent steps to update parameters in a way that necessitates re-normalization, reducing the effectiveness of gradient steps.\n\nFor instance, consider a layer with input $u$ plus a learned bias $b$, normalized by subtracting the activation mean: $\\hat{x} = x - E[x]$, where $x = u + b$, $X = \\{x_1,\\ldots,x_N\\}$ is the set of training values of $x$, and $E[x] = \\frac{1}{N} \\sum_{i=1}^N x_i$. If the gradient descent step ignores the dependence of $E[x]$ on $b$, it updates $b \\leftarrow b + \\Delta b$, where $\\Delta b \\propto -\\frac{\\partial \\ell}{\\partial \\hat{x}}$.\n\nThen $u + (b + \\Delta b) - E[u + (b + \\Delta b)] = u + b - E[u + b]$. Thus, the combination of updating $b$ and subsequent normalization changes results in no change in the layer's output or loss. As training continues, $b$ grows indefinitely while the loss remains unchanged. If normalization involves both centering and scaling activations, the problem worsens.\n\n---\n\nThe issue with the above approach is that gradient descent optimization does not account for the presence of normalization.\n\nTo solve this, the authors aim to ensure that the network always produces activations with the desired distribution for any parameter values. This allows the gradient of the loss with respect to model parameters to account for normalization and its dependence on model parameters $\\Theta$.\n\nLet $x$ be the layer input, treated as a vector, and $X$ the set of these inputs over the training dataset. Normalization can then be written as the transformation:\n\n$$\n\\hat{x} = \\text{Norm}(x, X)\n$$\n\nwhich depends not only on a given training example $x$ but also on all examples $X$.\n\nFor backpropagation, we need to compute\n\n$$\n\\frac{\\partial \\text{Norm}(x, X)}{\\partial x}\n$$\n\nand\n\n$$\n\\frac{\\partial \\text{Norm}(x, X)}{\\partial X}\n$$\n\nNeglecting the latter leads to the issue mentioned above.\n\nIn this framework, calculating the covariance matrix $\\text{Cov}[x] = E_{x \\in X}[xx^T] - E[x]E[x]^T$ and its inverse square root to produce whitened activations $\\text{Cov}[x]^{-1/2}(x - E[x])$, and the derivatives of these transformations for backpropagation is very expensive. This motivated the authors to seek an alternative method to normalize inputs in a differentiable way without needing to analyze the entire training set after each parameter update.\n\n\n> [!TIP]\n> \n> **Whitening vs. Standardization**\n> \n> **Standardization** usually acts on each feature separately. Z-score normalization subtracts its mean and divides by its standard deviation:\n> \n> $$\n> z = \\frac{x-\\mu}{\\sigma}\n> $$\n> \n> Here $x$ is the original value, $\\mu$ its mean and $\\sigma$ its standard deviation. This puts features on comparable scales, which can help scale-sensitive models such as neural networks or support vector machines. It gives zero mean and unit standard deviation when the variance is nonzero, but does not turn an arbitrary distribution into a Gaussian or remove correlations between features.\n> \n> **Whitening** goes further by removing linear correlations between features. A typical process is:\n> \n> 1. Center the data by subtracting its mean.\n> 2. Calculate the covariance matrix.\n> 3. Decompose the covariance matrix into eigenvectors and eigenvalues.\n> 4. Rotate and rescale using those vectors and values so the resulting covariance is the identity.\n> \n> This takes more matrix work than simple standardization. Singular covariances require regularization or removal of zero-eigenvalue directions.\n> \n> Whitened features are uncorrelated, not necessarily statistically independent. Those properties coincide under extra assumptions such as joint Gaussianity. PCA components also need eigenvalue-based rescaling to constitute PCA whitening. That is why the extra step can be useful in some methods without being necessary for every model.\n> \n> BN standardizes individual features rather than fully whitening them.\n\n\n## Solving the Problem\n\nDue to the high computational cost and potential differentiability issues of full whitening of each layer’s input, the authors made two necessary simplifications.\n\nFirst, instead of jointly whitening the input and output features of a layer, they normalized each scalar feature independently to have zero mean and unit variance. For a layer with $d$-dimensional input $x = (x^{(1)}, \\ldots, x^{(d)})$, each dimension is normalized as:\n\n$$\n\\hat{x}^{(k)} = \\frac{x^{(k)} - \\mathbb{E}[x^{(k)}]}{\\sqrt{\\mathrm{Var}[x^{(k)}]}}\n$$\n\nwhere the expectation and variance are computed based on the training dataset.\n\nNote that simply normalizing each input of the layer might alter the layer's representation power. For example, normalizing inputs to a sigmoid function restricts it to its non-linear region. To address this, the authors introduced transformations within the network that can represent the identity transformation. To achieve this, they introduced parameters $\\gamma^{(k)}$ and $\\beta^{(k)}$ for each activation value $x^{(k)}$ to scale and shift the normalized values:\n\n$$\ny^{(k)} = \\gamma^{(k)} \\hat{x}^{(k)} + \\beta^{(k)}\n$$\n\nThese parameters are learned along with the original model parameters, restoring the network's representation power.\n\nIn fact, by setting $\\gamma^{(k)} = \\sqrt{\\mathrm{Var}[x^{(k)}]}$ and $\\beta^{(k)} = \\mathbb{E}[x^{(k)}]$, the original activations can be recovered for fixed statistics with $\\epsilon$ omitted. A fixed parameter pair need not undo the different statistics of every batch.\n\nIn the batch setting using the entire training set, activations are typically normalized using the entire dataset. However, this is impractical for stochastic optimization.\n\nTherefore, the authors made a second simplification: using mini-batches during stochastic gradient training, each mini-batch produces an estimate of the mean and variance for each activation value. Thus, the statistics used for normalization can fully participate in gradient backpropagation. For convolutional layers, values in the same channel are pooled across the batch and spatial positions to estimate one channel's statistics. It is important to note that using mini-batches means computing variances per feature rather than a joint covariance matrix; in the joint case, regularization might be necessary, as the mini-batch size could be smaller than the number of activations to whiten, leading to a singular covariance matrix.\n\nConsider a mini-batch $B$ of size $m$. Since normalization is applied independently to each activation, let's focus on a specific activation $x^{(k)}$ and omit $k$ for simplicity. We have $m$ values of this activation in the mini-batch, $B = \\{x_1, \\ldots, x_m\\}$.\n\nLet $\\hat{x}_1, \\ldots, \\hat{x}_m$ be the normalized values, and $y_1, \\ldots, y_m$ the linear transformations of them. We call this transformation the BatchNorm transform $BN_{\\gamma, \\beta}$, described as follows. In the algorithm, $\\epsilon$ is a constant added to the mini-batch variance for numerical stability.\n\n\n\n![bn algo](/content/papers/assets/48b236d4b897164b93764eccfe22db29e5b9bce290d864336b8f12ec127c1e4b.jpg)\n\n\n\nIn the representation $y = BN_{\\gamma, \\beta}(x)$, the parameters $\\gamma$ and $\\beta$ are learned, but note that the BN transform does not independently process activations in each training example. Instead, $BN_{\\gamma, \\beta}(x)$ depends on both the training sample and the other samples in the mini-batch. The scaled and shifted values $y$ are propagated to other network layers.\n\nBN includes its statistics in the computation graph, allowing differentiation through normalization, and uses a learnable affine transformation to adjust the output scale. These are the concrete operations; reducing internal covariate shift is the original paper’s explanation, not a proven exclusive mechanism.\n\n### Increasing Learning Rates\n\nIn the original experiments, BN enabled higher learning rates and improved training of sigmoid networks.\n\nBN has a useful positive-scaling property. The following derivation assumes a fixed training batch, $a>0$, and omission of the numerical stability term $\\epsilon$. With $\\epsilon$ included, scale invariance is generally approximate.\n\n$$\n\\mathrm{BN}(W\\mathbf{u}) = \\mathrm{BN}((aW)\\mathbf{u})\n$$\n\nIt can be shown that\n\n$$\n\\frac{\\partial \\mathrm{BN}((aW)\\mathbf{u})}{\\partial \\mathbf{u}} = \\frac{\\partial \\mathrm{BN}(W\\mathbf{u})}{\\partial \\mathbf{u}}\n$$\n\n$$\n\\frac{\\partial \\mathrm{BN}((aW)\\mathbf{u})}{\\partial (aW)} = \\frac{1}{a} \\cdot \\frac{\\partial \\mathrm{BN}(W\\mathbf{u})}{\\partial W}\n$$\n\nUnder these assumptions, the input Jacobian is unchanged while the weight gradient scales by $1/a$.\n\nThe authors further hypothesize that BatchNorm may cause the singular values of the Jacobian matrix of a layer to be close to 1, which is beneficial for training.\n\n\n> [!TIP]\n> \n> Consider two consecutive layers with normalized inputs and the transformation between these normalized vectors: $\\hat{z} = F(\\hat{x})$.\n> \n> Assuming additionally that the components within each of $\\hat{x}$ and $\\hat{z}$ have been whitened, an additional assumption beyond BN, and $F(\\hat{x}) \\approx J\\hat{x}$ is a linear transformation for given model parameters, both $\\hat{x}$ and $\\hat{z}$ have unit covariance, and\n> \n> $$\n> I = \\mathrm{Cov}[\\hat{z}] = J\\mathrm{Cov}[\\hat{x}]J^T = JJ^T\n> $$\n> \n> Thus, $JJ^T = I$, meaning all nonzero singular values of $J$ are 1, preserving gradient magnitude during backpropagation.\n> \n> In practice, transformations are not linear, and normalized values are not guaranteed to be Gaussian-distributed or independent, but the authors still expect BatchNorm to improve gradient propagation. The specific impact of the algorithm on gradient propagation requires further study.\n\n\n### No Need for Dropout\n\nWhen training with BatchNorm, a training sample is observed together with other samples in the mini-batch, so the network no longer produces deterministic values for a given training sample. In the authors' experiments, this effect benefitted network generalization. This allowed Dropout to be removed in some experiments, but it is not a universal rule; some ensemble models in the paper still used a small amount of Dropout.\n\n## Discussion\n\n### MNIST Experiment\n\n\n\n![mnist](/content/papers/assets/35c003397d465148dd3115a282857806845d10e804c8e34a3e642c608d6c07fd.jpg)\n\n\n\nTo verify the impact of internal covariate shift on training and BatchNorm's ability to counteract it, the authors tested on the MNIST dataset for digit classification.\n\nA very simple network was used here, with input being 28×28 grayscale digit images and three hidden layers, each with 100 activations. Each hidden layer computes $y = g(Wu + b)$, where $g$ is a sigmoid nonlinearity, and weights $W$ are initialized as small random Gaussian values.\n\nAfter the last hidden layer, a fully connected layer with 10 activations (one per class) follows, using cross-entropy loss. The network was trained for 50,000 steps with mini-batches of 60 samples each. BatchNorm was added to each hidden layer.\n\nThe above figure shows the proportion of correct predictions on holdout test data as training progresses for both networks. The BatchNorm network achieved higher test accuracy. Panels (b) and (c) show the **15th, 50th and 85th percentiles** of one typical sigmoid input in the last hidden layer over training. The original network's distribution changes substantially in location and spread; the BatchNorm network's percentiles remain more stable. This supports the authors' training explanation in this experiment.\n\n### ImageNet Experiment\n\n\n\n![imagenet](/content/papers/assets/18102eb7995579cb2867a67bacd99614bbd8154b1ae42aba273c4bbf1343a624.jpg)\n\n\n\nThe authors then applied BatchNorm to a new variant of the Inception network and trained it on the ImageNet classification task.\n\nExperiments evaluated several modified versions of Inception with BatchNorm. In all cases, BatchNorm was applied to the input of each nonlinearity, leaving the rest of the architecture unchanged.\n\nSimply adding BatchNorm to the network did not fully utilize this method. Therefore, the authors further modified the network and its training parameters:\n\n- **Increased Learning Rate**: With BatchNorm, higher learning rates were achieved, speeding up training without adverse effects in these experiments.\n- **Removed Dropout**: As BatchNorm shared some regularization goals with Dropout, removing Dropout from the modified BatchNorm Inception accelerated training without increasing overfitting.\n- **Reduced L2 Weight Regularization**: In Inception, L2 loss on model parameters controlled overfitting, but in the modified BatchNorm Inception, this loss weight was reduced by 5 times.\n- **Accelerated Learning Rate Decay**: As BatchNorm networks train faster than Inception, the learning rate decay was accelerated by 6 times.\n- **Removed Local Response Normalization**: While Inception and other networks benefit from this, it was not needed with BatchNorm.\n- **More Thorough Shuffling of Training Samples**: Shuffling training data within shards prevented the same samples from appearing in the same mini-batches. This led to about a 1 percentage-point improvement in validation accuracy, consistent with BatchNorm acting as a regularizer.\n- **Reduced Photometric Distortions**: Since BatchNorm networks train faster and each training sample is observed fewer times, the training focused on more “real” images, reducing their distortions.\n\nThe authors evaluated the following networks, all trained on LSVRC2012 training data and tested on validation data:\n\n- **Inception**: The architecture described initially, with an initial learning rate of 0.0015.\n- **BN-Baseline**: Same as Inception but with BatchNorm applied before each nonlinearity.\n- **BN-x5**: Inception with BatchNorm. The initial learning rate increased five times to 0.0075. The same learning rate increase pushed the original Inception model to machine limits.\n- **BN-x30**: Similar to BN-x5 but with an initial learning rate of 0.045 (30 times that of Inception).\n- **BN-x5-Sigmoid**: Similar to BN-x5 but using sigmoid nonlinearities $g(t) = \\frac{1}{1+\\exp(-t)}$ instead of ReLU. The authors also tried training the original Inception with sigmoid, but the model accuracy remained random.\n\nThe above figure shows network validation accuracy as training progresses. Inception reached 72.2% accuracy after 31 million training steps. The table below shows the number of training steps each network took to reach the same 72.2% accuracy, the highest validation accuracy each network achieved, and the steps required to reach it.\n\n\n\n![imagenet table](/content/papers/assets/d7457b4b27f0871365e027c9cc2a62593c867ebb512fcfb21be1b0a0d9f8367e.jpg)\n\n\n\nUsing only BatchNorm (BN-Baseline), the same accuracy as Inception was achieved in less than half the training steps. BN-x5 required about 1/14 as many steps to reach 72.2% accuracy than Inception. Interestingly, further increasing the learning rate (BN-x30) initially slowed down training but allowed for higher final accuracy. After 6 million steps, it reached 74.8%, about 1/5 as many steps as Inception needed to reach 72.2%.\n\nAdditionally, the authors observed that deep networks with BatchNorm could train even using sigmoid nonlinearities, achieving 69.8% accuracy with BN-x5-Sigmoid.\n\n\n> [!TIP]\n> \n> Everyone knows training networks with sigmoid is incredibly difficult. Without BatchNorm, Inception with sigmoid never surpassed 1/1000 accuracy.\n\n\n## Conclusion\n\nBatchNorm standardizes features using batch statistics, then applies a learned scale and shift. In the original experiments, it allowed higher learning rates, accelerated training and provided some regularization.\n\nThe authors explained this through reduced internal covariate shift. [Later research (2018)](https://arxiv.org/abs/1805.11604) offered smoother optimization geometry as an important alternative explanation. BN does not fix the entire input distribution, so its name should not be treated as proof of its mechanism.\n\nSmall batches can produce noisy statistics. Convolutional BN typically computes statistics per channel over both the batch and spatial positions, so effective sample size depends on more than batch size. RepVGG quantization difficulties also involve the weight and activation ranges after merging branches and cannot simply be attributed to small batches:\n\n- [**[22.12] QARepVGG: Making RepVGG Great Again**](/blog/papers/reparameterization/qarepvgg)\n\nFinally, training and inference use different statistics—an easy engineering pitfall. Training normally uses the current batch; inference usually uses moving-average estimates accumulated during training, possibly followed by calibration, rather than recomputing statistics over the full training set. This can produce different behavior between the two stages. In practice, switch to evaluation mode, preserve the estimates and check the deployment data distribution so the model can work reliably in its intended setting.\n","imageSizes":{"/content/papers/assets/48b236d4b897164b93764eccfe22db29e5b9bce290d864336b8f12ec127c1e4b.jpg":{"width":1016,"height":824},"/content/papers/assets/35c003397d465148dd3115a282857806845d10e804c8e34a3e642c608d6c07fd.jpg":{"width":1224,"height":440},"/content/papers/assets/18102eb7995579cb2867a67bacd99614bbd8154b1ae42aba273c4bbf1343a624.jpg":{"width":1116,"height":624},"/content/papers/assets/d7457b4b27f0871365e027c9cc2a62593c867ebb512fcfb21be1b0a0d9f8367e.jpg":{"width":1064,"height":408}}}
