Layer Normalization
- Publication
- Publication venue not yet verified
- First submitted to arXiv
- arXiv:1607.06450
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{normalizationlayernorm2016,
title = {{Layer Normalization}},
author = {Jimmy Lei Ba and Jamie Ryan Kiros and Geoffrey E. Hinton},
year = {2016},
eprint = {1607.06450},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/1607.06450}
}
The previous note on BatchNorm left us with two questions. Small batches make statistical estimates very unstable, and inference requires switching to statistics accumulated during training.
Can we still normalize if we receive just one sample at a time?
Defining the Problem
Suppose a fully connected layer produces four numbers before its activation function.
Feeding it two samples at a time gives us a matrix of shape . is the batch size, and is the feature dimension of this layer.
BatchNorm fixes a feature position and computes the mean and variance from different samples at that position. A sample's output therefore depends on the other samples in its batch. With only one sample, each feature position has just one value. Its variance along the batch axis is zero, so this normalization cannot preserve input differences at that position.
Next, switching to a recurrent neural network (RNN) introduces another difficulty.
An RNN updates its hidden state whenever it reads a word, and the state distribution may differ between time steps. Accumulating separate BatchNorm statistics for each step means dealing with variable sentence lengths and longer sequences at inference time.
Can we make normalization use only the current sample at the current time step?
Solving the Problem
LayerNorm's operation is brief:
- Fix one sample, then compute the mean and variance along the features of this layer.
Let's take a simple example and look at the following numbers.
| Sample | Feature 1 | Feature 2 | Feature 3 | Feature 4 |
|---|---|---|---|---|
| A | 1 | 2 | 3 | 4 |
| B | 101 | 102 | 103 | 104 |
For BatchNorm, the first group is the first column, , followed by the second column, .
For LayerNorm, the first group is the first row, , followed by the second row, .
The table stays the same. The calculation goes from vertical to horizontal.
More formally, denote the matrix by , where indexes samples and indexes features.
BatchNorm's training statistics are:
LayerNorm instead uses:
The mean's subscript has changed. BatchNorm has one set of statistics per feature, while LayerNorm has one set per sample.
We then normalize in the familiar way and add learnable scale and offset parameters:
is a small constant that prevents a zero denominator.
The paper's main equations omit it, but the authors' public implementation includes it. and correspond to the paper's gain and bias. Each feature has its own parameter pair, shared by different samples in the same layer. The statistics vary with the sample; the learnable parameters do not.
Work Through the First Row
Sample A has mean:
Subtracting it gives . Square these four numbers, add them, and divide by 4:
Ignoring for now, division by gives approximately:
Now consider sample B. Its mean is 102.5. Subtracting that mean again gives , with the same variance.
The two rows therefore have identical LayerNorm results.
What about BatchNorm? The first column, , has mean 51 and variance 2500, producing after normalization. The other three columns behave similarly. Ignoring and before the affine transform, sample A becomes and sample B becomes .
This example shows what each method represents: BatchNorm expresses each feature's position relative to the other samples in the batch; LayerNorm expresses features within a sample relative to that sample's mean.
Replacing sample B with different numbers can change BatchNorm's result for A. LayerNorm's calculation for A never uses B.
Let's Write Some Code!
We can write the calculation directly in PyTorch:
import torch
a = torch.tensor([
[1.0, 2.0, 3.0, 4.0],
[101.0, 102.0, 103.0, 104.0],
])
eps = 1e-5
mu = a.mean(dim=-1, keepdim=True) # [2, 1]
var = a.var(dim=-1, correction=0, keepdim=True) # [2, 1]
a_hat = (a - mu) / torch.sqrt(var + eps) # [2, 4]
print(a_hat)
# tensor([[-1.3416, -0.4472, 0.4472, 1.3416],
# [-1.3416, -0.4472, 0.4472, 1.3416]])
dim=-1 selects the last axis: the direction containing our four features. keepdim=True preserves an axis of length 1, allowing each row's mean and variance to broadcast back over its four positions.
Do not omit correction=0, because our variance calculation divides by the feature count , not , matching torch.nn.LayerNorm.
If your hand calculation disagrees with a function's output, you can read our earlier discussion:
Next, add scale and offset. The authors' default initialization sets gain to 1 and bias to 0:
gamma = torch.ones(4)
beta = torch.zeros(4)
y = gamma * a_hat + beta
At this point, is the normalized result we just computed. After training, each position can have its own scale and offset.
Only then do we use the ready-made module to check the calculation:
layer_norm = torch.nn.LayerNorm(4, eps=eps)
torch.testing.assert_close(layer_norm(a), y)
Remove the Scale: What Remains?
Our earlier samples differed only by a common offset, which LayerNorm removes. Going further, multiplying every value in a row by the same positive number also yields the same normalized result.
Let the new vector be , where , is a scalar, and has a 1 at every position. The mean becomes and the variance becomes . Ignoring and assuming :
In other words, shifting an entire row or scaling it by a positive factor leaves the normalized result unchanged. Changing just one feature, however, usually changes the row's mean and variance, affecting the output. This is the invariance discussed in Section 5.1 of the paper.
Of course, the actual calculation includes . A common shift is still removed, but invariance to positive scaling holds only approximately when is sufficiently small relative to the variances both before and after scaling. A negative scale factor also reverses the signs of the normalized result.
This part is easy to misunderstand. Since LayerNorm has , can it restore the original input?
No.
Return to A and B. Their normalized results are identical, and they share , so their affine outputs remain identical. Fixed parameters cannot restore them to two different original vectors simultaneously.
This shows that LayerNorm changes how information is represented as well as adjusting numerical ranges. The subsequent network must continue learning from this new representation.
Discussion
Who Is Affected by Smaller Batches?
Figure 6 uses MNIST classification. The authors flatten each image into a 784-dimensional vector and feed it to a fully connected 784–1000–1000–10 network, trained with 55,000 examples and Adam. Here, “permutation invariant MNIST” means classification directly through a fully connected network without using the image's spatial adjacency structure.
Let's read the figure first. The left and right columns represent batch sizes 128 and 4. The upper and lower rows show training negative log-likelihood (NLL) and test error, both lower being better; training loss uses a logarithmic scale. In each plot, the red curve is LayerNorm, the green curve is BatchNorm, and the blue curve is the baseline without normalization.
Start on the left, at batch size 128. The upper-left red curve falls rapidly after roughly 20 epochs, showing that LayerNorm can reduce training loss faster. Yet the lower-left test-error plot shows no equivalent gap: red and green remain fairly close later in training.
Now look at the right column. At batch size 4, BatchNorm has noticeably higher training loss and larger fluctuations in test error. LayerNorm continues to reduce loss and maintains comparatively stable test performance. This matches the property we discussed earlier, since LayerNorm does not rely on the statistics of other samples in the batch and so has an advantage in small-batch training.
One detail matters when interpreting this figure. The authors apply LayerNorm only to the hidden fully connected layers, excluding the final softmax output layer. The BatchNorm comparison applies it to all layers and uses an unbiased variance estimator for small batches.
This is therefore not an ablation that changes only the normalization axis while keeping all other conditions identical. The horizontal axis uses epochs, which also cannot directly compare actual training times.
The figure shows that when batch size shrinks, BatchNorm's performance can suffer, while LayerNorm can maintain comparatively stable training behavior.
RNNs Already Have BatchNorm. Does LayerNorm Still Help?
After the fully connected architecture, let's turn to RNNs. The authors train a unidirectional attentive reader on a news dataset.
The task gives the model an article and a question, and the model must find the correct entity in the article to fill the question's blank. Entity names are anonymized, and the preprocessed version used in this experiment limits each article to four sentences.
Figure 2 compares four training configurations. The horizontal axis is training steps in thousands; the vertical axis is validation error, lower being better. The four curves represent the original LSTM (blue), BatchNorm only in the LSTM (green), BatchNorm throughout the model (red), and LayerNorm only in the LSTM (cyan).
Start with the blue original LSTM: its validation error falls more slowly, taking more updates to gradually converge. The cyan LN-LSTM reaches a low error early. Red BN-everywhere shows a similar rapid decline, while green BN-LSTM needs more steps to catch up.
Later in training, the gaps between the normalization methods narrow, but LN-LSTM still achieves slightly lower validation error. This suggests that LayerNorm can improve convergence speed and final performance in RNNs as well as in the fully connected network above.
We need to be precise about “speed” here. Figure 2 compares the number of parameter updates needed to reach a validation error, rather than the actual training time. It gives no time per step, and the BN curves come from previous research, so we cannot convert horizontal distances directly into multiples of wall-clock speedup.
The authors also report only validation results for this task, without test-set scores.
So this figure supports the finding that in this attentive-reader experiment, LayerNorm reaches a lower validation error with fewer updates, and its final result is slightly better than the comparisons. How much actual training time it saves needs an additional experiment.
Faster Training: How Does Retrieval Fare?
The preceding experiments mainly examine training behavior. This time, the authors further test LayerNorm's effect on task performance.
The experiment maps MSCOCO images and their captions into a shared vector space. Images use pretrained VGG features, while a GRU encodes sentences. The authors use their own order-embedding (OE) implementation as the baseline and compare a GRU version with LayerNorm, retaining the same initial hyperparameters and architecture choices.
The task works in two retrieval directions. “Caption retrieval” finds matching sentences from an image; “image retrieval” reverses this, finding the correct image from a sentence. R@1 measures the proportion for which a correct answer ranks first; higher is better.
| Configuration in the paper's Table 2 | Caption retrieval R@1 | Image retrieval R@1 |
|---|---|---|
| OE (authors' rerun) | 46.6% | 37.8% |
| OE + LN | 48.5% | 38.9% |
Excerpt from the original Table 2. Results are averaged over five test splits, each containing 1,000 images and 5,000 captions. Models are selected by validation retrieval performance.
With LayerNorm, caption-retrieval R@1 rises from 46.6% to 48.5%, and image retrieval rises from 37.8% to 38.9%: gains of 1.9 and 1.1 percentage points respectively.
The improvements are modest, but both directions improve; faster training has not sacrificed retrieval performance.
The training time is more interesting. Section 6.1 reports that, with LayerNorm, finding the best validation model takes about 60% of the original time, equivalent to about 40% less time.
Unlike the preceding experiments that compare convergence through training steps alone, this one directly reports actual training time, making LayerNorm's benefit more concrete.
In other words, this GRU experiment reaches its best validation model sooner while also slightly improving test-set retrieval performance.
What About Convolutional Networks?
After fully connected networks and RNNs, let's finish with CNNs.
The paper also includes preliminary convolutional-network experiments. LayerNorm converges faster than the baseline without normalization, but its overall performance remains below BatchNorm's.
Why might that happen?
The authors suggest that different spatial positions in a convolutional network may have different statistical properties, especially near image boundaries. Normalizing an entire layer together with LayerNorm may overlook these spatial differences and may not suit the convolutional structure.
This is the authors' interpretation, however. The section gives no complete architecture, dataset, or quantitative comparison, so we can only say that in these preliminary experiments, LayerNorm did not outperform BatchNorm.
Conclusion
LayerNorm's influence has long extended beyond the RNNs emphasized in the original paper.
The 2017 Transformer places it after each sublayer's residual addition, independently normalizing each position's feature vector. A method proposed for small batches and variable-length sequences thus found its way into an architecture centered on attention.
Looking back, LayerNorm lets each feature vector normalize independently, and that may matter more than the speedup in any single experiment from the original paper. It requires neither other samples in the batch nor a switch to previously accumulated statistics at inference time. For models with varying sequence lengths and frequently changing batch sizes, this greatly reduces the conditions normalization must accommodate and makes it easier to integrate into different architectures.
The two methods can still behave differently during training.
The 2020 On Layer Normalization in the Transformer Architecture compares Post-LN and Pre-LN. The former normalizes after residual addition; the latter normalizes the input before the sublayer's operation. Through theoretical analysis, machine-translation experiments, and BERT pretraining, the authors show that LayerNorm's placement affects gradient distributions at initialization and can even change the need for learning-rate warm-up. Whether a model uses LayerNorm and where it places it are separate questions. Both the residual path and normalization placement can affect training stability.
Convolutional networks offer a similar example. LayerNorm did not outperform BatchNorm in the preliminary 2016 experiments. But in 2022, ConvNeXt successfully replaces BatchNorm with LayerNorm after a series of architecture and training changes. The same paper says that directly replacing BatchNorm in the original ResNet makes results worse. These observations are compatible. Normalization must be considered together with the architecture.
LayerNorm itself can also be simplified. The 2019 RMSNorm removes mean subtraction and scales features by their root mean square instead. Its authors observe comparable task performance across several models while reducing computation time in their implementations. The 2023 LLaMA also uses RMSNorm. These developments make us reconsider which normalization operations need to remain and whether adjusting scale is enough. At least in some architectures, centering can be omitted.
Today, LayerNorm is a mature design that spans different architectures. It confines statistics to a single representation, remains independent of batch size, and uses consistent computation during training and inference. In exchange, every forward pass must still compute statistics from the current input. Whether centering is needed, where normalization should go, and whether its actual execution cost is worthwhile remain adjustable choices.
After all, what normalization method could be right for every situation?
In the end, we still have to decide which values should be normalized together and how much computation we are willing to spend on it.