Back to Blog
Articles

Discrepancy in LayerNorm Calculations?

Curious about the numbers? Let's calculate and compare.

A hand calculation of LayerNorm can be close to PyTorch's output without matching it exactly. This short example tracks the difference to the variance setting.

The LayerNorm formula is:

LayerNorm(x)=x−μVar[x]+ϵ×γ+β\text{LayerNorm}(x) = \frac{x - \mu}{\sqrt{\text{Var[}x\text{]} + \epsilon}} \times \gamma + \beta

Here μ\mu is the mean of xx and Var\text{Var} is its variance. First, leave out γ\gamma and β\beta:

python
import torch

x = torch.rand(16, 768)
mu = x.mean(dim=-1, keepdim=True)
var = x.var(dim=-1, keepdim=True)
eps = 1e-5
y = (x - mu) / (var + eps).sqrt()

This yields the following values:

python
# tensor([[ 0.1219, -0.0222, -1.4742,  ...,  0.1738, -0.6124, -0.3001],
#         [-1.6009, -1.5814,  1.5357,  ...,  0.1917,  1.3787, -0.2772],
#         [ 0.3738,  1.0520,  0.4403,  ...,  1.1353, -0.7488, -0.9137],
#         ...,
#         [ 0.8823, -1.5427,  0.4725,  ..., -1.2544, -1.5354, -0.4305],
#         [ 1.4548,  0.3059, -0.6732,  ..., -0.7109,  0.4908, -1.2447],
#         [-0.4067,  0.5974, -0.9113,  ..., -0.2511, -0.2279, -0.9675]])

Next, let's compare these results with PyTorch's torch.nn.LayerNorm:

python
layer_norm = torch.nn.LayerNorm(768, elementwise_affine=False, bias=False)

y_ln = layer_norm(x)

This yields:

python
# tensor([[ 0.1220, -0.0222, -1.4752,  ...,  0.1739, -0.6128, -0.3003],
#         [-1.6020, -1.5824,  1.5367,  ...,  0.1918,  1.3796, -0.2774],
#         [ 0.3741,  1.0527,  0.4406,  ...,  1.1360, -0.7493, -0.9143],
#         ...,
#         [ 0.8829, -1.5437,  0.4728,  ..., -1.2552, -1.5364, -0.4308],
#         [ 1.4557,  0.3061, -0.6736,  ..., -0.7113,  0.4911, -1.2455],
#         [-0.4069,  0.5978, -0.9119,  ..., -0.2513, -0.2281, -0.9681]])

The two outputs are close, but why do they differ?

Unbiased Estimation

By default, torch.var uses correction=1, applying Bessel's correction to the variance estimate.

This means that it divides by N-1 instead of N, whereas torch.nn.LayerNorm uses N.

So let's modify the torch.var function by setting correction=0:

python
var = x.var(dim=-1, correction=0, keepdim=True)

Now, let's compare the results again:

python
# tensor([[ 0.1220, -0.0222, -1.4752,  ...,  0.1739, -0.6128, -0.3003],
#         [-1.6020, -1.5824,  1.5367,  ...,  0.1918,  1.3796, -0.2774],
#         [ 0.3741,  1.0527,  0.4406,  ...,  1.1360, -0.7493, -0.9143],
#         ...,
#         [ 0.8829, -1.5437,  0.4728,  ..., -1.2552, -1.5364, -0.4308],
#         [ 1.4557,  0.3061, -0.6736,  ..., -0.7113,  0.4911, -1.2455],
#         [-0.4069,  0.5978, -0.9119,  ..., -0.2513, -0.2281, -0.9681]])

Now the numbers match perfectly!

Why Does LayerNorm Use correction=0?

LayerNorm computes the mean and variance over features of the same input, then normalizes that input with those values. It is not estimating a separate population variance from a sample, so the formula divides by NN. The default torch.var setting uses Bessel's correction for a different statistical purpose.

With 768 features, the numerical difference between the two denominators is small but visible in an elementwise comparison. It becomes more noticeable with fewer features.

Conclusion

If a hand calculation disagrees with torch.nn.LayerNorm, check the variance denominator first: torch.var(..., correction=0) matches the configuration used here.