A hand calculation of LayerNorm can be close to PyTorch's output without matching it exactly. This short example tracks the difference to the variance setting.
The LayerNorm formula is:
Here is the mean of and is its variance. First, leave out and :
import torch
x = torch.rand(16, 768)
mu = x.mean(dim=-1, keepdim=True)
var = x.var(dim=-1, keepdim=True)
eps = 1e-5
y = (x - mu) / (var + eps).sqrt()
This yields the following values:
# tensor([[ 0.1219, -0.0222, -1.4742, ..., 0.1738, -0.6124, -0.3001],
# [-1.6009, -1.5814, 1.5357, ..., 0.1917, 1.3787, -0.2772],
# [ 0.3738, 1.0520, 0.4403, ..., 1.1353, -0.7488, -0.9137],
# ...,
# [ 0.8823, -1.5427, 0.4725, ..., -1.2544, -1.5354, -0.4305],
# [ 1.4548, 0.3059, -0.6732, ..., -0.7109, 0.4908, -1.2447],
# [-0.4067, 0.5974, -0.9113, ..., -0.2511, -0.2279, -0.9675]])
Next, let's compare these results with PyTorch's torch.nn.LayerNorm:
layer_norm = torch.nn.LayerNorm(768, elementwise_affine=False, bias=False)
y_ln = layer_norm(x)
This yields:
# tensor([[ 0.1220, -0.0222, -1.4752, ..., 0.1739, -0.6128, -0.3003],
# [-1.6020, -1.5824, 1.5367, ..., 0.1918, 1.3796, -0.2774],
# [ 0.3741, 1.0527, 0.4406, ..., 1.1360, -0.7493, -0.9143],
# ...,
# [ 0.8829, -1.5437, 0.4728, ..., -1.2552, -1.5364, -0.4308],
# [ 1.4557, 0.3061, -0.6736, ..., -0.7113, 0.4911, -1.2455],
# [-0.4069, 0.5978, -0.9119, ..., -0.2513, -0.2281, -0.9681]])
The two outputs are close, but why do they differ?
Unbiased Estimation
By default, torch.var uses correction=1, applying Bessel's correction to the variance estimate.
This means that it divides by N-1 instead of N, whereas torch.nn.LayerNorm uses N.
So let's modify the torch.var function by setting correction=0:
var = x.var(dim=-1, correction=0, keepdim=True)
Now, let's compare the results again:
# tensor([[ 0.1220, -0.0222, -1.4752, ..., 0.1739, -0.6128, -0.3003],
# [-1.6020, -1.5824, 1.5367, ..., 0.1918, 1.3796, -0.2774],
# [ 0.3741, 1.0527, 0.4406, ..., 1.1360, -0.7493, -0.9143],
# ...,
# [ 0.8829, -1.5437, 0.4728, ..., -1.2552, -1.5364, -0.4308],
# [ 1.4557, 0.3061, -0.6736, ..., -0.7113, 0.4911, -1.2455],
# [-0.4069, 0.5978, -0.9119, ..., -0.2513, -0.2281, -0.9681]])
Now the numbers match perfectly!
Why Does LayerNorm Use correction=0?
LayerNorm computes the mean and variance over features of the same input, then normalizes that input with those values. It is not estimating a separate population variance from a sample, so the formula divides by . The default torch.var setting uses Bessel's correction for a different statistical purpose.
With 768 features, the numerical difference between the two denominators is small but visible in an elementwise comparison. It becomes more noticeable with fewer features.
Conclusion
If a hand calculation disagrees with torch.nn.LayerNorm, check the variance denominator first: torch.var(..., correction=0) matches the configuration used here.
