{"notes":[{"id":"classic-cnns/nfnet","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"NFNet","subtitle":"Life Without BN","author":"Z. Yuan","paperDate":"2021-02-11","publishedOn":"2025-01-21","world":"external/nfnet","paperTitle":"High-Performance Large-Scale Image Recognition Without Normalization","paperUrl":"https://arxiv.org/abs/2102.06171","bibliography":{"title":"High-Performance Large-Scale Image Recognition Without Normalization","authors":["Andrew Brock","Soham De","Samuel L. Smith","Karen Simonyan"],"url":"https://arxiv.org/abs/2102.06171","pdfUrl":"https://arxiv.org/pdf/2102.06171v1","codeUrl":null,"preprint":{"id":"2102.06171","submittedOn":"2021-02-11","url":"https://arxiv.org/abs/2102.06171"},"publication":{"venue":"ICML","year":2021,"kind":"conference","url":"https://proceedings.mlr.press/v139/brock21a.html"},"citation":"@misc{classiccnnsnfnet2021,\n  title = {{High-Performance Large-Scale Image Recognition Without Normalization}},\n  author = {Andrew Brock and Soham De and Samuel L. Smith and Karen Simonyan},\n  year = {2021},\n  eprint = {2102.06171},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2102.06171}\n}\n","citationPath":"/content/papers/citations/classic-cnns/nfnet.91baba59df96.bib"},"thumbnail":{"src":"/content/papers/assets/7552bff30433f4f33bec5edb2421acd94d61d452888dde6df17557778e8fb1c6.png","width":1004,"height":783},"locale":"en","path":"/content/papers/en/classic-cnns/nfnet.6c1bcaebe98e.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"what-batchnorm-actually-does-for-us","text":"What BatchNorm Actually Does for Us","depth":3},{"id":"what-mean-shift-looks-like","text":"What Mean Shift Looks Like","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"first-finishing-nf-resnet","text":"First, Finishing NF-ResNet","depth":3},{"id":"adaptive-gradient-clipping","text":"Adaptive Gradient Clipping","depth":3},{"id":"the-nfnet-architecture","text":"The NFNet Architecture","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"what-each-change-contributes","text":"What Each Change Contributes","depth":3},{"id":"is-removing-bn-really-faster","text":"Is Removing BN Really Faster?","depth":3},{"id":"against-other-models","text":"Against Other Models","depth":3},{"id":"fine-tuning-after-pretraining","text":"Fine-Tuning After Pretraining","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"normalization/batchnorm","relation":"referenced","topic":""}]},{"id":"normalization/batchnorm","category":"normalization","legacyPath":"normalization/1502-batchnorm","legacySlug":"1502-batchnorm","legacyPaths":[],"title":"BatchNorm","subtitle":"Batch Normalization","author":"Z. Yuan","paperDate":"2015-02-11","publishedOn":"2024-08-01","paperTitle":"Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift","paperUrl":"https://arxiv.org/abs/1502.03167","bibliography":{"title":"Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift","authors":["Sergey Ioffe","Christian Szegedy"],"url":"https://arxiv.org/abs/1502.03167","pdfUrl":"https://arxiv.org/pdf/1502.03167","codeUrl":null,"preprint":{"id":"1502.03167","submittedOn":"2015-02-11","url":"https://arxiv.org/abs/1502.03167"},"publication":{"venue":"ICML","year":2015,"kind":"conference","url":"https://proceedings.mlr.press/v37/ioffe15.html"},"citation":"@misc{normalizationbatchnorm2015,\n  title = {{Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift}},\n  author = {Sergey Ioffe and Christian Szegedy},\n  year = {2015},\n  eprint = {1502.03167},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1502.03167}\n}\n","citationPath":"/content/papers/citations/normalization/batchnorm.bfa8abb33f32.bib"},"thumbnail":{"src":"/content/papers/assets/48b236d4b897164b93764eccfe22db29e5b9bce290d864336b8f12ec127c1e4b.jpg","width":1016,"height":824},"locale":"en","path":"/content/papers/en/normalization/batchnorm.149073a22595.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"increasing-learning-rates","text":"Increasing Learning Rates","depth":3},{"id":"no-need-for-dropout","text":"No Need for Dropout","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"mnist-experiment","text":"MNIST Experiment","depth":3},{"id":"imagenet-experiment","text":"ImageNet Experiment","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"normalization/layernorm","relation":"next","topic":"Choosing the normalization axes"}]}],"text":"After 2015, nearly every image classification network had BatchNorm in it. It makes deep networks easy to train, allows large learning rates, and throws in a bit of regularization on the side. It was so handy that people rarely stopped to think about its drawbacks.\n\n\n> [!TIP]\n> \n> For BatchNorm, you can start with our note:\n> \n> - **[[15.02] BatchNorm: Batch Normalization](/blog/papers/normalization/batchnorm)**\n\n\nIt has quite a few drawbacks, actually. It has to compute batch statistics, which costs memory and time. It uses batch statistics during training and moving averages at inference, so it behaves differently in the two. It also makes the samples in a batch affect one another, and that's the most troublesome part. It works worse when batches are too small, it's error-prone in distributed training, and in contrastive learning it needs special handling so the model can't peek at the answer through the batch statistics.\n\nAndrew Brock, Soham De, Samuel L. Smith, and Karen Simonyan at DeepMind wanted to remove BatchNorm entirely. In their previous paper they had already built NF-ResNets without normalization layers, whose accuracy caught up with BN-ResNets of the same size; this paper tackles the two problems that remained.\n\n## Defining the Problem\n\n### What BatchNorm Actually Does for Us\n\nTo remove BatchNorm, you first have to know what it does, then find a replacement for each job one by one. The authors list four:\n\n1. **It shrinks the residual branch's output.** When BN sits on the residual branch, the branch output is fairly small at initialization, so the signal mainly travels through the shortcut and gradients are more stable early in training.\n2. **It eliminates mean shift.** Activation functions like ReLU and GELU have outputs whose mean isn't 0, and as layers stack up, the features of different samples grow more and more alike. BN pulls the mean back to 0 at every layer.\n3. **It has a regularizing effect.** Batch statistics are noisy, which is a bit like adding noise during training.\n4. **It makes large-batch training practical.** BN smooths the loss landscape and allows larger learning rates, which is what makes large-batch training worthwhile.\n\nThe earlier NF-ResNet handled the first two with initialization and weight standardization, and covered the third with Dropout and Stochastic Depth. Two problems remained. Training is unstable at batch sizes of 4096 and above, and the best results still couldn't catch up with EfficientNet.\n\n### What Mean Shift Looks Like\n\nThe second item is fairly abstract, so let's first see what happens without normalization. We take 64 mutually unrelated random samples and pass them, again and again, through a ReLU and a layer of He-initialized weights, 20 layers in all, measuring the mean cosine similarity between different samples. A second group uses NF-ResNet's approach instead, which we'll explain in a moment:\n\n```python\nimport math\nimport torch\nimport torch.nn.functional as F\n\ntorch.manual_seed(0)\nx = torch.randn(64, 512)                      # 64 mutually unrelated samples, 512 features each\n\ndef ws(W):\n    \"\"\"Scaled Weight Standardization: subtract each output unit's weight mean, then divide by std·√N\"\"\"\n    N = W.shape[1]\n    return (W - W.mean(1, keepdim=True)) / (W.std(1, unbiased=False, keepdim=True) * math.sqrt(N))\n\ndef mean_cos(h):\n    \"\"\"Mean cosine similarity between different samples\"\"\"\n    h = F.normalize(h, dim=1)\n    sim = h @ h.T\n    return (sim.sum() - sim.diagonal().sum()).item() / (h.shape[0] * (h.shape[0] - 1))\n\ngamma = math.sqrt(2 / (1 - 1 / math.pi))      # gain for ReLU\nprint(f\"γ = {gamma:.4f}\")\nplain, std = x.clone(), x.clone()\nfor layer in range(1, 21):\n    W = torch.randn(512, 512)\n    plain = F.relu(plain) @ (W * math.sqrt(2 / 512)).T      # He initialization, no normalization\n    std = (gamma * F.relu(std)) @ ws(W).T                   # standardized weights plus gain\n    if layer in (1, 5, 10, 20):\n        print(f\"Layer {layer:>2} | He init: cross-sample similarity {mean_cos(plain):.2f}\"\n              f\" | Scaled WS: similarity {mean_cos(std):.2f}, variance {std.var().item():.2f}\")\n```\n\n```text\nγ = 1.7129\nLayer  1 | He init: cross-sample similarity 0.31 | Scaled WS: similarity 0.00, variance 0.99\nLayer  5 | He init: cross-sample similarity 0.72 | Scaled WS: similarity -0.00, variance 0.98\nLayer 10 | He init: cross-sample similarity 0.87 | Scaled WS: similarity 0.00, variance 0.95\nLayer 20 | He init: cross-sample similarity 0.96 | Scaled WS: similarity 0.00, variance 0.88\n```\n\nThe 64 input samples start out unrelated, yet after 20 layers the mean similarity in the He-initialized group has reached 0.96, and nearly all the samples look the same. The reason is that ReLU outputs are all greater than or equal to 0, so every layer adds the same positive mean to every sample. Research cited by the paper shows that this shift can make a deep network predict the same class for every sample at initialization.\n\n## Solving the Problem\n\n### First, Finishing NF-ResNet\n\nThe group on the right uses two tricks from the earlier NF-ResNet paper.\n\nThe first is **Scaled Weight Standardization**. For each output unit of a convolutional layer, the weights first have their own mean subtracted, then are divided by their standard deviation and $\\sqrt{N}$:\n\n$$\n\\hat{W}_{ij}=\\frac{W_{ij}-\\mu_i}{\\sqrt{N}\\,\\sigma_i},\n$$\n\nwhere $N$ is the fan-in. After the mean is subtracted, each output unit's weights sum to 0, which cancels out the shared positive value the activation function brings in. That's why the similarity above stays at 0.\n\nThe second is a gain $\\gamma$ multiplied after the activation function. For ReLU, $\\gamma=\\sqrt{2/(1-1/\\pi)}$, about 1.71, which exactly makes up for the variance lost when ReLU cuts off the negative values. Together, the two keep each layer's output variance at roughly 1. The 0.88 after 20 layers comes from small deviations piling up layer by layer, a far smaller problem than the other group's samples nearly all turning into the same one.\n\nWith a branch that “roughly preserves variance at every layer,” NF-ResNet's residual block is written as\n\n$$\nh_{i+1}=h_i+\\alpha\\,f_i\\!\\left(h_i/\\beta_i\\right).\n$$\n\n$\\beta_i$ is the precomputed standard deviation of the input, so the signal entering the branch is always scaled back to variance 1; $\\alpha$ is a small constant (usually 0.2) that controls how much each block increases the variance. The variance along the main path can therefore be computed directly, growing by $\\alpha^2$ with each block. Let's build 12 blocks to check:\n\n```python\nimport math\nimport torch\nimport torch.nn.functional as F\n\ntorch.manual_seed(0)\ngamma = math.sqrt(2 / (1 - 1 / math.pi))\n\ndef ws(W):\n    N = W.shape[1]\n    return (W - W.mean(1, keepdim=True)) / (W.std(1, unbiased=False, keepdim=True) * math.sqrt(N))\n\ndef branch(h):\n    \"\"\"Residual branch: two layers of \"gain-scaled ReLU + standardized weights\", output variance ≈ input variance\"\"\"\n    for _ in range(2):\n        h = (gamma * F.relu(h)) @ ws(torch.randn(512, 512)).T\n    return h\n\nalpha = 0.2\nh = torch.randn(4096, 512)\nexpected = 1.0\nfor i in range(1, 13):\n    beta = math.sqrt(expected)                # precomputed standard deviation of the input\n    h = h + alpha * branch(h / beta)          # h_{i+1} = h_i + α·f(h_i / β_i)\n    expected += alpha ** 2                    # Var(h_{i+1}) = Var(h_i) + α²\n    if i in (1, 4, 8, 12):\n        print(f\"After block {i:>2}: expected variance {expected:.2f}, actual {h.var().item():.2f}\")\n```\n\n```text\nAfter block  1: expected variance 1.04, actual 1.04\nAfter block  4: expected variance 1.16, actual 1.16\nAfter block  8: expected variance 1.32, actual 1.32\nAfter block 12: expected variance 1.48, actual 1.48\n```\n\nThe actual variance matches the expected value exactly. This replaces BatchNorm's first job. Because $\\alpha$ is small, the signal mainly travels through the shortcut, and at first the branch only makes small corrections. In blocks that shrink the feature map, the shortcut also takes the scaled input, and the variance count restarts from $1+\\alpha^2$. On top of that, each residual branch ends with a learnable scalar initialized to 0 (SkipInit), so the whole branch is switched off at the start of training.\n\nAt this point, NF-ResNet can already match BN-ResNet at batch size 1024, but it becomes unstable once the batch gets large.\n\n### Adaptive Gradient Clipping\n\nLarge-batch training calls for large learning rates, and once the gradients get big, a single step can push the parameters too far. The usual fix is gradient clipping:\n\n$$\nG\\rightarrow\n\\begin{cases}\n\\lambda\\dfrac{G}{\\lVert G\\rVert} & \\text{if }\\lVert G\\rVert>\\lambda,\\\\\nG & \\text{otherwise.}\n\\end{cases}\n$$\n\nThe authors found that this does hold up at somewhat larger batches, but the threshold $\\lambda$ is very hard to tune. Whenever the model depth, batch size, or learning rate changes, it has to be searched for all over again.\n\nThe trouble is that a fixed threshold ignores how large the parameters themselves are. The same gradient of 0.01 is only a 1% change on a weight of size 1, but on a weight of size 0.01 it flips the whole thing. So the authors look at a ratio instead:\n\n$$\n\\frac{\\lVert G^\\ell\\rVert_F}{\\lVert W^\\ell\\rVert_F},\n$$\n\nwhich roughly tells you by what fraction one update step will change the weights. In practice it isn't computed once for the whole layer but **unit-wise**, that is, for each row of the weight matrix (for a convolution, each output channel together with its spatial extent):\n\n$$\nG_i^\\ell\\rightarrow\n\\begin{cases}\n\\lambda\\dfrac{\\lVert W_i^\\ell\\rVert_F^\\star}{\\lVert G_i^\\ell\\rVert_F}\\,G_i^\\ell & \\text{if }\\dfrac{\\lVert G_i^\\ell\\rVert_F}{\\lVert W_i^\\ell\\rVert_F^\\star}>\\lambda,\\\\[6pt]\nG_i^\\ell & \\text{otherwise.}\n\\end{cases}\n$$\n\nHere $\\lVert W_i\\rVert_F^\\star=\\max(\\lVert W_i\\rVert_F,\\epsilon)$ with $\\epsilon=10^{-3}$, which keeps zero-initialized parameters from always having their gradients clipped to 0. This is Adaptive Gradient Clipping (AGC).\n\nLet's see how it clips with a small example of three units:\n\n```python\nimport torch\n\nW = torch.tensor([[1.0, 0.0, 0.0],\n                  [0.0, 0.1, 0.0],\n                  [0.0, 0.0, 2.0]])           # three output units with weights of different sizes\nG = torch.tensor([[0.005, 0.0, 0.0],\n                  [0.0, 0.05, 0.0],\n                  [0.0, 0.0, 0.01]])          # three rows of gradients\n\nlam, eps = 0.01, 1e-3\nw_norm = W.norm(dim=1, keepdim=True).clamp(min=eps)\ng_norm = G.norm(dim=1, keepdim=True)\nratio = g_norm / w_norm\nclipped = torch.where(ratio > lam, G * (lam * w_norm / g_norm), G)\n\nfor i in range(3):\n    print(f\"Unit {i}: ‖W‖={w_norm[i].item():.2f}, ‖G‖={g_norm[i].item():.3f}, ratio {ratio[i].item():.3f}\"\n          f\" → clipped ‖G‖={clipped[i].norm().item():.4f}\")\nprint(\"Overall gradient norm:\", round(G.norm().item(), 4))\n```\n\n```text\nUnit 0: ‖W‖=1.00, ‖G‖=0.005, ratio 0.005 → clipped ‖G‖=0.0050\nUnit 1: ‖W‖=0.10, ‖G‖=0.050, ratio 0.500 → clipped ‖G‖=0.0010\nUnit 2: ‖W‖=2.00, ‖G‖=0.010, ratio 0.005 → clipped ‖G‖=0.0100\nOverall gradient norm: 0.0512\n```\n\nOnly unit 1 got clipped. Its weight is only 0.1 while its gradient is 0.05, enough to change half of it in one step, so it gets pushed down to $\\lambda\\times0.1=0.001$. The other two units have ratios below the threshold and are left untouched. With ordinary global clipping, all three units would be scaled down by the same factor, and the one that really needs reining in might not get reined in enough.\n\nThe code above gives the same result as timm's `timm.utils.agc.adaptive_clip_grad`, with the three units coming out at 0.005, 0.001, and 0.01 after clipping.\n\n[![Two line charts. (a) The horizontal axis is batch size (256 to 4096) and the vertical axis is Top-1 accuracy; the blue lines are BatchNorm, the orange lines NF-ResNet, and the green lines NF-ResNet+AGC, with solid lines for ResNet50 and dashed lines for ResNet200. The orange lines plunge after batch size 2048, while the green lines stay close to or above the blue ones all the way. (b) ResNet50's accuracy under different clipping thresholds (0.01 to 0.16), one line per batch size; the purple line for batch size 4096 collapses after threshold 0.02](/content/papers/assets/9c1e7cd2c8c545de3ebab245e654b0dd69b3921c4f0479f0e99131833bdfc5f2.png)](/content/papers/assets/9c1e7cd2c8c545de3ebab245e654b0dd69b3921c4f0479f0e99131833bdfc5f2.png)\n\n*Figure 2 from the paper, excerpted from [**page 4 of the v1 PDF**](https://arxiv.org/pdf/2102.06171v1#page=4). (a) AGC lets NF-ResNets use larger batches; (b) performance under different clipping thresholds $\\lambda$.*\n\nIn the left plot, the orange lines are NF-ResNet without AGC, which collapse beyond batch size 2048; the green lines, with AGC, hold up all the way to 4096, with accuracy about the same as BatchNorm's blue lines or even better. The right plot is about the threshold. The larger the batch, the smaller the threshold needs to be, and at batch size 4096 only 0.01 and 0.02 hold up.\n\nThe authors also tested which layers need clipping. It's better not to clip the final fully connected layer; the first convolutional layer can often go unclipped, but at batch size 4096 the weights of all four stages have to be clipped. All later experiments apply AGC to every parameter except the last layer.\n\n### The NFNet Architecture\n\nWith AGC in hand, the authors went on to design a new network. This time the target wasn't FLOPs or parameter count, but **actual training speed on TPUs and GPUs**. The authors point out that EfficientNet has very low FLOPs but doesn't train any faster for it. EfficientNet-B0 has only a tenth of ResNet-50's FLOPs, yet trains about as slowly on a GPU or TPU.\n\n[![Diagram of the NFNet bottleneck block. The input is first multiplied by 1/β, passes through a 1x1 convolution, two 3x3 convolutions, and another 1x1 convolution, is multiplied by α, and is added to the shortcut. On the right are the stage widths and depths: ResNet widths [256, 512, 1024, 2048] and NFNet widths [256, 512, 1536, 1536]; ResNet depths [3, 4, 6, 3], [3, 4, 23, 3]…, and NFNet depths [1, 2, 6, 3] times N](/content/papers/assets/6dc27a9ccca4de619c057562ea6384aab3c095ddc08b22f475601ab03fcf03bb.png)](/content/papers/assets/6dc27a9ccca4de619c057562ea6384aab3c095ddc08b22f475601ab03fcf03bb.png)\n\n*Figure 3 from the paper, excerpted from [**page 5 of the v1 PDF**](https://arxiv.org/pdf/2102.06171v1#page=5). The NFNet bottleneck block, and how it differs from ResNet in width and depth.*\n\nThe starting point is SE-ResNeXt-D with its activation function switched to GELU, followed by four changes:\n\n1. **Group width fixed at 128.** Finer groups mean fewer FLOPs, but compute density drops along with them, so on accelerators there's no real speedup.\n2. **Depth pattern [1, 2, 6, 3].** Larger models simply multiply it by N; F1, for example, is [2, 4, 12, 6]. ResNets traditionally deepen only the middle two stages, which the authors consider suboptimal.\n3. **Width pattern [256, 512, 1536, 1536].** Capacity is concentrated in the third stage and the fourth stage is slightly reduced, keeping training speed roughly the same.\n4. **One more $3\\times3$ grouped convolution in the block.** These are the two consecutive 3x3s in the figure. The authors say it adds only a few FLOPs and, on their accelerators, has almost no effect on training time.\n\nWhen scaling the model up, only depth and resolution increase, not width; the authors found that widening doesn't work well for ResNet-style backbones. Each step up roughly doubles the training time, and Dropout goes up along with it, from 0.2 for F0 to 0.5 for F6. Without BatchNorm's implicit regularization, the model overfits badly unless explicit regularization like Dropout steps in.\n\n## Discussion\n\n### What Each Change Contributes\n\n| Setting | F0 | F1 | F2 | F3 |\n| --- | ---: | ---: | ---: | ---: |\n| Baseline | 80.4% | 81.7% | 82.0% | 82.3% |\n| + Modified width | 80.9% | 81.8% | 82.0% | 82.3% |\n| + Second $3\\times3$ | 81.3% | 82.2% | 82.4% | 82.7% |\n| + MixUp | 82.2% | 82.9% | 83.1% | 83.5% |\n| + RandAugment | 83.2% | 84.6% | 84.8% | 85.0% |\n| + CutMix | **83.6%** | **84.7%** | **85.1%** | **85.7%** |\n| Original width + the augmentations above | 83.1% | 84.5% | 85.0% | 85.5% |\n\n*Table: Excerpted from Table 2 of the paper, ImageNet top-1 accuracy averaged over 3 random seeds, trained for 360 epochs at batch size 4096.*\n\nTogether, the two architectural changes add 0.4 to 0.9 percentage points. What really opens up the gap is data augmentation, with RandAugment alone worth 1 to 1.7 percentage points. These strong augmentations are exactly what the original NF-ResNet couldn't handle, and AGC makes them usable. The last row switches back to ResNet's original width pattern, and all four sizes do worse, which shows the width change really does help.\n\nThe authors also applied the same augmentations to EfficientNet. RandAugment helps, but going beyond 2 layers, or adding MixUp and CutMix, brings no further improvement. On this basis they argue that NFNet's gains don't come from augmentation alone.\n\n### Is Removing BN Really Faster?\n\nWhen the same architectures are trained with BatchNorm instead, the accuracy and per-step training time are:\n\n| Model | NF version | BN version |\n| --- | --- | --- |\n| F0 | 83.6%, 73.3 ms | 83.4%, 111.7 ms |\n| F1 | 84.7%, 158.5 ms | 84.4%, 258.0 ms |\n| F2 | 85.1%, 295.8 ms | 85.1%, 396.3 ms |\n\n*Table: Excerpted from Table 6 of the paper, ImageNet top-1 accuracy and per-step training time on TPUv3.*\n\nAccuracy is about the same, with the NF version slightly better or tied, while per-step training time drops by 25% to 40%. The authors also mention that the larger F4 and F5 didn't train stably with BatchNorm, with or without AGC. Their guess is that the bfloat16 used to fit the models into memory doesn't get along with the computation of batch statistics.\n\n### Against Other Models\n\n[![Scatter-and-line plot with per-step training time on TPUv3 (seconds) on the horizontal axis and ImageNet Top-1 accuracy (80% to 87%) on the vertical axis. The solid red line is NFNet-F0 to F5, sitting above and to the left of all the other models; the dashed blue line is EfficientNet B2 to B7, green is LambdaNet, orange is BoTNet, and the black dots are DeiT. NFNet-F1 matches the accuracy of EffNet-B7 at the far right, with a much shorter training time](/content/papers/assets/7552bff30433f4f33bec5edb2421acd94d61d452888dde6df17557778e8fb1c6.png)](/content/papers/assets/7552bff30433f4f33bec5edb2421acd94d61d452888dde6df17557778e8fb1c6.png)\n\n*Figure 1 from the paper, excerpted from [**page 1 of the v1 PDF**](https://arxiv.org/pdf/2102.06171v1#page=1). ImageNet validation accuracy versus TPUv3 training time, single model, single crop.*\n\nThe whole red line lies above and to the left of the other models, meaning higher accuracy for the same training time. The most often quoted pair of numbers is NFNet-F1 versus EfficientNet-B7. Both reach 84.7%, but F1 needs only 158.5 ms per step while B7 needs 1397.0 ms, which makes F1 nearly 9 times faster. The largest, NFNet-F5, reaches 86.0%; with the SAM optimizer, F5 gets to 86.3% and F6 to 86.5%, the best results at the time without extra data.\n\nThis red line, though, is plotted against training time. Measured by parameter count, NFNet is anything but frugal. F0 has 71.5M parameters and 12.38 GFLOPs, while EfficientNet-B0 has only 5.3M and 0.39 GFLOPs; F1 also has twice as many parameters as B7. The authors are explicit about this in the paper. What they optimize is training time, and FLOPs and parameter count were never the target.\n\n### Fine-Tuning After Pretraining\n\nIn the last set of experiments, the authors pretrain on 300 million labeled images and then fine-tune on ImageNet. With this much data, BatchNorm's regularizing effect may actually get in the way, and the authors suspect networks without BN suit this setting better:\n\n| Model | 224px | 320px | 384px |\n| --- | ---: | ---: | ---: |\n| BN-ResNet-50 | 78.1% | 79.6% | 79.9% |\n| NF-ResNet-50 | **79.5%** | **80.9%** | **81.1%** |\n| BN-ResNet-200 | 81.8% | 83.1% | 83.5% |\n| NF-ResNet-200 | **82.9%** | **84.1%** | **84.3%** |\n\n*Table: Two groups excerpted from Table 4 of the paper, top-1 accuracy after pretraining and fine-tuning on ImageNet. The full table also includes ResNet-101 and 152, with consistent results.*\n\nThe NF version wins in every group, by about 1 percentage point. The authors then pretrain a slightly wider NFNet-F4+ for 20 epochs, which reaches 89.2% after fine-tuning, the highest result achieved with transfer learning at the time.\n\n## Conclusion\n\nWith BatchNorm gone, training actually got faster. But everything BatchNorm used to handle on the side has to be put back by hand, one piece at a time. Mean shift goes to Scaled Weight Standardization and the gain, the residual branch's scale to $\\alpha$, $\\beta$, and SkipInit, regularization to Dropout and strong data augmentation, and large-batch training is held together by AGC.\n\nIf you want to use it directly, timm has `dm_nfnet_f0` through `dm_nfnet_f6`, plus the `eca_nfnet_l0` series that timm trained itself and slimmed down for GPUs; AGC can also be applied on its own to your own model with `adaptive_clip_grad`.\n\nOh, right, if you're going to train any NFNet variant, remember to turn on gradient clipping. Don't tempt fate!\n","imageSizes":{"/content/papers/assets/9c1e7cd2c8c545de3ebab245e654b0dd69b3921c4f0479f0e99131833bdfc5f2.png":{"width":1004,"height":417},"/content/papers/assets/6dc27a9ccca4de619c057562ea6384aab3c095ddc08b22f475601ab03fcf03bb.png":{"width":966,"height":288},"/content/papers/assets/7552bff30433f4f33bec5edb2421acd94d61d452888dde6df17557778e8fb1c6.png":{"width":1004,"height":783}}}