{"notes":[{"id":"classic-cnns/convnext-v2","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"ConvNeXt V2","subtitle":"Convolutions Can Fill in the Blanks Too","author":"Z. Yuan","paperDate":"2023-01-02","publishedOn":"2025-09-27","world":"external/convnext-v2","paperTitle":"ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders","paperUrl":"https://arxiv.org/abs/2301.00808","bibliography":{"title":"ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders","authors":["Sanghyun Woo","Shoubhik Debnath","Ronghang Hu","Xinlei Chen","Zhuang Liu","In So Kweon","Saining Xie"],"url":"https://arxiv.org/abs/2301.00808","pdfUrl":"https://arxiv.org/pdf/2301.00808v1","codeUrl":null,"preprint":{"id":"2301.00808","submittedOn":"2023-01-02","url":"https://arxiv.org/abs/2301.00808"},"publication":{"venue":"CVPR","year":2023,"kind":"conference","url":"https://openaccess.thecvf.com/content/CVPR2023/html/Woo_ConvNeXt_V2_Co-Designing_and_Scaling_ConvNets_With_Masked_Autoencoders_CVPR_2023_paper.html"},"citation":"@misc{classiccnnsconvnextv22023,\n  title = {{ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders}},\n  author = {Sanghyun Woo and Shoubhik Debnath and Ronghang Hu and Xinlei Chen and Zhuang Liu and In So Kweon and Saining Xie},\n  year = {2023},\n  eprint = {2301.00808},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2301.00808}\n}\n","citationPath":"/content/papers/citations/classic-cnns/convnext-v2.065fc6e2e545.bib"},"thumbnail":{"src":"/content/papers/assets/0af11b7bcef637ee0205fe47b31901b1b775a0336ccab0042e73e8d9643589c0.png","width":1640,"height":600},"locale":"en","path":"/content/papers/en/classic-cnns/convnext-v2.0556702ca43f.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"why-mae-doesnt-carry-over-easily","text":"Why MAE Doesn't Carry Over Easily","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"fcmae-a-fully-convolutional-mae","text":"FCMAE: A Fully Convolutional MAE","depth":3},{"id":"changing-only-the-pretraining-isnt-enough","text":"Changing Only the Pretraining Isn't Enough","depth":3},{"id":"feature-collapse","text":"Feature Collapse","depth":3},{"id":"grn-making-channels-compete","text":"GRN: Making Channels Compete","depth":3},{"id":"the-convnext-v2-block","text":"The ConvNeXt V2 Block","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"architecture-and-pretraining-have-to-change-together","text":"Architecture and Pretraining Have to Change Together","depth":3},{"id":"against-transformers-with-mae","text":"Against Transformers with MAE","depth":3},{"id":"detection-and-segmentation","text":"Detection and Segmentation","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/convnext","relation":"referenced","topic":""},{"id":"vision-transformers/mae","relation":"referenced","topic":""}]},{"id":"classic-cnns/convnext","category":"classic-cnns","legacyPath":"classic-cnns/2201-convnext","legacySlug":"2201-convnext","legacyPaths":[],"title":"ConvNeXt","subtitle":"Making Convolutions Great Again","author":"Z. Yuan","paperDate":"2022-01-10","publishedOn":"2024-06-30","paperTitle":"A ConvNet for the 2020s","paperUrl":"https://arxiv.org/abs/2201.03545","bibliography":{"title":"A ConvNet for the 2020s","authors":["Zhuang Liu","Hanzi Mao","Chao-Yuan Wu","Christoph Feichtenhofer","Trevor Darrell","Saining Xie"],"url":"https://arxiv.org/abs/2201.03545","pdfUrl":"https://arxiv.org/pdf/2201.03545","codeUrl":"https://github.com/facebookresearch/ConvNeXt","preprint":{"id":"2201.03545","submittedOn":"2022-01-10","url":"https://arxiv.org/abs/2201.03545"},"publication":{"venue":"CVPR","year":2022,"kind":"conference","url":"https://arxiv.org/abs/2201.03545"},"citation":"@misc{classiccnnsconvnext2022,\n  title = {{A ConvNet for the 2020s}},\n  author = {Zhuang Liu and Hanzi Mao and Chao-Yuan Wu and Christoph Feichtenhofer and Trevor Darrell and Saining Xie},\n  year = {2022},\n  eprint = {2201.03545},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2201.03545}\n}\n","citationPath":"/content/papers/citations/classic-cnns/convnext.801b4060b807.bib"},"thumbnail":{"src":"/content/papers/assets/37882616c5c72df33699003256d14bf7c58c8ea5d1e9e4fb5b1c33268c9a6923.jpg","width":1224,"height":432},"locale":"en","path":"/content/papers/en/classic-cnns/convnext.1a8312c7b164.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"redesigning-network-architecture","text":"Redesigning Network Architecture","depth":3},{"id":"introducing-modern-training-techniques","text":"Introducing Modern Training Techniques","depth":3},{"id":"macro-design","text":"Macro Design","depth":3},{"id":"resnext-ify","text":"ResNeXt-ify","depth":3},{"id":"inverted-bottleneck","text":"Inverted Bottleneck","depth":3},{"id":"large-kernel-convolution","text":"Large Kernel Convolution","depth":3},{"id":"micro-design","text":"Micro Design","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"summary-of-model-configurations","text":"Summary of Model Configurations","depth":3},{"id":"performance-on-imagenet","text":"Performance on ImageNet","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"vision-transformers/convmixer","relation":"comparison","topic":"Convolution for mixing visual patches"},{"id":"vision-transformers/caformer","relation":"comparison","topic":"Convolution for mixing visual patches"},{"id":"reparameterization/replknet","relation":"comparison","topic":"Large kernels and simpler convolutional backbones"}]},{"id":"vision-transformers/mae","category":"vision-transformers","legacyPath":"vision-transformers/2111-mae","legacySlug":"2111-mae","legacyPaths":[],"title":"MAE","subtitle":"A Quarter of the Clue","author":"Z. Yuan","paperDate":"2021-11-11","publishedOn":"2024-08-27","paperTitle":"Masked Autoencoders Are Scalable Vision Learners","paperUrl":"https://arxiv.org/abs/2111.06377","bibliography":{"title":"Masked Autoencoders Are Scalable Vision Learners","authors":["Kaiming He","Xinlei Chen","Saining Xie","Yanghao Li","Piotr Dollár","Ross Girshick"],"url":"https://arxiv.org/abs/2111.06377","pdfUrl":"https://arxiv.org/pdf/2111.06377","codeUrl":"https://github.com/facebookresearch/mae","preprint":{"id":"2111.06377","submittedOn":"2021-11-11","url":"https://arxiv.org/abs/2111.06377"},"publication":{"venue":"CVPR","year":2022,"kind":"conference","url":"https://openaccess.thecvf.com/content/CVPR2022/html/He_Masked_Autoencoders_Are_Scalable_Vision_Learners_CVPR_2022_paper.html"},"citation":"@misc{visiontransformersmae2021,\n  title = {{Masked Autoencoders Are Scalable Vision Learners}},\n  author = {Kaiming He and Xinlei Chen and Saining Xie and Yanghao Li and Piotr Dollár and Ross Girshick},\n  year = {2021},\n  eprint = {2111.06377},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2111.06377}\n}\n","citationPath":"/content/papers/citations/vision-transformers/mae.762f979b1c32.bib"},"thumbnail":{"src":"/content/papers/assets/f6a24c00fab9f102a2680899072c0e9de2ff2ae0cd7197666455a34c566f6542.jpg","width":1224,"height":716},"locale":"en","path":"/content/papers/en/vision-transformers/mae.b12d186b7307.md","headings":[{"id":"define-the-problem","text":"Define the problem","depth":2},{"id":"solve-the-problem","text":"Solve the problem","depth":2},{"id":"give-the-encoder-only-visible-patches","text":"Give the encoder only visible patches","depth":3},{"id":"reconstruction-target-and-loss","text":"Reconstruction target and loss","depth":3},{"id":"experimental-results-and-discussion","text":"Experimental results and discussion","depth":2},{"id":"establish-the-baseline-first","text":"Establish the baseline first","depth":3},{"id":"why-75-masking","text":"Why 75% masking?","depth":3},{"id":"the-decoder-need-not-match-the-encoders-weight","text":"The decoder need not match the encoder's weight","depth":3},{"id":"pixels-or-tokens-as-targets","text":"Pixels or tokens as targets?","depth":3},{"id":"the-retained-augmentations","text":"The retained augmentations","depth":3},{"id":"mask-shape-changes-the-task","text":"Mask shape changes the task","depth":3},{"id":"read-accuracy-with-training-time","text":"Read accuracy with training time","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{"defining-the-problem":"define-the-problem","solving-the-problem":"solve-the-problem","model-architecture":"give-the-encoder-only-visible-patches","training-method":"reconstruction-target-and-loss","discussion":"experimental-results-and-discussion","how-much-masking-is-optimal":"why-75-masking","decoder-parameters":"the-decoder-need-not-match-the-encoders-weight","learning-targets":"pixels-or-tokens-as-targets","data-augmentation":"the-retained-augmentations","masking-strategy":"mask-shape-changes-the-task","performance-on-imagenet":"read-accuracy-with-training-time","less-augmentation-does-not-mean-none":"the-retained-augmentations"},"guide":null,"relatedReading":[{"id":"vision-transformers/beit","relation":"comparison","topic":"Reconstructing pixels or visual tokens"},{"id":"image-generation/mar","relation":"comparison","topic":"Reconstructing pixels or visual tokens"}]}],"text":"ConvNeXt proved that a pure convolutional network can tie Swin Transformer under supervised learning. At around the same time, MAE set off a wave in self-supervised learning for vision. You mask out most of an image and have the model fill it back in, and ViTs pretrained this way score clearly better after fine-tuning than with supervised training.\n\n\n> [!TIP]\n> \n> For these two, you can start with our notes:\n> \n> - **[[22.01] ConvNeXt: Making Convolutions Great Again](/blog/papers/classic-cnns/convnext)**\n> - **[[21.11] MAE: A Quarter of the Clue](/blog/papers/vision-transformers/mae)**\n\n\nConnecting the two looks like the natural next step. Among this paper's authors, Zhuang Liu is the first author of ConvNeXt, Saining Xie is an author of both ConvNeXt and MAE, and Xinlei Chen is also an author of MAE. The first author is Sanghyun Woo of KAIST, who did this work during an internship at FAIR; he and fellow author In So Kweon are also both authors of CBAM.\n\nThey found that applying MAE directly to ConvNeXt doesn't work well.\n\n## Defining the Problem\n\n### Why MAE Doesn't Carry Over Easily\n\nMAE's encoder and decoder are asymmetric. After the image is cut into patches, 75% of them are masked, and the ViT encoder takes in only the remaining 25%. The masked parts never enter the encoder at all, so pretraining is fast and cheap. This design relies on the Transformer's ability to process sequences, which computes just fine with a few tokens missing.\n\nConvolutional networks don't have it so easy. Convolution slides over a complete 2D feature map and has to preserve the image's spatial structure, so the masked regions can't simply be taken out. A common workaround is to feed learnable mask tokens in at the input, but then pretraining saves no computation. And since there are no mask tokens at inference, training and testing are inconsistent, a problem that grows with the masking ratio.\n\nEven with these engineering problems solved, the authors observed a second problem. The features ConvNeXt learns under masked pretraining are flawed in themselves. We'll come back to that shortly.\n\n## Solving the Problem\n\n### FCMAE: A Fully Convolutional MAE\n\n[![Flowchart in four parts from left to right. On the far left is a 224×224 input drawn as 7×7 squares, red for visible patches and gray for masked ones, with about 60% gray. An arrow leads to the ConvNeXt encoder, whose box reads sparse convolution, computes visible patches only, and dense convolution plus a binary mask on TPU. Next comes the decoder, whose box reads one ConvNeXt block, dimension 512, plus mask tokens. On the far right is the reconstruction, again 7×7 squares, with only the originally masked squares marked in light red, showing that the MSE loss is computed only at those positions](/content/papers/assets/0af11b7bcef637ee0205fe47b31901b1b775a0336ccab0042e73e8d9643589c0.png)](/content/papers/assets/0af11b7bcef637ee0205fe47b31901b1b775a0336ccab0042e73e8d9643589c0.png)\n\n*Figure: Drawn for this note from Section 3 of the paper and the official code; the paper is licensed under CC BY-NC-ND 4.0, so the original figure is not reproduced here.*\n\nThe authors' solution is to treat the masked image as sparse data. The idea comes from 3D point clouds. Point clouds are sparse to begin with, and the sparse convolutions that process them compute only at positions that have data. A masked image works the same way, with the visible patches as the points that have data.\n\nDuring pretraining, the convolutions in the encoder are swapped for submanifold sparse convolutions, which compute only at visible positions and keep their outputs only at those positions. For fine-tuning, they're switched back to ordinary convolutions, reusing the weights directly. On accelerators like TPUs, the authors instead use ordinary convolution and multiply by a binary mask before and after each convolution. This is numerically identical, just more wasteful in computation. The paper's main experiments were all run this way on TPU v3.\n\nWhat happens with ordinary convolution? Let's stack three layers of a 7×7 depthwise convolution and see:\n\n```python\nimport torch\nimport torch.nn.functional as F\n\ntorch.manual_seed(0)\nkeep = torch.zeros(49)\nkeep[torch.randperm(49)[:19]] = 1                       # keep 19 of the 7×7 patches, masking about 60%\nmask = F.interpolate(keep.view(1, 1, 7, 7), size=56)    # upsample to the first stage's 56×56, 8×8 per patch\nx = torch.randn(1, 1, 56, 56) * mask                    # masked positions are 0\nhole = mask == 0\n\nw = torch.randn(1, 1, 7, 7) / 7                         # one 7×7 depthwise convolution\ndense, sparse = x, x\nfor layer in range(1, 4):\n    dense = F.conv2d(dense, w, padding=3)\n    sparse = F.conv2d(sparse, w, padding=3) * mask      # after each layer, reset the masked positions to 0\n    print(f\"Layer {layer}: share of masked positions holding values, ordinary convolution {(dense[hole] != 0).float().mean():.0%}, \"\n          f\"with mask {(sparse[hole] != 0).float().mean():.0%}\")\n```\n\n```\nLayer 1: share of masked positions holding values, ordinary convolution 50%, with mask 0%\nLayer 2: share of masked positions holding values, ordinary convolution 84%, with mask 0%\nLayer 3: share of masked positions holding values, ordinary convolution 100%, with mask 0%\n```\n\nOrdinary convolution treats the masked patches as an area of zero-valued pixels and computes over them anyway. The surrounding values seep in layer after layer, and after three layers the holes are completely filled, so the encoder ends up computing features at the masked positions too. With the mask, the holes stay holes, and the encoder works only at the visible positions, the same idea as MAE's ViT taking in only the visible patches. On ConvNeXt-B, accuracy is 79.3% without sparse convolution and 83.7% with it.\n\nThe rest is simple. The mask removes 60% of the image in 32×32 units; it's generated at the last stage's 7×7 resolution and upsampled stage by stage. Data augmentation is just random resized cropping. The decoder is a single ConvNeXt block with dimension 512. The authors also tried UNet and Transformer decoders, with similar accuracy, and a single ConvNeXt block had the shortest pretraining time, 7.7 hours against UNet's 12.9. As in MAE, the loss is MSE computed only on the masked patches, with the target being pixels normalized within each patch.\n\n### Changing Only the Pretraining Isn't Enough\n\nWith FCMAE in place, the authors pretrained ConvNeXt-B for 800 epochs and fine-tuned it for 100, reaching 83.7%. That beats the 82.7% of 100 epochs of supervised training from scratch. But the original ConvNeXt paper got 83.8% with 300 epochs of supervised training, so after a long detour through self-supervised pretraining, it still didn't beat supervised learning. ViTs pretrained with MAE, by contrast, clearly beat their supervised versions.\n\n### Feature Collapse\n\nThe authors plotted the activation map of every channel in the pretrained ConvNeXt and found that many channels were either dead across the whole map or saturated across the whole map, and the activations of different channels looked very much alike. The authors call this feature collapse. It shows up mainly in the MLP layer in each block that expands the dimension by 4×.\n\nTo quantify it, the authors computed the mean cosine distance between channels for each layer's features; the larger the distance, the more diverse the features. They compared several models on 1,000 images randomly drawn from the ImageNet validation set. ConvNeXt pretrained with FCMAE stays low all the way down, a clear case of collapse. Supervised ConvNeXt drops only in the last few layers, which the authors attribute to the cross-entropy loss making the model focus on the features classification needs. MAE-pretrained ViT stays high throughout.\n\n### GRN: Making Channels Compete\n\nThe authors took inspiration from lateral inhibition in biology. Neurons inhibit one another, which makes strong responses stand out more and makes the overall response more diverse. They designed a new normalization layer called Global Response Normalization (GRN), which works in three steps:\n\n1. **Global aggregation.** Compute each channel's L2 norm over the whole feature map, giving $\\|X_i\\|$.\n2. **Normalization.** Divide each channel's norm by the mean of all channels' norms to get the relative importance $N_i$. Channels above 1 are stronger than average, and those below 1 are weaker.\n3. **Calibration.** Multiply the original features by this relative importance.\n\nTo make training easier, the authors also add learnable $\\gamma$ and $\\beta$ and a residual connection:\n\n$$\nX_i \\leftarrow \\gamma\\, X_i\\, N_i + \\beta + X_i ,\n$$\n\n$\\gamma$ and $\\beta$ are initialized to 0, so GRN does nothing at first and only gradually takes effect during training. The official implementation is just three lines. Let's copy it and try it on five channels of different strengths:\n\n```python\nimport torch\n\ndef grn(x, gamma, beta, eps=1e-6):\n    # x: (N, H, W, C), channels last as in the official implementation\n    gx = torch.norm(x, p=2, dim=(1, 2), keepdim=True)   # L2 norm of each channel over the whole feature map\n    nx = gx / (gx.mean(dim=-1, keepdim=True) + eps)     # divided by the mean over all channels\n    return gamma * (x * nx) + beta + x\n\ntorch.manual_seed(0)\nstrength = torch.tensor([4.0, 2.0, 1.0, 0.5, 0.1])      # 5 channels of different strengths\nx = torch.randn(1, 14, 14, 5) * strength\n\nzero = torch.zeros(5)\nprint(\"With γ and β both 0, output equals input:\", torch.equal(grn(x, zero, zero), x))\n\ny = grn(x, torch.ones(5), zero)                         # suppose γ = 1 after training\ngx = torch.norm(x, p=2, dim=(1, 2)).flatten()\ngy = torch.norm(y, p=2, dim=(1, 2)).flatten()\nprint(\"Relative importance N:\", [round(v, 2) for v in (gx / gx.mean()).tolist()])\nprint(\"Norms before GRN:\", [round(v, 1) for v in gx.tolist()])\nprint(\"Norms after GRN:\", [round(v, 1) for v in gy.tolist()])\nprint(f\"Norm ratio of strongest to weakest channel: {gx.max() / gx.min():.0f}× → {gy.max() / gy.min():.0f}×\")\n```\n\n```\nWith γ and β both 0, output equals input: True\nRelative importance N: [2.64, 1.3, 0.66, 0.33, 0.07]\nNorms before GRN: [57.3, 28.1, 14.4, 7.2, 1.5]\nNorms after GRN: [208.6, 64.6, 23.9, 9.6, 1.6]\nNorm ratio of strongest to weakest channel: 39× → 132×\n```\n\nThe strongest channel is amplified 3.6×, while the weakest barely changes, so the gap between strong and weak widens from 39× to 132×. How much each channel gets amplified depends on how strong it is compared with the others, and that's the competition between channels.\n\nThe authors tried alternatives for each of the three steps, and the current version came out on top every time:\n\n| What's compared | Replaced with | ImageNet fine-tuning accuracy |\n| --- | --- | ---: |\n| Global aggregation | Global average pooling | 83.7% |\n| | L1 norm | 84.3% |\n| | **L2 norm** | **84.6%** |\n| Normalization layer | LRN | 83.2% |\n| | BatchNorm | 80.5% |\n| | LayerNorm | 83.8% |\n| | **GRN** | **84.6%** |\n| Feature re-weighting | SE (109M parameters) | 84.4% |\n| | CBAM (109M parameters) | 84.5% |\n| | **GRN (89M parameters)** | **84.6%** |\n| Stage where GRN is used | Pretraining only, removed for fine-tuning | 78.8% |\n| | Added only for fine-tuning | 80.6% |\n| | **Both stages** | **84.6%** |\n\n*Table: Compiled from Table 2 of the paper, ConvNeXt-B fine-tuned after 800 epochs of FCMAE pretraining. The baseline without GRN is 83.7%.*\n\nSE and CBAM can also do similar feature re-weighting, but they need 20M more parameters, while GRN adds none at all ($\\gamma$ and $\\beta$ are small enough to ignore). The last group is worth noting too. GRN has to be there for both pretraining and fine-tuning, and leaving it out of either one costs a lot.\n\n### The ConvNeXt V2 Block\n\nGRN goes right after the dimension expansion in the ConvNeXt block, the layer where the collapse is worst. With GRN in place, the authors found LayerScale had become redundant and removed it.\n\n[![Two-column block diagrams. The left column is the ConvNeXt V1 block, from top to bottom: a 7×7 depthwise convolution, LayerNorm, a 1×1 convolution expanding the channels from C to 4C, GELU, a 1×1 convolution reducing the channels from 4C back to C, and LayerScale, with the result added to the input; LayerScale is drawn as a dashed box, with a note below saying V2 removes it. The right column is the ConvNeXt V2 block, in the same order but with a red GRN box between GELU and the second 1×1 convolution, and no LayerScale; the note below says GRN computes a global L2 norm per channel and divides it by the mean over all channels](/content/papers/assets/149efd3dc24d22851f33f2ac0eb44a73e3dd77789df3b9f7e3427f4e09da8489.png)](/content/papers/assets/149efd3dc24d22851f33f2ac0eb44a73e3dd77789df3b9f7e3427f4e09da8489.png)\n\n*Figure: Drawn for this note from Section 4 of the paper and the official code.*\n\nApart from this one spot, ConvNeXt V2's architecture is exactly the same as V1's. The authors built it in eight sizes. The smallest, Atto, has just 3.7M parameters, followed by Femto, Pico, Nano, Tiny, Base, and Large, and the largest, Huge, at about 660M, is new in this paper.\n\n## Discussion\n\n### Architecture and Pretraining Have to Change Together\n\nThis paper's central result is this table:\n\n| Model | Supervised | FCMAE |\n| --- | ---: | ---: |\n| ConvNeXt V1-B | 83.8% | 83.7% |\n| ConvNeXt V2-B | 84.3% | **84.6%** |\n| ConvNeXt V1-L | 84.3% | 84.4% |\n| ConvNeXt V2-L | 84.5% | **85.6%** |\n\n*Table: Excerpted from Table 3 of the paper, ImageNet-1K fine-tuning accuracy; FCMAE pretraining for 800 epochs.*\n\nV1 with FCMAE pretraining barely improves. V2 with supervised training doesn't improve much either, with L gaining just 0.2. Use both together, and B gains 0.8 while L gains 1.3. Changing only the architecture or only the pretraining isn't enough, and the authors call this co-design.\n\nAcross the eight sizes, the result holds:\n\n| Size | Parameters | V1 supervised | V2 supervised | V2 + FCMAE |\n| --- | ---: | ---: | ---: | ---: |\n| Atto | 3.7M | 75.7% | 76.2% | **76.7%** |\n| Femto | 5.2M | 77.5% | 78.0% | **78.5%** |\n| Pico | 9.1M | 79.5% | 79.7% | **80.3%** |\n| Nano | 15.6M | 80.8% | 81.2% | **81.9%** |\n| Tiny | 28.6M | 82.1% | 82.5% | **83.0%** |\n| Base | 89M | 83.8% | 84.3% | **84.9%** |\n| Large | 198M | 84.3% | 84.5% | **85.8%** |\n| Huge | 660M | — | — | **86.3%** |\n\n*Table: Excerpted from Table 14 of the paper, 224×224 input; the FCMAE pretraining here is 1600 epochs.*\n\nEvery size improves by 0.8 to 1.5 percentage points, with Large improving the most.\n\nTo be clear, though, these numbers compare accuracy after fine-tuning, and training cost is left out of the comparison. FCMAE first pretrains for 1600 epochs, after which Base and Large are fine-tuned for 100 epochs; the small models from Atto to Nano need 600 epochs for the fine-tuning alone.\n\n### Against Transformers with MAE\n\nWith pretraining on ImageNet-1K only, compared with other masked pretraining methods:\n\n| Size | ViT + MAE | Swin + SimMIM | ConvNeXt V2 + FCMAE |\n| --- | ---: | ---: | ---: |\n| B | 83.6% | 84.0% | **84.9%** |\n| L | **85.9%** | 85.4% | 85.8% |\n| H | **86.9%** | 85.7% (SwinV2) | 86.3% |\n\n*Table: Excerpted from Table 4 of the paper. The Swin column is the result of 800 epochs of pretraining; the others use each method's best setting.*\n\nAgainst Swin + SimMIM, ConvNeXt V2 wins at every size. Against ViT + MAE, L is roughly a tie, with ViT-L having over 50% more parameters (307M versus 198M); at H, ConvNeXt V2 trails by 0.6. The authors' explanation is that large ViTs get more out of self-supervised pretraining.\n\nAdding a step of supervised intermediate fine-tuning on ImageNet-22K, ConvNeXt V2-H reaches 88.9% with 512×512 input, the best result at the time using only public data. V2's Base and Large also beat V1's next sizes up, Large and XLarge, respectively.\n\nThe appendix also includes a comparison with contrastive learning. With the same ConvNeXt V2-B, fine-tuning after MoCo v3 pretraining gives 83.7%, lower than the 84.3% of supervised training, while FCMAE gives 84.9%.\n\n### Detection and Segmentation\n\nOn COCO with Mask R-CNN, box AP climbs step by step from V1 supervised to V2 supervised to V2 + FCMAE, going 50.3, 51.0, and 52.9 for Base and 50.6, 51.5, and 54.4 for Large, with Huge reaching 55.7. Against Swin pretrained with SimMIM, it wins at every size.\n\nOn ADE20K with UperNet, V2 + FCMAE is also much better than V1 supervised, rising from 49.9 to 52.1 for Base and from 50.5 to 53.7 for Large. Swin-B + SimMIM, however, gets 52.8, so at the Base level ConvNeXt V2 actually loses by 0.7. The paper's wording is that it “performs comparably” with Swin at Base and Large and only wins at Huge.\n\n## Conclusion\n\nConvNeXt has learned to fill in the blanks, on the condition that the architecture changes too. Sparse convolution during pretraining lets it look only at the visible patches, the way ViT does; GRN in the block makes the channels compete with one another so they stop collapsing into a single mass. Only with both in place did masked pretraining become truly useful for convolutional networks for the first time.\n\nGRN is just three lines and costs no extra parameters, which makes it the easiest thing to take home from this paper. FCMAE is far less approachable. The paper pretrains on TPU v3-256 for 800 to 1600 epochs, in exchange for roughly 1 percentage point at each size.\n\nSo the more practical route is to fine-tune straight from the [official weights](https://github.com/facebookresearch/ConvNeXt-V2), and timm has every size from Atto to Huge.\n\nLet the TPUs do the fill-in-the-blank homework; we'll just copy the answers.\n","imageSizes":{"/content/papers/assets/0af11b7bcef637ee0205fe47b31901b1b775a0336ccab0042e73e8d9643589c0.png":{"width":1640,"height":600},"/content/papers/assets/149efd3dc24d22851f33f2ac0eb44a73e3dd77789df3b9f7e3427f4e09da8489.png":{"width":1640,"height":1000}}}