Back to paper library
arXiv 2022CVPR 2022Classic CNNsPaper notes

ConvNeXt

Making Convolutions Great Again

Original paper

A ConvNet for the 2020s

Paper authors

Zhuang Liu · Hanzi Mao · Chao-Yuan Wu · Christoph Feichtenhofer · Trevor Darrell · Saining Xie

Publication
CVPR 2022Conference
First submitted to arXiv
arXiv:2201.03545
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{classiccnnsconvnext2022,
  title = {{A ConvNet for the 2020s}},
  author = {Zhuang Liu and Hanzi Mao and Chao-Yuan Wu and Christoph Feichtenhofer and Trevor Darrell and Saining Xie},
  year = {2022},
  eprint = {2201.03545},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2201.03545}
}
On this page

Looking back at the 2010s, deep learning made significant strides, profoundly impacting multiple fields.

At the heart of this progress was the revival of neural networks, particularly Convolutional Neural Networks (ConvNet).

Over the past decade, the field of visual recognition has successfully transitioned from handcrafted features to the design of ConvNet architectures.

Although the concept of ConvNet dates back to the 1980s, it wasn't until 2012 that we truly witnessed its potential in visual feature learning, marked by the emergence of AlexNet, which heralded the "ImageNet moment" in computer vision.

Since then, the field has advanced rapidly with landmark works such as:

These seminal works emphasized efficiency and scalability, popularizing many practical design principles.

But after ViT arrived, the conversation changed abruptly.

Defining the Problem

Vision Transformers (ViT) changed how researchers thought about network design. Apart from the initial layer that splits an image into patches, the original ViT made relatively few changes to the NLP Transformer. Patchification itself still gives the model some image structure.

One significant advantage of ViT is its scalability: as the model and dataset sizes grow, its performance significantly surpasses standard ResNet.

To close the performance gap on more vision tasks, hierarchical Transformers adopted local designs. Swin, for example, uses shifted local windows rather than a literal sliding window, bringing some ConvNet-like locality back to attention.

Swin Transformer was a milestone in this regard, showing that Transformers could serve as general-purpose vision backbones for classification and dense prediction tasks.

Its success also suggests that locality and hierarchy, long familiar in ConvNets, still matter.

From these advancements, it becomes apparent that many of the improvements in Vision Transformers are essentially restoring the strengths of convolutions.

These attempts have a cost: windowed attention, cyclic shifts and relative position bias make implementation more involved.

Ironically, ConvNet already possesses many of the required attributes, albeit in a simple, straightforward manner.

ConvNets lost research momentum as Transformers surpassed older ConvNet baselines on many vision tasks.

The authors argue: it shouldn't be this way.

Solving the Problem

Redesigning Network Architecture

Paper Figure 2: dark bars show top-1 accuracy in the roughly 4.5G-FLOP ResNet-50/Swin-T regime; gray bars show the larger ResNet-200/Swin-B regime. Stars mark GFLOPs at each step, and hatched alternatives were not adopted. The 82.0% endpoint belongs to this ablation; Table 1 later reports 82.1% for its model row.

For simplicity, the authors start with ResNet-50 and use the similarly sized Swin-T as a comparison. They also check a larger ResNet-200/Swin-B regime. Each change follows the previous one, so the gains cannot be added as if they were independent experiments.

Introducing Modern Training Techniques

ViT brought attention to modern training recipes along with its architecture. Before changing the ResNet blocks, the authors first apply a comparable recipe to the ConvNet. Some of these techniques predate ViT; it did not invent them all.

Macro Design

This part considers two design factors:

  1. Stage Compute Ratio

    In ResNet, the cross-stage compute distribution design is mainly empirical.

    For instance, ResNet's stage-4 is designed to be compatible with downstream tasks (like object detection) where detector heads operate on a 14×14 feature plane.

    Swin-T puts more blocks in the third stage; its block-count pattern is (2, 2, 6, 2), and larger Swin variants deepen that stage further. The paper's informal 1:1:3:1 and 1:1:9:1 descriptions should not be read as exact measured FLOP ratios.

    To maintain similar FLOPs to Swin-T, the authors adjusted ResNet-50's block numbers per stage from (3, 4, 6, 3) to (3, 3, 9, 3).

From this point onward, the model uses the (3, 3, 9, 3) block pattern.

  1. Stem Structure Design

    In ConvNet, the stem refers to the initial layer of input.

    Typically, it is used for downsampling to quickly reduce the spatial size of the input.

    In ViT, the stem is a patchify layer that splits the image into a series of patches using a large kernel convolution of 16×16.

    In Swin-T, the stem is a 4x4 large convolution with a stride of 4.

    Here, the authors adopted the same design as Swin-T and adjusted the ResNet-50's stem to a 4×4 non-overlapping convolution.

ResNeXt-ify

In this section, the authors adopted the ResNeXt design philosophy, which offers a better FLOPs/accuracy trade-off compared to a standard ResNet.

The core design of ResNeXt is grouped convolution, where convolutional filters are divided into different groups.

The guiding principle of ResNeXt is "use more groups, expand width," applying grouped convolution in the 3×3 convolution layer.

In this paper, the authors used depthwise convolution, a special form of grouped convolution where the number of groups equals the number of channels.

Depthwise convolution mixes spatial information within each channel; later 1×1 layers mix channels. Like self-attention, it combines information across positions, but its kernel weights are fixed after training rather than recomputed from each input.

Using depthwise convolution alone reduces FLOPs from about 4.4G to 2.4G, while top-1 accuracy falls from 79.5% to 78.3% in this ablation.

Following ResNeXt's strategy, the authors increased the network width to match the same channel numbers as Swin-T (from 64 to 96).

Inverted Bottleneck

Paper Figure 3: (a) compresses 384 channels to 96 before depthwise convolution; (b) expands to 384 first, forming the paper's MobileNetV2-inspired variant; (c) moves depthwise convolution before 1×1 expansion. These are the block configurations tested here.

Each block in Transformers has an inverted bottleneck structure, typically with an expansion ratio of 4.

MobileNetV2 is another example of an inverted bottleneck with a depthwise convolution after expansion. Figure 3(b) is the ConvNeXt paper's variant inspired by it, not a diagram of MobileNetV2's original block.

In the figure above, (a) shows the ResNeXt-style bottleneck, (b) the inverted variant, and (c) another option that moves the 3×3 depthwise convolution to the front. That last choice prepares for the large-kernel experiment below.

Large Kernel Convolution

One important characteristic of ViT is its non-local self-attention, granting the network global receptive fields in each layer.

This contrasts with traditional ConvNet, which uses smaller kernels (like 3×3, popularized by VGGNet).

To explore the application of large kernels, the article proposed moving the depthwise convolution layer upward, akin to placing the MHSA block before the MLP layer in Transformers.

Moving the depthwise convolution to (c) reduces FLOPs to about 4.1G, but by itself lowers accuracy from 80.6% to 79.9%.

After these adjustments, the study tested different kernel sizes (from 3×3 to 11×11):

  • 3x3: 79.9%
  • 7x7: 80.6%

The authors find that a 7×7 kernel restores accuracy from 79.9% to 80.6%, with FLOPs around 4.2G. The kernel is still local; it does not give every layer ViT's global attention.

Further increasing kernel size (beyond 7×7) showed no additional performance gains in the ResNet-200 model, indicating performance saturation at the 7×7 size.

Micro Design

  1. Replacing ReLU with GELU

    Rectified Linear Unit (ReLU) has been the preferred activation function in ConvNet due to its simplicity and computational efficiency.

    Gaussian Error Linear Unit (GELU), a smoother variant of ReLU, has gradually gained favor in state-of-the-art Transformer models, such as Google's BERT and OpenAI's GPT-2.

    The authors replaced ReLU with GELU in ConvNet, and the model's accuracy remained unchanged.

  1. Reducing Activation Functions

    Paper Figure 4: Swin has separate attention and MLP residual paths; ResNet and ConvNeXt each have one. ConvNeXt applies 7×7 depthwise convolution, LayerNorm, 1×1 expansion, GELU and 1×1 projection before adding the shortcut.

    In Transformer architectures, particularly in the MLP block, typically only one activation function is used.

    In contrast, the practice in ResNet modules is to use activation functions after each convolution layer, even after small 1×1 convolution layers.

    The authors removed all GELU activation layers in the residual network block except for one between the two 1×1 convolution layers, mimicking the Transformer block's style.

    This adjustment improves accuracy by 0.7 percentage points, from 80.6% to 81.3%, close to Swin-T in this setup.

  1. Reducing Normalization Layers

    Transformers generally have fewer normalization layers, whereas in ConvNet, Batch Normalization appears after each convolution layer.

    The authors remove two BatchNorm layers, keeping one after the depthwise convolution and before the 1×1 expansion.

  1. Replacing BatchNorm with LayerNorm

    Transformers use simpler Layer Normalization, achieving good performance across different application scenarios.

    Directly replacing BN with LN in the original ResNet results in suboptimal performance.

    With all the modifications in network architecture and training techniques, the authors revisited the impact of replacing BN with LN, finding no difficulty in training the ConvNet model using LN.

  1. Downsampling Layer

    In the ResNet architecture, spatial downsampling is performed at the beginning of each stage using 3×3 convolutions with a stride of 2 and 1×1 convolutions with a stride of 2 in the shortcut connections.

    Compared with ResNet, Swin Transformers add separate downsampling layers between stages. The authors try an independent 2×2, stride-2 convolution, but this change alone makes training diverge.

    Further investigation showed that adding normalization layers where spatial resolution changes can help stabilize training.

    This includes multiple Layer Normalization (LN) layers used in Swin Transformers: one before each downsampling layer, one after the stem, and one after the final global average pooling.

    Based on these results, the authors decided to use separate downsampling layers in their final model, naming it ConvNeXt.

Discussion

Summary of Model Configurations

These models represent a "modernized" upgrade to the ResNet architecture, with different variants primarily differing in the number of channels (C) and the number of blocks per stage (B).

The specific configurations for each variant are as follows:

  1. ConvNeXt-T (Tiny):

    • Channels C = (96, 192, 384, 768)
    • Blocks per stage B = (3, 3, 9, 3)
  2. ConvNeXt-S (Small):

    • Channels C = (96, 192, 384, 768)
    • Blocks per stage B = (3, 3, 27, 3)
  3. ConvNeXt-B (Base):

    • Channels C = (128, 256, 512, 1024)
    • Blocks per stage B = (3, 3, 27, 3)
  4. ConvNeXt-L (Large):

    • Channels C = (192, 384, 768, 1536)
    • Blocks per stage B = (3, 3, 27, 3)
  5. ConvNeXt-XL (Extra Large):

    • Channels C = (256, 512, 1024, 2048)
    • Blocks per stage B = (3, 3, 27, 3)

The number of channels doubles with each new stage, following a design philosophy similar to ResNet and Swin Transformers.

This staged design offers several model sizes for different data and compute budgets.

Performance on ImageNet

Paper Table 1: upper rows train directly on ImageNet-1K; lower rows pretrain on ImageNet-22K and fine-tune on 1K. Top-1 accuracy is on the 1K validation set, and throughput is measured on a V100 GPU. Resolution and training data differ across rows.

The roadmap's 82.0% is the three-seed mean for the ResNet-50-scale ablation; this table reports 82.1% for the final ConvNeXt-T model.

  1. ImageNet-1K Performance Comparison:

    • ConvNeXt vs. ConvNet Benchmarks: ConvNeXt competes with strong ConvNet benchmarks (like RegNet and EfficientNet) in terms of accuracy and inference throughput.
    • Comparison with Swin Transformers: At 224×224, ConvNeXt-T and Swin-T both use 4.5G FLOPs; their top-1 accuracies are 82.1% and 81.3%, a 0.8-percentage-point gap. V100 throughput is 774.7 versus 757.9 images/s. ConvNeXt needs neither shifted windows nor relative position bias.
    • Specific model performance: At 384×384, ConvNeXt-B versus Swin-B reaches 85.1% versus 84.5% top-1, 95.7 versus 85.1 images/s and 45.0G versus 47.1G FLOPs. That is a 0.6-point accuracy gain and about 12.5% higher V100 throughput under this setup.
  2. ImageNet-22K Performance Comparison:

    • Impact of large-scale pretraining: A carefully designed ConvNet can match or exceed similarly sized Swin models after ImageNet-22K pretraining. At 384×384, ConvNeXt-B and Swin-B reach 86.8% and 86.4% top-1 on ImageNet-1K after fine-tuning; their inference-throughput comparison uses the paper's V100 setup.
    • ConvNeXt-XL performance: After ImageNet-22K pretraining and ImageNet-1K fine-tuning, ConvNeXt-XL reaches 87.8% top-1 at 384×384, showing that this design can scale.
  3. Advantages of the ConvNeXt Architecture:

    • Among the directly ImageNet-1K-trained rows, EfficientNetV2-L with progressive training reaches one of the strongest accuracies; training recipes differ across models.
    • After large-scale pretraining, ConvNeXt-XL reports 87.8% versus EfficientNetV2-XL (21k) at 87.3%. ConvNeXt-XL also uses 350M versus 208M parameters and 179.0G versus 94.0G FLOPs, while V100 throughput is 30.2 versus 56.5 images/s. The pretraining datasets and procedures are not identical, so the accuracy gap comes with a larger, slower model in this benchmark.

Conclusion

In the 2020s, hierarchical vision Transformers such as Swin made traditional ConvNets look like the weaker choice.

ConvNeXt shows that a simple, carefully modernized ConvNet can still compete with leading vision Transformers in the reported ImageNet, COCO and ADE20K experiments. This does not settle every task or deployment setting; data, memory and target hardware still matter.

This battle is just beginning!