{"notes":[{"id":"classic-cnns/inceptionnext","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"InceptionNeXt","subtitle":"Split into a Cross","author":"Z. Yuan","paperDate":"2023-03-29","publishedOn":"2025-10-30","world":"external/inceptionnext","paperTitle":"InceptionNeXt: When Inception Meets ConvNeXt","paperUrl":"https://arxiv.org/abs/2303.16900","bibliography":{"title":"InceptionNeXt: When Inception Meets ConvNeXt","authors":["Weihao Yu","Pan Zhou","Shuicheng Yan","Xinchao Wang"],"url":"https://arxiv.org/abs/2303.16900","pdfUrl":"https://arxiv.org/pdf/2303.16900v3","codeUrl":"https://github.com/sail-sg/inceptionnext","preprint":{"id":"2303.16900","submittedOn":"2023-03-29","url":"https://arxiv.org/abs/2303.16900"},"publication":{"venue":"CVPR","year":2024,"kind":"conference","url":"https://openaccess.thecvf.com/content/CVPR2024/html/Yu_InceptionNeXt_When_Inception_Meets_ConvNeXt_CVPR_2024_paper.html"},"citation":"@misc{classiccnnsinceptionnext2023,\n  title = {{InceptionNeXt: When Inception Meets ConvNeXt}},\n  author = {Weihao Yu and Pan Zhou and Shuicheng Yan and Xinchao Wang},\n  year = {2023},\n  eprint = {2303.16900},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2303.16900}\n}\n","citationPath":"/content/papers/citations/classic-cnns/inceptionnext.82116f5daa6e.bib"},"thumbnail":{"src":"/content/papers/assets/1c7bbb1c754725c6cd60caf24d092587cc19bbcf03b619f64821ad4a7fb93429.png","width":1446,"height":786},"locale":"en","path":"/content/papers/en/classic-cnns/inceptionnext.5a5e98fee98e.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"metanext","text":"MetaNeXt","depth":3},{"id":"where-the-time-goes","text":"Where the Time Goes","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"inception-depthwise-convolution","text":"Inception Depthwise Convolution","depth":3},{"id":"a-cross-shaped-receptive-field","text":"A Cross-Shaped Receptive Field","depth":3},{"id":"building-the-network","text":"Building the Network","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"imagenet-1k","text":"ImageNet-1K","depth":3},{"id":"metanext-cannot-accommodate-attention","text":"MetaNeXt Cannot Accommodate Attention","depth":3},{"id":"semantic-segmentation","text":"Semantic Segmentation","depth":3},{"id":"ablation","text":"Ablation","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/convnext","relation":"referenced","topic":""},{"id":"vision-transformers/poolformer","relation":"referenced","topic":""},{"id":"classic-cnns/inception-v3","relation":"referenced","topic":""}]},{"id":"classic-cnns/convnext","category":"classic-cnns","legacyPath":"classic-cnns/2201-convnext","legacySlug":"2201-convnext","legacyPaths":[],"title":"ConvNeXt","subtitle":"Making Convolutions Great Again","author":"Z. Yuan","paperDate":"2022-01-10","publishedOn":"2024-06-30","paperTitle":"A ConvNet for the 2020s","paperUrl":"https://arxiv.org/abs/2201.03545","bibliography":{"title":"A ConvNet for the 2020s","authors":["Zhuang Liu","Hanzi Mao","Chao-Yuan Wu","Christoph Feichtenhofer","Trevor Darrell","Saining Xie"],"url":"https://arxiv.org/abs/2201.03545","pdfUrl":"https://arxiv.org/pdf/2201.03545","codeUrl":"https://github.com/facebookresearch/ConvNeXt","preprint":{"id":"2201.03545","submittedOn":"2022-01-10","url":"https://arxiv.org/abs/2201.03545"},"publication":{"venue":"CVPR","year":2022,"kind":"conference","url":"https://arxiv.org/abs/2201.03545"},"citation":"@misc{classiccnnsconvnext2022,\n  title = {{A ConvNet for the 2020s}},\n  author = {Zhuang Liu and Hanzi Mao and Chao-Yuan Wu and Christoph Feichtenhofer and Trevor Darrell and Saining Xie},\n  year = {2022},\n  eprint = {2201.03545},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2201.03545}\n}\n","citationPath":"/content/papers/citations/classic-cnns/convnext.801b4060b807.bib"},"thumbnail":{"src":"/content/papers/assets/37882616c5c72df33699003256d14bf7c58c8ea5d1e9e4fb5b1c33268c9a6923.jpg","width":1224,"height":432},"locale":"en","path":"/content/papers/en/classic-cnns/convnext.1a8312c7b164.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"redesigning-network-architecture","text":"Redesigning Network Architecture","depth":3},{"id":"introducing-modern-training-techniques","text":"Introducing Modern Training Techniques","depth":3},{"id":"macro-design","text":"Macro Design","depth":3},{"id":"resnext-ify","text":"ResNeXt-ify","depth":3},{"id":"inverted-bottleneck","text":"Inverted Bottleneck","depth":3},{"id":"large-kernel-convolution","text":"Large Kernel Convolution","depth":3},{"id":"micro-design","text":"Micro Design","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"summary-of-model-configurations","text":"Summary of Model Configurations","depth":3},{"id":"performance-on-imagenet","text":"Performance on ImageNet","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"vision-transformers/convmixer","relation":"comparison","topic":"Convolution for mixing visual patches"},{"id":"vision-transformers/caformer","relation":"comparison","topic":"Convolution for mixing visual patches"},{"id":"reparameterization/replknet","relation":"comparison","topic":"Large kernels and simpler convolutional backbones"}]},{"id":"vision-transformers/poolformer","category":"vision-transformers","legacyPath":"vision-transformers/2111-poolformer","legacySlug":"2111-poolformer","legacyPaths":[],"title":"PoolFormer","subtitle":"You Need a Meta!","author":"Z. Yuan","paperDate":"2021-11-22","publishedOn":"2024-07-10","paperTitle":"MetaFormer Is Actually What You Need for Vision","paperUrl":"https://openaccess.thecvf.com/content/CVPR2022/html/Yu_MetaFormer_Is_Actually_What_You_Need_for_Vision_CVPR_2022_paper.html","bibliography":{"title":"MetaFormer Is Actually What You Need for Vision","authors":["Weihao Yu","Mi Luo","Pan Zhou","Chenyang Si","Yichen Zhou","Xinchao Wang","Jiashi Feng","Shuicheng Yan"],"url":"https://openaccess.thecvf.com/content/CVPR2022/html/Yu_MetaFormer_Is_Actually_What_You_Need_for_Vision_CVPR_2022_paper.html","pdfUrl":"https://openaccess.thecvf.com/content/CVPR2022/papers/Yu_MetaFormer_Is_Actually_What_You_Need_for_Vision_CVPR_2022_paper.pdf","codeUrl":"https://github.com/sail-sg/poolformer","preprint":{"id":"2111.11418","submittedOn":"2021-11-22","url":"https://arxiv.org/abs/2111.11418"},"publication":{"venue":"CVPR","year":2022,"kind":"conference","url":"https://openaccess.thecvf.com/content/CVPR2022/html/Yu_MetaFormer_Is_Actually_What_You_Need_for_Vision_CVPR_2022_paper.html"},"citation":"@misc{visiontransformerspoolformer2021,\n  title = {{MetaFormer Is Actually What You Need for Vision}},\n  author = {Weihao Yu and Mi Luo and Pan Zhou and Chenyang Si and Yichen Zhou and Xinchao Wang and Jiashi Feng and Shuicheng Yan},\n  year = {2021},\n  eprint = {2111.11418},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2111.11418}\n}\n","citationPath":"/content/papers/citations/vision-transformers/poolformer.f2f96629d283.bib"},"thumbnail":{"src":"/content/papers/assets/3cb141e9bb1247410c8214276b21b28ed1d536060d470406f1bfb0c88fe68df2.jpg","width":1498,"height":548},"locale":"en","path":"/content/papers/en/vision-transformers/poolformer.ceadb42af53a.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"input-embedding","text":"Input Embedding","depth":3},{"id":"token-mixer","text":"Token-Mixer","depth":3},{"id":"channel-mlp","text":"Channel MLP","depth":3},{"id":"is-token-mixer-really-the-key","text":"Is Token-Mixer Really the Key?","depth":3},{"id":"solution","text":"Solution","depth":2},{"id":"model-architecture","text":"Model Architecture","depth":3},{"id":"model-configuration","text":"Model Configuration","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"imagenet-experiments","text":"ImageNet Experiments","depth":3},{"id":"ablation-experiments","text":"Ablation Experiments","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"vision-transformers/mlp-mixer","relation":"background","topic":"Separating the backbone from token mixing"},{"id":"vision-transformers/caformer","relation":"next","topic":"Separating the backbone from token mixing"}]},{"id":"classic-cnns/inception-v3","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Inception-v3","subtitle":"Big Kernels Bow Out","author":"Z. Yuan","paperDate":"2015-12-02","publishedOn":"2023-11-15","world":"external/inception-v3","paperTitle":"Rethinking the Inception Architecture for Computer Vision","paperUrl":"https://arxiv.org/abs/1512.00567","bibliography":{"title":"Rethinking the Inception Architecture for Computer Vision","authors":["Christian Szegedy","Vincent Vanhoucke","Sergey Ioffe","Jonathon Shlens","Zbigniew Wojna"],"url":"https://arxiv.org/abs/1512.00567","pdfUrl":"https://arxiv.org/pdf/1512.00567v3","codeUrl":null,"preprint":{"id":"1512.00567","submittedOn":"2015-12-02","url":"https://arxiv.org/abs/1512.00567"},"publication":{"venue":"CVPR","year":2016,"kind":"conference","url":"https://openaccess.thecvf.com/content_cvpr_2016/html/Szegedy_Rethinking_the_Inception_CVPR_2016_paper.html"},"citation":"@misc{classiccnnsinceptionv32015,\n  title = {{Rethinking the Inception Architecture for Computer Vision}},\n  author = {Christian Szegedy and Vincent Vanhoucke and Sergey Ioffe and Jonathon Shlens and Zbigniew Wojna},\n  year = {2015},\n  eprint = {1512.00567},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1512.00567}\n}\n","citationPath":"/content/papers/citations/classic-cnns/inception-v3.5636c0bddf2d.bib"},"thumbnail":{"src":"/content/papers/assets/c87d3d6fa9ea76d954eba2570f702150eca32d62c7cdb3d724075a4be9a63080.png","width":419,"height":358},"locale":"en","path":"/content/papers/en/classic-cnns/inception-v3.e66992f80aec.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"splitting-55-into-two-33-layers","text":"Splitting 5×5 into Two 3×3 Layers","depth":3},{"id":"splitting-further-into-1n-and-n1","text":"Splitting Further into 1×n and n×1","depth":3},{"id":"what-auxiliary-classifiers-really-do","text":"What Auxiliary Classifiers Really Do","depth":3},{"id":"shrinking-the-feature-map-without-choking-the-information","text":"Shrinking the Feature Map Without Choking the Information","depth":3},{"id":"putting-the-pieces-together","text":"Putting the Pieces Together","depth":3},{"id":"keeping-the-model-from-getting-too-sure-of-itself","text":"Keeping the Model from Getting Too Sure of Itself","depth":3},{"id":"training-setup","text":"Training Setup","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"adding-one-change-at-a-time","text":"Adding One Change at a Time","depth":3},{"id":"how-much-does-lower-resolution-cost","text":"How Much Does Lower Resolution Cost?","depth":3},{"id":"comparison-with-other-models","text":"Comparison with Other Models","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/inception-v1","relation":"referenced","topic":""},{"id":"normalization/batchnorm","relation":"referenced","topic":""},{"id":"classic-cnns/convnext","relation":"referenced","topic":""}]}],"text":"ConvNeXt-T takes 4.5 GMACs, while ResNet-50 takes 4.1 GMACs. The two are close. Train them on an A100, though, and ResNet-50 processes 969 images per second; ConvNeXt-T manages just 575, about 60% as many.\n\nWhy such a large speed gap at similar compute? The bottleneck is the 7×7 depthwise convolution. Each channel works on its own, with few multiply-adds but plenty of data to move, leaving the GPU waiting on memory much of the time. Expanding ConvNeXt's kernel from 3×3 to 7×7 brought better accuracy along with this speed problem.\n\nThis paper comes from the National University of Singapore, Singapore Management University, Sea AI Lab, and Skywork AI. First author Weihao Yu also wrote MetaFormer, which broke Transformers down into a “token mixer plus MLP” framework. Here, he applies the same way of breaking things down to ConvNeXt.\n\n\n> [!TIP]\n> \n> For ConvNeXt and MetaFormer, start with these two notes:\n> \n> - **[[22.01] ConvNeXt: Making Convolutions Great Again](/blog/papers/classic-cnns/convnext)**\n> - **[[21.11] PoolFormer: You Need a Meta!](/blog/papers/vision-transformers/poolformer)**\n\n\n[![Figure 1 from the paper: the horizontal axis is training throughput on an A100 in images per second, and the vertical axis is ImageNet top-1 accuracy. ConvNeXt-T/k7 sits toward the upper left at about 82.1% and 575 images/s; switching to 5×5 and 3×3 kernels moves it rightward while accuracy drops. ResNet-50 is at the lower right, at about 970 images/s but just over 78% accuracy. InceptionNeXt-T sits at the upper right, with a red arrow marking a 1.6× speedup over ConvNeXt-T/k7.](/content/papers/assets/1f6716ae5b81879c492b1f16031d8c3a3e65f652d97318b88209b2b60ec9d7c4.png)](/content/papers/assets/1f6716ae5b81879c492b1f16031d8c3a3e65f652d97318b88209b2b60ec9d7c4.png)\n\n*Figure 1 from Yu et al., taken from [**PDF page 1**](https://arxiv.org/pdf/2303.16900v3#page=1). The original caption is omitted.*\n\nThe upper-right corner is where this paper aims to be: accuracy on par with ConvNeXt-T, at a speed close to ResNet-50.\n\n## Defining the Problem\n\n### MetaNeXt\n\nThe authors first abstract the ConvNeXt block.\n\n[![Figure 2 from the paper: four blocks arranged from left to right. MetaFormer has two residual connections, one enclosing Norm and a dashed Token Mixer box, the other enclosing Norm and MLP. MetaNeXt has one residual connection, with Token Mixer, Norm, and MLP along the same path. ConvNeXt replaces the Token Mixer with a 7×7 DWConv. InceptionNeXt's Token Mixer is a dashed box that splits the input into four parts, applies 3×3, 1×11, and 11×1 DWConv and an identity branch, then concatenates them.](/content/papers/assets/1c7bbb1c754725c6cd60caf24d092587cc19bbcf03b619f64821ad4a7fb93429.png)](/content/papers/assets/1c7bbb1c754725c6cd60caf24d092587cc19bbcf03b619f64821ad4a7fb93429.png)\n\n*Figure 2 from Yu et al., taken from [**PDF page 2**](https://arxiv.org/pdf/2303.16900v3#page=2). The original caption is omitted.*\n\nThe MetaFormer block on the far left has two residual connections: one for the token mixer and one for the MLP. Transformers use this structure. ConvNeXt has just one residual connection, with spatial mixing, normalization, and the MLP along a single path. The authors call this single-residual framework MetaNeXt. ConvNeXt and the paper's InceptionNeXt are two implementations, differing only in their token mixers.\n\n### Where the Time Goes\n\nThe authors run a preliminary experiment on ConvNeXt-T, changing only the depthwise convolution:\n\n| Depthwise kernel | Fraction of channels convolved | Training (images/s) | Inference (images/s) | top-1 |\n| --- | ---: | ---: | ---: | ---: |\n| 7×7 | 1 | 575 | 2413 | 82.1% |\n| 5×5 | 1 | 675 | 2704 | 82.0% |\n| 3×3 | 1 | 798 | 2802 | 81.5% |\n| 3×3 | 1/4 | 871 | 2808 | 81.3% |\n| 3×3 | 1/8 | 901 | 2833 | 80.8% |\n| 3×3 | 1/16 | 916 | 2846 | 80.1% |\n\n*Table: selected rows from the paper's Table 1. A100, batch size 128, TF32.*\n\nShrinking the kernel from 7×7 to 3×3 makes training 39% faster and costs 0.6 percentage points of accuracy. Taking a cue from ShuffleNet V2, the authors then apply the depthwise convolution to only some channels and leave the rest untouched. At a fraction of 1/4, training gets a little faster again, losing just another 0.2 points.\n\nTwo findings emerge. Large-kernel depthwise convolution is the speed bottleneck, and each depthwise layer needs to process only a fraction of the channels. The problem is that a smaller kernel also shrinks the receptive field. Where can those 0.6 points be recovered?\n\n## Solving the Problem\n\n### Inception Depthwise Convolution\n\nThe authors split the channels into four groups, each doing something different:\n\n- A 3×3 depthwise convolution\n- A 1×11 horizontal band convolution\n- An 11×1 vertical band convolution\n- An unchanged identity path\n\nEach of the three convolution branches gets $C/8$ channels; the remaining $5C/8$ pass straight through. The four groups are concatenated at the end. Splitting $k\\times k$ into $1\\times k$ and $k\\times 1$ is an old Inception-v3 trick, which explains the name.\n\n\n> [!TIP]\n> \n> Inception-v3 used this trick to break up large kernels. See:\n> \n> - **[[15.12] Inception-v3: Big Kernels Bow Out](/blog/papers/classic-cnns/inception-v3)**\n\n\nThe parameter count goes from $k^2C$ for a depthwise convolution to $(2k+9)C/8$: $2k$ comes from the two bands, and 9 from the 3×3 square. Its dependence on kernel size becomes linear. Let's write the PyTorch implementation supplied in the paper and count the parameters:\n\n```python\nimport torch\nimport torch.nn as nn\n\nclass InceptionDWConv2d(nn.Module):\n    def __init__(self, c, square=3, band=11, ratio=1/8):\n        super().__init__()\n        g = int(c * ratio)                                    # Channels in each convolution branch\n        self.hw = nn.Conv2d(g, g, square, padding=square // 2, groups=g)\n        self.w = nn.Conv2d(g, g, (1, band), padding=(0, band // 2), groups=g)\n        self.h = nn.Conv2d(g, g, (band, 1), padding=(band // 2, 0), groups=g)\n        self.split = (g, g, g, c - 3 * g)                     # 3×3, 1×11, 11×1, identity\n\n    def forward(self, x):\n        x_hw, x_w, x_h, x_id = torch.split(x, self.split, dim=1)\n        return torch.cat((self.hw(x_hw), self.w(x_w), self.h(x_h), x_id), dim=1)\n\nc = 96                                                        # Channels in InceptionNeXt-T's first stage\nparams = lambda m: sum(p.numel() for p in m.parameters())\nprint(\"7×7 depthwise:\", params(nn.Conv2d(c, c, 7, padding=3, groups=c)))\nprint(\"3×3 depthwise:\", params(nn.Conv2d(c, c, 3, padding=1, groups=c)))\nprint(\"Inception depthwise:\", params(InceptionDWConv2d(c)))\n```\n\n```text\n7×7 depthwise: 4800\n3×3 depthwise: 960\nInception depthwise: 408\n```\n\nInception depthwise convolution has fewer parameters than even a 3×3 depthwise convolution, because five eighths of the channels do no convolution at all.\n\n### A Cross-Shaped Receptive Field\n\nAt this point, it may seem like a poor trade. A 7×7 kernel lets every channel see a full 7×7 region. Now each channel sees only a small 3×3 square, a horizontal line, or a vertical line.\n\n[Someone asked on GitHub](https://github.com/sail-sg/inceptionnext/issues/3) whether this “decomposition” is mathematically equivalent to a 7×7 convolution. The author answered that it is not; “decompose” here is closer to “replace.” How much can the replacement actually see?\n\nThe MLP is the key. The 1×1 convolution after the depthwise convolution mixes channels. Features processed by a horizontal band in one block may enter a vertical-band branch in the next. Simplify each block to “spatial convolution followed by 1×1 convolution,” then backpropagate from the center output to find the input positions it can see:\n\n```python\ndef seen(blocks, make_mixer, size=33):\n    torch.manual_seed(0)\n    layers = []\n    for _ in range(blocks):                                   # Each block: spatial convolution, then 1×1 channel mixing\n        layers += [make_mixer(), nn.Conv2d(c, c, 1)]\n    net = nn.Sequential(*layers)\n    x = torch.randn(1, c, size, size, requires_grad=True)\n    net(x)[0, :, size // 2, size // 2].sum().backward()       # Find input positions visible to the center output\n    grid = x.grad[0].abs().sum(0) > 0\n    rows, cols = grid.nonzero().T\n    return f\"sees {int(grid.sum())} positions, spanning {int(rows.max() - rows.min() + 1)}×{int(cols.max() - cols.min() + 1)}\"\n\nfor blocks in (1, 2):\n    print(f\"Stack {blocks} block(s)\")\n    print(\"  7×7 depthwise:\", seen(blocks, lambda: nn.Conv2d(c, c, 7, padding=3, groups=c)))\n    print(\"  Inception depthwise:\", seen(blocks, lambda: InceptionDWConv2d(c)))\n```\n\n```text\nStack 1 block(s)\n  7×7 depthwise: sees 49 positions, spanning 7×7\n  Inception depthwise: sees 25 positions, spanning 11×11\nStack 2 block(s)\n  7×7 depthwise: sees 169 positions, spanning 13×13\n  Inception depthwise: sees 149 positions, spanning 21×21\n```\n\nWith one block, Inception depthwise sees a cross: a 3×3 center plus two lines of length 11, totaling 25 positions. In the second block, horizontal lines meet vertical lines to fill an 11×11 square, with four arms extending outward, totaling 149 positions. Two 7×7 layers cover 169 positions in a 13×13 square: a similar count, across a smaller span.\n\n### Building the Network\n\nThe number of blocks and channels per stage follows ConvNeXt, as does the four-stage structure. There are two differences. Normalization uses BatchNorm because speed is a priority here and BN can be merged into a convolution at inference. The final stage's MLP expansion ratio drops from 4 to 3, shifting the saved parameters into the classifier head. The smallest Atto variant increases each branch's fraction to 1/4 and uses bands of length 9.\n\n## Discussion\n\n### ImageNet-1K\n\nTraining follows the DeiT and ConvNeXt recipes for 300 epochs. Throughput is measured on an A100 with TF32:\n\n| Model | Parameters | MACs | Training (images/s) | Inference (images/s) | top-1 |\n| --- | ---: | ---: | ---: | ---: | ---: |\n| RSB-ResNet-50 | 26M | 4.1G | 969 | 3149 | 79.8% |\n| ConvNeXt-T | 29M | 4.5G | 575 | 2413 | 82.1% |\n| InceptionNeXt-T | 28M | 4.2G | **901** | **2900** | **82.3%** |\n| ConvNeXt-S | 50M | 8.7G | 361 | 1535 | 83.1% |\n| InceptionNeXt-S | 49M | 8.4G | **521** | **1750** | **83.5%** |\n| ConvNeXt-B | 89M | 15.4G | 267 | 1122 | 83.8% |\n| InceptionNeXt-B | 87M | 14.9G | **375** | **1244** | **84.0%** |\n\n*Table: selected rows from the paper's Table 4. RSB-ResNet-50 is ResNet-50 retrained with the newer “ResNet strikes back” recipe.*\n\nInceptionNeXt-T gains 0.2 percentage points over ConvNeXt-T, with 57% faster training and 20% faster inference. Against ResNet-50, it reaches 93% of the training throughput and 92% of the inference throughput. Calling that “ResNet-50 speed” is a little optimistic, though it is already close.\n\nThe speedup shrinks as models get larger: training is 57% faster for T, 44% for S, and 40% for B. Depthwise convolution scales linearly with channel count, while the MLP scales quadratically. Wider models spend more time in the MLP, leaving a smaller fraction to save by changing only the token mixer.\n\nThe smallest Atto variant goes the other way: training becomes 219% faster, but accuracy falls 0.4 percentage points below ConvNeXt-A.\n\nSpeed also depends on the hardware. In the same table, measured with FP32 on a 2080Ti, T's inference is 39% faster, more than the 20% on an A100. The later RDNet paper independently measures inference latency at batch size 128: 132 ms for InceptionNeXt-T and 150 ms for ConvNeXt-T, equivalent to a 14% speedup.\n\n### MetaNeXt Cannot Accommodate Attention\n\nSince MetaNeXt simplifies MetaFormer, the authors also try putting self-attention into the single-residual block, producing MetaNeXt-Attn. It fails to train properly, reaching just 3.9% ImageNet accuracy, against 79.8% for a similarly sized DeiT-S.\n\nMerging two residual connections into one saves time, but limits the complexity of the token mixer. This framework accommodates convolution; attention fails in this experiment.\n\n### Semantic Segmentation\n\nWith UperNet on ADE20K, trained for 160K iterations:\n\n| Size | Swin | ConvNeXt | InceptionNeXt |\n| --- | ---: | ---: | ---: |\n| T | 45.8 | 46.7 | **47.9** |\n| S | 49.5 | 49.6 | **50.0** |\n| B | 49.7 | 49.9 | **50.6** |\n\n*Table: from the paper's Table 6; values are mIoU.*\n\nThe T gap is 1.2, much larger than the 0.2 in classification. The test protocol matters, though. [A GitHub user reported](https://github.com/sail-sg/inceptionnext/issues/10) that plugging InceptionNeXt into their segmentation model gave results close to ConvNeXt. The author replied that the table follows ConvNeXt in reporting multi-scale test mIoU; with single-scale testing, InceptionNeXt and ConvNeXt are close. The segmentation training code was also not released, and several users ask for it in the issue.\n\n### Ablation\n\nStarting from InceptionNeXt-T, change one thing at a time:\n\n| Change | Training (images/s) | Inference (images/s) | top-1 |\n| --- | ---: | ---: | ---: |\n| Baseline | 901 | 2900 | 82.3% |\n| Remove horizontal band | 947 | 3093 | 81.9% |\n| Remove vertical band | 954 | 3173 | 81.9% |\n| Remove 3×3 square | 940 | 3004 | 82.0% |\n| Put the two bands in sequence | 903 | 2971 | 82.1% |\n| Increase band length from 11 to 13 | 896 | 2895 | 82.0% |\n| Increase branch fraction from 1/8 to 1/4 | 834 | 2499 | 82.2% |\n| Decrease branch fraction from 1/8 to 1/16 | 936 | 3097 | 81.8% |\n| Replace BatchNorm with LayerNorm | 721 | 2646 | 82.4% |\n\n*Table: selected rows from the paper's Table 8.*\n\nThe two bands matter most. Removing either costs 0.4 points; they sustain the receptive field. Removing the 3×3 square costs just 0.3, so the authors suggest a square-free variant if speed matters more. Raising the branch fraction to 1/4 brings no gain and only slows things down; lowering it to 1/16 costs 0.5 points.\n\nLayerNorm adds 0.1 percentage points while cutting training throughput by 20%, so the authors choose BatchNorm.\n\n## Conclusion\n\nInceptionNeXt replaces a 7×7 depthwise convolution with a small square, two bands, and a large group of idle channels. It preserves ConvNeXt-T's accuracy and brings training speed back close to ResNet-50.\n\nIt uses only PyTorch's built-in depthwise convolutions, with no custom operators, and timm provides pretrained weights such as `inception_next_tiny`. To speed up a ConvNeXt-style model, swapping out the token mixer is an easy experiment.\n\nAs for those 1.2 points in the segmentation table, ask whether the test is single-scale or multi-scale before deciding to change backbones.\n","imageSizes":{"/content/papers/assets/1f6716ae5b81879c492b1f16031d8c3a3e65f652d97318b88209b2b60ec9d7c4.png":{"width":924,"height":666},"/content/papers/assets/1c7bbb1c754725c6cd60caf24d092587cc19bbcf03b619f64821ad4a7fb93429.png":{"width":1446,"height":786}}}