VanillaNet: the Power of Minimalism in Deep Learning
- Publication
- NeurIPS 2023Conference
- First submitted to arXiv
- arXiv:2305.12972
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{reparameterizationvanillanet2023,
title = {{VanillaNet: the Power of Minimalism in Deep Learning}},
author = {Hanting Chen and Yunhe Wang and Jianyuan Guo and Dacheng Tao},
year = {2023},
eprint = {2305.12972},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2305.12972}
}
Even after a decade of deep convolutional networks dominating the field, it's refreshing to see such a minimalist approach like VanillaNet emerge.
Problem Definition
In recent years, the field of computer vision has advanced rapidly. From the early days of AlexNet to the latest CNN and Transformer hybrid architectures, there's been a relentless push towards more complex models.
To climb the ImageNet leaderboard, researchers have continuously increased model complexity, leading to a corresponding rise in computational costs and resource demands. Moreover, deploying these complex architectures has become increasingly challenging. For instance, ResNet consumes a significant amount of additional memory during operation, while the shifted window mechanism in Swin Transformer requires complex engineering solutions, including rewriting CUDA code.
Why Have Non-Residual Convolutional Architectures Fallen Out of Favor?
Take AlexNet and VGG, for example.
Greater depth can make optimization harder; ResNet’s residual connections help address degradation. VanillaNet revisits that design space.
As a result, simple network designs seemed to have hit a dead end, receiving little attention.
Modern GPUs support substantial parallelism, but latency also depends on memory traffic, operator-launch overhead and sequential depth. FLOPs and parameter count alone cannot predict it; the bottleneck varies with hardware, batch size and implementation.
This realization brought about the insight that the ultimate bottleneck was destined from the start.
Solution
Model Architecture
The authors of this paper propose a simple neural network architecture called VanillaNet, as shown in the figure above.
Looking at this architecture feels like a blast from the past:
- AlexNet! What are you doing here?
The figure shows VanillaNet-6, with one main convolutional block per stage. Deeper variants add blocks to selected stages.
In the Stem stage, the authors use a convolutional layer with a stride of 4 to map the image into a -dimensional feature space.
In stages 1–3, 1×1 convolutions expand channels while stride-2 MaxPool reduces spatial size; pooling does not double the channel count. Stage 4 keeps the channel count, followed by global average pooling and classification.
The main channel-mixing convolutions are 1×1, but the stem uses 4×4 kernels and the activation module below includes larger depthwise kernels. The code implements classification as a 1×1 convolution after global average pooling, equivalent to a fully connected layer. Training-time BatchNorm can be fused for deployment.
There are no residual connections, attention mechanisms, or shifted windows here—just basic convolution and pooling.
Does This Really Work?
How could such a straightforward architecture possibly perform well?
The authors identify limited nonlinear expressiveness as a bottleneck; the baseline does not lack nonlinear layers altogether.
We know that the strength of neural networks lies in their non-linearity, which is key to fitting complex functions. How can we enhance the non-linear capacity of this network? We'll explore this later, but first, let's look at another issue: depth.
Depth Training Strategy
In convolutional neural networks, the combination of convolutional layers and activation functions is crucial for capturing the non-linear features of data.
Early in training, each main convolution is replaced by two convolutions with an activation between them. Only that intermediate activation gradually becomes an identity. The series-informed activation after the block remains, so the final network is not purely linear.
The entire transformation of the activation function can be described by the following formula:
where is a hyperparameter that changes with training iterations. At the start of training, , meaning , with the activation function at full strength. As training progresses, gradually increases until, by the end of training, , meaning , and the activation function becomes an identity mapping.
At the end of training, since the activation function has become an identity mapping, the two convolutional layers can be merged into a single layer, reducing inference time and improving model efficiency. The merging process is akin to reparameterization, familiar from models like RepVGG:
-
BatchNorm Merging:
First fix BN running statistics, then fuse BN with its preceding convolution; is the running variance and the numerical-stability term.
Let the weight and bias matrices of the convolutional layer be and , and the parameters of batch normalization (scaling, shifting, mean, and variance) be .
The merged weight and bias matrices are:
where the subscript represents the value of the th output channel.
-
Merging Two Convolutional Layers:
After merging the batch normalization layers, we proceed to merge the two convolutional layers.
Let and be the input and output features, respectively. The convolution can be expressed as:
where denotes the convolution operation, denotes matrix multiplication, and is generated by the im2col operation, transforming the input into a matrix corresponding to the convolution kernel shape. For a convolution, the im2col operation simplifies to a reshaping operation, so the weight matrices and of the two convolutional layers can be merged into one matrix:
The order here is followed by , with biases broadcast over spatial positions. Without an intermediate nonlinearity, the two layers become one affine convolution and require fewer inference operations.
This depth training strategy offers several advantages:
- Enhanced Non-linearity: Using strong non-linear activation functions in the early training stages helps the model capture complex patterns in the data, improving performance.
- Gradual Transition to a Simplified Model: As training progresses, the activation functions gradually become identity mappings, simplifying the final model structure and improving inference speed.
- Easy Layer Merging: At the end of training, the two convolutional layers can be easily merged into one, reducing computational costs and improving efficiency.
Series-Informed Activation Functions
The authors motivate the design through the expressive capacity of shallow networks. Limited nonlinearity is their proposed bottleneck for this architecture, not a theorem that all shallow networks cannot represent complex functions.
In fact, there are two ways to enhance the non-linearity of neural networks:
- Stacking Non-linear Activation Layers
- Increasing the Non-linearity of Each Activation Layer
Current trends in network design favor the former approach, but this leads to high latency when parallel computing capacity is abundant.
A direct way to enhance the non-linearity of an activation layer is to stack it. The "serial" stacking of activation functions is the core idea of deep networks.
In contrast, the authors of this paper chose to "parallelize" the stacking of activation functions.
Let a single activation function in a neural network be , which could be a common function like ReLU or Tanh.
Parallel stacking of activation functions can be expressed as:
where represents the number of stacked activation functions, and and are the scaling and bias for each activation function. Parallel stacking significantly enhances the non-linearity of the activation function, and the above equation can be seen as a series, essentially a sum of several operations.
The authors then sum weighted activations from neighboring spatial locations, so the activation module also aggregates spatial information. This is local neighborhood aggregation, not necessarily global image context in a single layer, and is related to BNET.
Given an input feature , where , , and are the height, width, and channels, respectively, the activation function is expressed as:
where , , and .
At spatial radius , the expression has no neighborhood mixing and becomes a pointwise scaled/shifted activation. It equals exactly only if and . Here is a spatial radius, unlike the number of terms in the preceding scalar illustration. The authors use ReLU as the basic activation.
Discussion
Earlier, we mentioned that the limited capabilities of shallow networks are mainly due to "insufficient non-linearity." With the introduction of parallel stacking of activation functions, can this design improve the model's performance?
Ablation Study - Activation Function
The table above shows the performance of VanillaNet with different values of .
At , with no neighborhood aggregation, this configuration reaches 60.53% Top-1 accuracy in the reported ImageNet experiment; the paper treats it as the ordinary ReLU comparison.
When , the accuracy jumps to 74.53%, marking a significant improvement. When , the accuracy and inference speed reach an optimal balance.
Ablation Study - Depth Training Strategy
Given the extremely shallow structure of VanillaNet, the authors propose increasing non-linearity during training to enhance performance.
The table above analyzes the proposed depth training technique. VanillaNet-6 with the series-informed activation but without depth training reaches 75.23% Top-1 accuracy. Adding depth training raises it to 76.36%, showing a benefit for this shallow network.
Additionally, the authors further applied the depth training and series-informed activation functions to other networks to demonstrate the generalization ability of these two techniques.
The table also compares AlexNet and ResNet-50 on ImageNet. AlexNet’s baseline is 57.52%; the paper calls it a 12-layer structure, using a depth-counting convention that includes operations beyond the usual eight learned layers.
By applying the proposed depth training and series-informed activation functions, AlexNet's performance rose from 57.52% to 63.59%, a gain of 6.07 percentage points, demonstrating that these techniques are highly effective for shallow networks.
For the more complex ResNet-50, adding both techniques raises accuracy from 76.13% to 76.27%, still below the 76.30% obtained by adding only the series-informed activation. The authors take this as a sign that the deeper network already has more nonlinearity; this comparison supports only a limited benefit from depth training in this setup.
Ablation Study - Residual Connections
Finally, how effective are the residual connections popular in general network structures?
In the VanillaNet-6 ablation, two parameter-free shortcut designs reduce accuracy from 76.36% to 75.92% and 75.72%.
The authors suggest that this shallow architecture needs stronger nonlinear expressiveness and may not benefit from an identity path. A residual connection does not delete the nonlinear branch, however; the accuracy loss remains an observation for the tested designs.
Performance on ImageNet
To demonstrate the effectiveness of the proposed methods, the authors conducted experiments on the ImageNet dataset.
ImageNet-1K contains about 1.28 million training images, 50,000 validation images and 1,000 classes. Original image dimensions vary; 224×224 is the main experiment’s preprocessed input size. VanillaNet is shallow but wide with many parameters, so the authors use strong regularization against overfitting.
Latency is measured on an NVIDIA A100 at batch size 1.
In this scenario, the study found that inference speed had little correlation with the number of FLOPs and parameters. For example, although MobileNetV3-Large has very low FLOPs (0.22B), its GPU latency is 7.83ms, even higher than that of VanillaNet-13, which has 11.9B FLOPs.
The study shows that under these settings, inference speed is highly related to the complexity and number of layers in the network. For instance, the inference speed difference between ShuffleNetV2x1.5 and ShuffleNetV2x2 is minimal (7.23ms and 7.84ms), despite a significant difference in FLOPs (about 0.3B vs. 0.6B; these figures are not parameter counts). This is because the difference lies only in the number of channels.
In the table, straightforward architectures such as VGGNet and VanillaNet show low latency. Some ResNet configurations, despite their residual branches, are also faster than many complex models. These are comparisons on an A100 at batch size 1.
Conclusion
Under the paper’s measured conditions, VanillaNet combines a simple architecture with competitive accuracy and latency against several contemporary, more complex networks, highlighting the potential of a minimalist approach to deep learning.
This work challenges established expectations for a basic model architecture and offers another direction for designing simple, effective networks.