Very Deep Convolutional Networks for Large-Scale Image Recognition
- Publication
- ICLR 2015Conference
- First submitted to arXiv
- arXiv:1409.1556
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{classiccnnsvgg2014,
title = {{Very Deep Convolutional Networks for Large-Scale Image Recognition}},
author = {Karen Simonyan and Andrew Zisserman},
year = {2014},
eprint = {1409.1556},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/1409.1556}
}
In the early stages of neural networks, the exploration of depth was stagnant for a long time.
One important obstacle was the vanishing gradient problem. As the network depth increased, the gradient information could not effectively propagate to the earlier layers, making training difficult.
Thus, after AlexNet, researchers began to explore the depth of networks and proposed a series of deep network structures, among which VGG is a significant representative.
Problem Definition
In previous studies, the size of convolutional kernels was typically 5×5 or 7×7. VGG proposed using multiple 3×3 convolutional kernels to replace the larger ones. The authors argued that using multiple layers of small convolutional kernels instead of a few layers of large kernels brings several advantages:
-
Improved Discriminative Power of Decision Functions:
Stacking multiple small convolutional kernels (e.g., 3×3) and introducing non-linear activation functions (such as ReLU) between each layer can enhance the model's non-linear representation capabilities. This makes the decision functions more discriminative, allowing them to capture complex features in the data more effectively compared to using a single large convolutional kernel (e.g., 7×7).
-
Reduction in Parameter Count:
Assuming every layer has C input and output channels, and ignoring biases, the number of parameters required for stacking three 3×3 convolutional layers is:
In contrast, a single 7×7 convolutional layer requires:
.
The three-layer stack therefore uses about 45% fewer weights: . The paper’s 81% describes the reverse comparison: is about 81% greater than .
-
Increased Theoretical Receptive Field:
With stride=1 and dilation=1, stacking small kernels expands the theoretical receptive field. The influence of individual input positions requires separate analysis.
For instance, stacking two 3×3 convolutional layers results in an theoretical receptive field of 5×5, while stacking three 3×3 convolutional layers results in a receptive field of 7×7.
Thus, the receptive field can be flexibly adjusted by increasing the number of convolutional layers without needing larger kernels.
-
Regularization Effect:
The authors view a stack of 3×3 layers as a structural constraint on a single 7×7 convolution: the large filter must be decomposed through smaller filters, with ReLU between them. This explains their design; the comparisons in the paper do not independently show that this choice reduces overfitting.
-
Introduction of More Non-linearity:
Although a 1×1 convolutional layer is essentially a linear projection, when combined with a non-linear activation function, it can increase the model's non-linearity without changing the convolutional receptive field.
The authors use these small filters as a shared design while increasing network depth step by step, then measure the effect on ImageNet classification. Training the deeper networks still requires the initialization and learning-rate choices described below.
Problem Solving
Network Architecture
The authors compare five main configurations, A through E, plus A-LRN. A and A-LRN test the effect of adding local response normalization (LRN) within this VGG configuration; B through E vary the number and types of convolutional layers.
During training, the input to the ConvNet was fixed-size 224 × 224 RGB images. The only preprocessing was to subtract the mean RGB value computed on the training set from each pixel.
The authors used very small receptive field filters for convolution: 3 × 3 (and 1 × 1 in some configurations). The convolution stride was fixed at 1 pixel, and spatial padding was performed to preserve spatial resolution after convolution.
Spatial pooling was carried out by five max-pooling layers, which were performed over a 2 × 2 pixel window with a stride of 2.
Following the convolutional layers were three fully connected (FC) layers: the first two with 4096 channels each, and the third with 1000 channels corresponding to the ImageNet classes. A softmax layer follows that third FC layer.
All hidden layers were equipped with ReLU activation functions. Except for one network, all other networks did not include local response normalization (LRN).
The table above summarizes the five ConvNet configurations (A-E), mainly differing in depth:
- Network A has 11 weight layers (8 convolutional layers and 3 FC layers).
- Network E has 19 weight layers (16 convolutional layers and 3 FC layers).
- The width of the layers starts at 64 and increases by a factor of 2 after each max-pooling layer until it reaches 512.
- The deeper networks do not have significantly more weights than the shallower ones with larger layers and receptive fields.
The use of 3×3 convolutional kernels instead of larger receptive fields is a notable characteristic of the VGG network.
Training Details
The training procedure largely followed the approach of Krizhevsky et al. (2012):
-
Training Method:
- Training was performed using mini-batch gradient descent with momentum. The batch size was set to 256, and the momentum was 0.9.
- L2 regularization was used (weight decay multiplier of ).
- Dropout with a rate of 0.5 was applied to the first two fully connected layers.
-
Learning Rate:
- The initial learning rate was . When validation accuracy stopped improving, it was reduced tenfold; this happened three times.
- Training was stopped after 370K iterations (74 epochs).
-
Weight Initialization:
Initialization is crucial for deep networks. The authors used the layers of configuration A to initialize the first four convolutional layers and the last three fully connected layers of the deeper architectures, with the middle layers randomly initialized. This approach helped avoid training stagnation due to gradient instability in deep networks.
-
Image Cropping and Augmentation:
To obtain fixed-size 224×224 ConvNet input images, random crops were taken from rescaled training images. Additionally, random horizontal flips and RGB color shifts were used to augment the training set.
Regarding training image sizes, the authors considered two methods for setting training scales:
- Fixed S: Training was performed at two fixed scales, S = 256 and S = 384. Training began with S = 256, and the weights from this pre-training were used to initialize the network at S = 384.
- Multi-scale Training: S was randomly sampled from the range [Smin, Smax], with Smin = 256 and Smax = 512. This approach allowed better handling of objects of varying sizes in images.
This paper's implementation was based on the C++ Caffe toolbox with significant modifications to support multi-GPU training and evaluation.
On a system equipped with four NVIDIA Titan Black GPUs, training a single network took approximately 2-3 weeks.
Discussion
LRN Contribution
At the same scale, A has 10.4% validation top-5 error and A-LRN has 10.5%. LRN, previously used in AlexNet, did not improve this comparison and increased memory use and computation, so the authors omitted it from B–E. This is evidence for the tested A configuration, not a claim that LRN is useless in every architecture.
At the same scale, the 11-layer A has 10.4% error and the 19-layer E has 9.0%. C and D both have 16 weight layers, yet score 9.4% and 8.8% respectively: beyond adding 1×1 non-linearity, the spatial reach of 3×3 filters matters. E reaches 8.0% after multi-scale training, so that figure cannot be attributed to depth alone. The authors observe saturation near 19 layers and suggest that larger datasets might still benefit from deeper models.
Performance on ImageNet
According to the table above, deep convolutional networks (ConvNets) significantly outperformed the best results of the previous generation models in the ILSVRC-2012 and ILSVRC-2013 competitions.
In this table, VGG's two-model combination with multi-crop and dense evaluation has 6.8% test top-5 error, close to the 6.67% official test result of the seven-model GoogLeNet submission. This later two-model setup differs from VGG's actual seven-model, dense-evaluation competition submission, which scored 7.3% on the test set. A single VGG scores 7.0% test top-5 error; the single GoogLeNet row reports 7.9% validation top-5 error and leaves its test column empty. The arithmetic gap is 0.9 percentage points, but the splits differ, so it is not a same-condition comparison. The two-model VGG result is still well below Clarifai's 11.7% without external training data in the earlier ILSVRC challenge.
Conclusion
VGG holds its small-filter design largely constant while comparing 11 to 19 weight layers. This makes the question of whether depth helps visible in both the configurations and their results. Stacked 3×3 filters provide a wider theoretical receptive field and more non-linearity; on the paper's ImageNet setup, the deeper D and E configurations obtain lower classification error.
In Appendix B, the authors also use fixed ImageNet-pretrained VGG features with a linear SVM on other datasets. Those results support transfer to the tested tasks.
The simple design and systematic comparison made VGG an important reference for later vision models. It is still worth revisiting.