ImageNet Classification with Deep Convolutional Neural Networks
- Publication
- NeurIPS (NIPS) 2012Conference
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@inproceedings{classiccnnsalexnet2012,
title = {{ImageNet Classification with Deep Convolutional Neural Networks}},
author = {Alex Krizhevsky and Ilya Sutskever and Geoffrey E. Hinton},
year = {2012},
booktitle = {NeurIPS (NIPS)},
url = {https://proceedings.neurips.cc/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html}
}
The study of neural networks began long before AlexNet made its mark.
AlexNet was an important milestone, with remarkable results in the 2012 ImageNet challenge.
Defining the Problem
By this time, the application of convolutional networks was beginning to show promise.
Before AlexNet, LeNet-5 successfully utilized Convolutional Neural Networks (CNNs) to tackle the problem of handwritten digit recognition.
Additionally, this architecture had already been applied to small-scale datasets such as CIFAR-10 and MNIST.
Compared to traditional feedforward neural networks (fully connected neural networks), CNNs benefited from their inductive bias, providing powerful learning capabilities for image data. Their structure effectively captured spatial dependencies and statistical regularities in images, and required far fewer parameters, making them easier to train.
Despite the attractive features of CNNs, large-scale applications on high-resolution images were still very expensive.
As luck would have it, NVIDIA was there too. Two GTX 580 GPUs with 3 GB of memory each, combined with optimized 2D convolutions, provided enough compute to train this large CNN.
Finally, the issue of large-scale datasets was addressed.
The large-scale image dataset ImageNet, introduced in 2009, provided enough labeled examples to effectively train these models while augmentation and regularization remained necessary to control overfitting.
*
With algorithms, data, and computational power in place, the next question was: how to design a deeper and larger CNN to improve image recognition performance?
Solving the Problem
Model Architecture
AlexNet utilized a deep convolutional neural network with 5 convolutional layers and 3 fully connected layers.
This architecture influenced CNN research for years afterward; many later designs developed from this foundation.
ReLU Activation Function
AlexNet uses ReLU instead of the then-common sigmoid or tanh. In the paper's Figure 1, a different four-layer CNN reached 25% CIFAR-10 training error about six times faster with ReLU than with tanh. That result is specific to this training comparison, not a measured sixfold speedup of AlexNet itself. ReLU avoids saturation on its positive side but does not eliminate every gradient problem.
The formula is: .
Meaning: when the input is greater than 0, the output equals the input; when the input is less than 0, the output is 0.
Multi-GPU Training
Why does the AlexNet diagram look cut in half?
That is how the original paper draws it. The second, fourth and fifth convolutions connect only to features on the same GPU; the third receives features from both GPUs, and fully connected layers see all units in the preceding layer. These are not two independent CNNs. Relative to a narrower one-GPU comparison model, the two-GPU network reduced top-1 and top-5 error by 1.7 and 1.2 percentage points, respectively; the two networks did not have exactly equal capacity.
Local Response Normalization
Before Batch Normalization, the authors used LRN to improve generalization. ReLU itself does not require normalization to avoid saturation on its positive branch.
The basic idea of LRN is to normalize each neuron's activation value by considering the activations in neighboring channels at the same spatial position, thereby suppressing overly active neurons and promoting competition among neurons.
This mechanism, similar to lateral inhibition in biological neurons, offers an analogy for competition across channels, rather than a guarantee that every neuron will remain active.
In AlexNet, LRN is defined as follows:
Given a neuron's activation value at position , obtained by applying the -th convolutional kernel and ReLU nonlinearity, the normalized activation value is calculated as:
The parameters are:
- : activation value at position after applying the -th convolutional kernel and ReLU.
- : normalized activation value.
- , , , and : hyperparameters, adjusted via validation set:
- : total number of convolutional kernels in that layer.
- : index of neighboring convolutional kernels in the range .
Overlapping Max Pooling
In AlexNet, the max-pooling layer has a size of 3x3 and a stride of 2.
This operation reduces the feature map width and height by roughly half, lowering computation.
Compared with non-overlapping 2×2 pooling at stride 2 and the same output size, 3×3 overlapping pooling reduced top-1 and top-5 error by 0.4 and 0.3 percentage points. The authors also observed slightly less overfitting during training.
Dropout
During training, the authors noticed that the model tended to overfit. To address this, they introduced Dropout.
Dropout randomly drops (sets to zero) some neurons during each training iteration.
This means the network computes with a randomly selected subset of neurons during each forward pass.
Dropped neurons do not contribute to forward propagation and are not updated during backpropagation.
This prevents the network from relying on specific neurons, reducing complex co-adaptation between neurons.
Model Implementation
Let's take a detailed look at the network composition:
- Input 224x224 image.
- First convolutional layer with 11x11 kernel size, stride 4, 96 feature maps, output size 55x55.
- Followed by max-pooling.
- Second convolutional layer with 5x5 kernel size, 256 feature maps, output size 27x27; it reads only the preceding features on the same GPU.
- Another max-pooling layer.
- Third convolutional layer with 3x3 kernel size, 384 feature maps, output size 13x13.
- Fourth convolutional layer with 3x3 kernel size, 384 feature maps, output size 13x13; it reads only the preceding features on the same GPU.
- Fifth convolutional layer with 3x3 kernel size, 256 feature maps, output size 13x13, with the same restricted connections.
- Finally, three fully connected layers: first with 4096 neurons, second with 4096 neurons, and the last with 1000 neurons corresponding to the 1000 classes in ImageNet.
Key points from the paper:
- The five convolutional layers and first two fully connected layers use ReLU. The final layer produces logits; softmax converts them to class probabilities.
- Local Response Normalization is applied after the first and second convolutional layers.
- Max-pooling is applied after LRN and the fifth convolutional layer, with a size of 3x3 and stride of 2.
- Dropout is applied to the hidden activations of the first two fully connected layers with a dropout rate of 0.5.
Now that the overall structure is clear, we can write a single-device model sketch. groups=2 represents the paper's restricted connections. For a 224×224 input, the first convolution needs padding=2 to produce 55×55; the final pooling output is 6×6.
PyTorch LRN multiplies the sum of squares by alpha / size, so alpha=0.0005 here matches in the formula above. The model returns logits for a loss such as CrossEntropyLoss. This is an architectural example, not the paper’s complete augmentation, initialization, and training procedure:
import torch.nn as nn
class AlexNet(nn.Module):
def __init__(self, num_classes=1000):
super(AlexNet, self).__init__()
self.stage1 = nn.Sequential(
nn.Conv2d(3, 96, kernel_size=11, stride=4, padding=2),
nn.ReLU(),
nn.LocalResponseNorm(size=5, alpha=0.0005, beta=0.75, k=2),
nn.MaxPool2d(kernel_size=3, stride=2)
)
self.stage2 = nn.Sequential(
nn.Conv2d(96, 256, kernel_size=5, padding=2, groups=2),
nn.ReLU(),
nn.LocalResponseNorm(size=5, alpha=0.0005, beta=0.75, k=2),
nn.MaxPool2d(kernel_size=3, stride=2)
)
self.stage3 = nn.Sequential(
nn.Conv2d(256, 384, kernel_size=3, padding=1),
nn.ReLU()
)
self.stage4 = nn.Sequential(
nn.Conv2d(384, 384, kernel_size=3, padding=1, groups=2),
nn.ReLU()
)
self.stage5 = nn.Sequential(
nn.Conv2d(384, 256, kernel_size=3, padding=1, groups=2),
nn.ReLU(),
nn.MaxPool2d(kernel_size=3, stride=2)
)
self.classifier = nn.Sequential(
nn.Flatten(),
nn.Linear(256 * 6 * 6, 4096),
nn.ReLU(),
nn.Dropout(0.5),
nn.Linear(4096, 4096),
nn.ReLU(),
nn.Dropout(0.5),
nn.Linear(4096, num_classes),
)
def forward(self, x):
x = self.stage1(x)
x = self.stage2(x)
x = self.stage3(x)
x = self.stage4(x)
x = self.stage5(x)
x = self.classifier(x)
return x
Discussion
What Did the Convolutional Network Learn?
The authors visualized all 96 first-layer kernels. They learned features such as edges, color and texture. This image alone does not show which objects the later layers recognize.
Previously, such features were manually designed and fed to machine learning algorithms using techniques like SIFT, HOG, and Sobel operators.
After AlexNet, the question was less about which features to design by hand and more about how to design a network that learns better features for itself.
ILSVRC-2010 Results
| Model | Top-1 | Top-5 |
|---|---|---|
| Sparse coding [2] | 47.1% | 28.2% |
| SIFT + FVs [24] | 45.7% | 25.7% |
| CNN | 37.5% | 17.0% |
On the ILSVRC-2010 dataset, the authors reported these test errors in their 2012 paper; this was not an entry in the 2010 competition:
- Top-1 error rate: 37.5%
- Top-5 error rate: 17.0%
Compared to the best methods at the time:
- Sparse coding: Top-1 error rate of 47.1%, Top-5 error rate of 28.2%.
- SIFT + Fisher Vectors (FVs): Top-1 error rate of 45.7%, Top-5 error rate of 25.7%.
This shows that AlexNet significantly outperformed other methods in ILSVRC-2010.
ILSVRC-2012 Results
| Model | Top-1 (val) | Top-5 (val) | Top-5 (test) |
|---|---|---|---|
| SIFT + FVs [7] | — | — | 26.2% |
| 1 CNN | 40.7% | 18.2% | — |
| 5 CNNs | 38.1% | 16.4% | 16.4% |
| 1 CNN* | 39.0% | 16.6% | — |
| 7 CNNs* | 36.7% | 15.4% | 15.3% |
- * indicates the use of additional ImageNet 2011 Fall pretraining; only the two added models in the 7 CNNs* ensemble use it.
In the ILSVRC-2012 competition, due to the unavailability of test set labels, the test error rates of all models could not be fully reported, but here are some key results:
- A single CNN model achieved a Top-5 error rate of 18.2% on the validation set.
- Averaging the predictions of five similar CNN models reduced the Top-5 error rate to 16.4%.
- A CNN model with a sixth convolutional layer, pre-trained on the full ImageNet 2011 Fall release (15 million images and 22,000 classes), and fine-tuned on ILSVRC-2012, achieved a validation Top-5 error rate of 16.6%.
- Averaging the predictions of two pre-trained CNN models with the previous five CNN models further reduced the test Top-5 error rate to 15.3%.
The second-place method, which averaged the predictions of multiple classifiers computed on different types of dense sampling features, achieved a Top-5 error rate of 26.2%.
Conclusion
AlexNet achieved remarkable results in ILSVRC-2012, marking the rise of deep learning in computer vision. The 15.3% top-5 test error came from an ensemble of seven models, two of which had extra ImageNet 2011 Fall pretraining; the single model had 18.2% top-5 validation error.
Its network design and training method had a lasting influence on deep learning research.