Densely Connected Convolutional Networks
- Publication
- CVPR 2017Conference
- First submitted to arXiv
- arXiv:1608.06993
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{classiccnnsdensenet2016,
title = {{Densely Connected Convolutional Networks}},
author = {Gao Huang and Zhuang Liu and Laurens van der Maaten and Kilian Q. Weinberger},
year = {2016},
eprint = {1608.06993},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/1608.06993}
}
As CNNs have become deeper, gradients propagated back to earlier layers can become very small, making training difficult.
Various studies began exploring how to improve the flow of information through deep networks, one of the most significant being ResNet.
Defining the Problem
ResNet uses additive identity shortcuts to make information preservation explicit, yet many layers may contribute little. Stochastic depth explored this observation:
Besides ResNet, another famous method is Highway Networks:
This paper doesn’t aim to refute the above research but proposes a new connection method to make information flow more efficiently through the network.
Solving the Problem
Network Architecture
This diagram is very intuitive but can be misleading:
This is not the entire network architecture; it describes the structure of a Dense Block.
In this structure, each layer’s input concatenates the block input and all preceding outputs within that block along the channel dimension, and there is no downsampling.
For a block with layers, this pattern creates direct connections. Unlike ResNet's elementwise addition, concatenation preserves earlier maps as inputs to later layers; transition layers connect blocks at different spatial sizes.
The authors want earlier information and gradients to have shorter paths, so they are less likely to be diluted along the way:
- If information can fade as it passes through the network, why not connect every layer?
Transition Layers and Average Pooling
As shown above.
In Convolutional Neural Networks (CNNs), downsampling layers (such as pooling layers) are essential for changing the feature map size.
To effectively downsample, the authors divided the network into multiple densely connected Dense Blocks.
Between each Dense Block, there are transition layers that perform convolution and pooling operations.
The entire transition layer consists of Batch Normalization, ReLU, and a 1×1 convolutional layer, followed by a 2×2 average pooling layer.
Growth Rate
An important feature of DenseNet is the growth rate.
In DenseNet, if each function produces k feature maps, then the -th layer has input feature maps, where is the number of channels in the input layer.
A significant difference between DenseNet and other network architectures is that it can have very narrow layers, such as k = 12. This hyperparameter k is known as the network’s growth rate.
Research shows that relatively small growth rates can achieve state-of-the-art results on the tested datasets. This is because each layer can access all previous feature maps within its block, leveraging the network's "collective knowledge." The growth rate controls the amount of new information each layer contributes to the global state, which can be accessed directly by subsequent layers in the same dense block without duplicating it layer by layer.
Bottleneck Layers
Although each layer only produces k output feature maps, it typically has more inputs.
Research suggests introducing a 1×1 convolution before each 3×3 convolution as a bottleneck layer to reduce the number of input feature maps, thereby improving computational efficiency.
This design is particularly effective for DenseNet, leading the authors to refer to such a network as DenseNet-B.
The bottleneck layer structure is: BN-ReLU-Conv(1×1)-BN-ReLU-Conv(3×3). In experiments, each 1×1 convolution produces 4k feature maps.
Compression
To further enhance model compactness, the authors designed a method to reduce the number of feature maps in the transition layers.
If a Dense Block contains m feature maps, the subsequent transition layer will produce output feature maps, where is called the compression factor. When , the number of feature maps in the transition layer remains unchanged.
The authors referred to DenseNets with as DenseNet-C and set in their experiments. When using both bottleneck layers and transition layers with , the model is called DenseNet-BC.
Implementation Details
For all datasets except ImageNet, the authors experimented with DenseNets having three Dense Blocks.
Each Dense Block contained the same number of layers. Before entering the first Dense Block, the input image underwent a convolution with 16 (or twice the growth rate for DenseNet-BC) output channels. For convolutions with kernel size 3×3, each side of the input was zero-padded to maintain the feature map size.
Between two consecutive Dense Blocks, the authors used a 1×1 convolution followed by a 2×2 average pooling as the transition layer. At the end of the last Dense Block, global average pooling was performed, followed by a softmax classifier. The feature map sizes in the three Dense Blocks were 32×32, 16×16, and 8×8, respectively.
The authors used the following basic DenseNet configurations in their experiments:
- L = 40, k = 12
- L = 100, k = 12
- L = 100, k = 24
For DenseNet-BC, the configurations were:
- L = 100, k = 12
- L = 250, k = 24
- L = 190, k = 40
In experiments on the ImageNet dataset, the authors used a DenseNet-BC structure with four Dense Blocks, experimenting with 224×224 input images.
The initial convolution layer had 2k convolutions of size 7×7 with a stride of 2.
The number of feature maps in all other layers also depended on the setting of k.
Discussion
Experimental Results
The above image plots the single-crop top-1 validation errors of DenseNets and ResNets as functions of the number of parameters and FLOPs, showing the differences in performance and computational resource usage.
In the left plot, we see the relationship between validation error and the number of parameters for DenseNet and ResNet. DenseNet-201, with around 20M parameters, has a validation error similar to ResNet-101, which has over 40M parameters. This indicates that DenseNet can achieve similar performance to ResNet with fewer parameters.
The right plot relates validation error to FLOPs. The authors note that a DenseNet near ResNet-50's compute reaches an error comparable to ResNet-101's, which uses about twice the compute. This comparison is specific to the plotted models and evaluation setup; it is not a measured latency ratio on every device.
The unpictured Table 3 separates crop protocols: DenseNet-201 has 22.58% single-crop / 21.46% ten-crop top-1 validation error, and DenseNet-264 has 22.15% / 20.80%. Figure 3 plots only the single-crop results.
Compact Models
Within a dense block, a feature map learned by one layer can be accessed directly by all subsequent layers, a direct result of input concatenation.
This encourages feature reuse throughout the network, leading to a more compact model.
These two panels compare DenseNet variants at left and pre-activation ResNets in the middle panel of the original figure. The authors train small networks of different depths on C10+; the plotted y-axis is test error, where lower is better. DenseNet-BC is the most parameter-efficient of the shown DenseNet variants.
Moreover, DenseNet-BC requires only about one-third the parameters of ResNet to achieve the same accuracy.
A DenseNet-BC with only 0.8M trainable parameters can achieve accuracy comparable to a 1001-layer ResNet (10.2M parameters).
Feature Reuse
Within a single Dense Block, later layers directly concatenate all earlier feature maps. Between blocks of different spatial sizes, information passes through a transition layer; those distant layers are not directly connected.
The authors conducted an experiment to investigate whether the trained network utilized this feature. They trained a DenseNet (L=40, k=12) and computed the average absolute weights connecting each convolutional layer to earlier source layers, normalized by the number of input feature maps.
The three heatmaps put the target layer on the horizontal axis and the source layer on the vertical axis. The color bar goes from lower blue to higher red. It measures mean absolute weights normalized by the number of input feature maps, not information flow or each path's gradient magnitude.
Several features can be observed from the image:
- All layers distribute their weights across multiple inputs within the same block, indicating that features extracted by earlier layers are directly used by deeper layers.
- Transition layers distribute their weights across all layers in the previous Dense Block, with information flowing through a few indirect paths from the first layer to the last.
- Layers within the second and third Dense Blocks consistently assign minimal weights to the outputs of transition layers (the top row of the triangle), suggesting that transition layers output many redundant features.
- The final classifier uses features from across the block, with weights concentrated more heavily on later maps; the authors suggest that higher-level features emerge late in the network.
Conclusion
DenseNet brings together the benefits of identity mappings, implicit deep supervision and varied depths. By reusing features throughout the network, it learns more compact and accurate models in the reported experiments. Its compact internal representation and reduced feature redundancy suggest it may be a useful feature extractor for other computer vision tasks based on convolutional features. “Implicit deep supervision” describes the short paths through the network, not an extra classifier attached to every layer.
Future work could explore DenseNet for feature transfer and extend its use in computer vision.