Squeeze-and-Excitation Networks
- Publication
- CVPR 2018Conference
- First submitted to arXiv
- arXiv:1709.01507
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{lightweightsenet2017,
title = {{Squeeze-and-Excitation Networks}},
author = {Jie Hu and Li Shen and Samuel Albanie and Gang Sun and Enhua Wu},
year = {2017},
eprint = {1709.01507},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/1709.01507}
}
In an era where everyone is striving to top the ImageNet leaderboard, various network architectures emerge endlessly. The SENet architecture was proposed to enhance the global inter-channel relationships within a network, thereby improving its performance.
Defining the Problem
Past research observations from VGG and ResNet indicate that increasing depth can improve model quality, and residual connections help the network learn better features. Another line of work uses grouped and multi-branch convolutions, as in Inception. This is a relationship between design ideas, not a claim that all grouped-convolution research preceded Inception.
These studies reflect an assumption:
- Channel relationships are usually modeled implicitly by local-receptive-field functions whose learned parameters do not change with each input instance.
However, the authors believe this assumption is problematic:
- Can information from the whole feature map dynamically adjust channels for each input?
Stacked CNNs can already have global receptive fields. SE explicitly generates input-dependent channel weights.
Solving the Problem
The authors aim to introduce global information into the network channels without significantly increasing computational load and parameters. Thus, they proposed the Squeeze-and-Excitation (SE) module to introduce global information with minimal additional computational cost.
Model Architecture
This is another architecture that is easier to understand through code rather than text.
Let’s implement a simplified version. It assumes positive integer in_channels and reduction_ratio, with in_channels >= reduction_ratio. The C//16 comment refers to the default ratio:
import torch
import torch.nn as nn
class SEBlock(nn.Module):
def __init__(self, in_channels, reduction_ratio=16):
super(SEBlock, self).__init__()
self.squeeze = nn.AdaptiveAvgPool2d(1)
self.excitation = nn.Sequential(
nn.Conv2d(in_channels, in_channels // reduction_ratio, kernel_size=1, stride=1, padding=0, bias=False),
nn.ReLU(),
nn.Conv2d(in_channels // reduction_ratio, in_channels, kernel_size=1, stride=1, padding=0, bias=False),
nn.Sigmoid()
)
def forward(self, x: torch.Tensor) -> torch.Tensor:
# x: (B, C, H, W)
# (B, C, H, W) -> (B, C, 1, 1)
out = self.squeeze(x)
# (B, C, 1, 1) -> (B, C//16, 1, 1) -> (B, C, 1, 1)
out = self.excitation(out)
# (B, C, H, W) * (B, C, 1, 1) -> (B, C, H, W)
return x * out
This code only illustrates the SE block itself. In SE-ResNet, the block acts on the residual branch before that branch is added to the shortcut.
Isn't it easier to understand through code?
In SENet, global information is introduced through Global Average Pooling. This involves averaging the feature values of each channel's feature map and passing them through two fully connected transformations with ReLU between them and sigmoid at the end to obtain channel scaling weights.
Although this global information isn't extensive, experiments have shown that this design can enhance network performance.
Now, after looking at the code, let's revisit the diagram provided by the authors:
In SENet, the steps are as follows:
- Global Pooling: Introduce global information.
- FC: Dimension reduction.
- ReLU: Introduce non-linearity.
- FC: Dimension increase.
- Sigmoid: Obtain importance weights for each channel.
The Sigmoid function is used because the authors wanted channels to combine rather than compete. Each channel gets its own continuous scaling weight, computed from the shared multi-channel descriptor. The weights need not sum to 1 as softmax outputs do, so several channels can receive large weights simultaneously.
Discussion
Image Classification Results
SE blocks can be integrated into many CNNs; parameter and runtime overhead depend on channel counts and placement.
For example, comparing ResNet-50 with SE-ResNet-50:
For a 224 × 224 pixel input image, ResNet-50 requires about 3.86 GFLOPs for a single forward pass. Each SE block involves a global average pooling operation during the squeeze phase and two small FC layers during the excitation phase. With a reduction ratio of 16, SE-ResNet-50 requires about 3.87 GFLOPs, only a 0.26% increase compared to the original ResNet-50.
In terms of parameter increase, the addition is about 10%, not shown in the table but mentioned in the paper.
In Table 2, under the same training setup, ResNet-50 and SE-ResNet-50 have top-5 error rates of 7.48% and 6.62%. The latter approaches ResNet-101's 6.52% at roughly half its GFLOPs. The original column reports earlier published results; the 0.86-percentage-point difference is against the authors' retrained baseline.
Results on Lightweight Models
On MobileNet, the SE variant reduces ImageNet top-1 error from the authors' retrained 28.4% baseline to 25.3%, a 3.1-percentage-point reduction. The 29.4% original column is a separate published result.
On ShuffleNet, the retrained baseline drops from 32.6% to 31.0%, a 1.6-point reduction. Parameters nevertheless rise from 4.2M to 4.7M for MobileNet and from 1.8M to 2.4M for ShuffleNet; that matters on small models.
Importance of Squeeze
The authors removed global pooling and used spatial 1×1 convolutions for channel mapping, calling the result NoSqueeze. It still worked! But against the same ResNet-50 baseline of 23.30% top-1 error, full SE reaches 22.28% (1.02 points lower) while NoSqueeze reaches 22.93% (only 0.37 points lower). Both use 28.1M parameters, yet NoSqueeze costs 4.27 GFLOPs versus full SE's 3.87.
Importance of Excitation
The authors sampled 50 ImageNet validation images each for goldfish, pug, plane, and cliff, then plotted class-mean excitation values for sampled channels at different depths. all is the mean across all 1,000 classes. This is an illustrative sample:
- Early in the network, activations are similar across categories, but deeper layers show more category-specific activations.
- In deeper layers (e.g., SE_4_6 and SE_5_1), specific channels exhibit strong category preferences, indicating SE modules' significant role in these layers for feature discrimination.
- In the SE_5_2 layer, many activation values approach 1, so scaling for those channels approaches an identity mapping.
- At the network's end, the SE_5_3 curves also have similar shapes across classes. Together with the last-stage removal experiment, this suggests room to save parameters in this configuration.
The same SE block design can play different roles at different network depths.
Other Experimental Results
Within the reported ablation setup, the authors found:
- Table 12: replacing the final excitation nonlinearity yields top-1 errors of 22.28% for sigmoid, 23.00% for tanh, and 23.47% for ReLU.
- Table 11: average pooling has 22.28% top-1 error versus 22.57% for max pooling.
- Table 10: reduction ratio 16 gives 22.28% top-1 error with 28.1M parameters, a useful trade-off rather than the highest accuracy.
For more detailed tables, please refer to the original paper. We believe these experiments are more relevant to engineering practices, so we won't list them all.
Conclusion
By adding SE blocks, SENet improves a CNN's ability to learn and generalize in the reported experiments. Its key idea is a simple way to bring attention into a convolutional network: use global information to help the network focus on useful channels for each input.
The ImageNet comparisons and ablations support that idea. In Table 16, global information helps more than adding the same number of parameters without a squeeze operation. There is a cost, though: ResNet-50 rises only from 3.86 to 3.87 GFLOPs, but parameters increase by about 10%.