Back to paper library
arXiv 2017CVPR 2018Efficient modelsPaper notes

SENet

Squeeze and Excitation Network

Original paper

Squeeze-and-Excitation Networks

Paper authors

Jie Hu · Li Shen · Samuel Albanie · Gang Sun · Enhua Wu

Publication
CVPR 2018Conference
First submitted to arXiv
arXiv:1709.01507
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{lightweightsenet2017,
  title = {{Squeeze-and-Excitation Networks}},
  author = {Jie Hu and Li Shen and Samuel Albanie and Gang Sun and Enhua Wu},
  year = {2017},
  eprint = {1709.01507},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1709.01507}
}
On this page

In an era where everyone is striving to top the ImageNet leaderboard, various network architectures emerge endlessly. The SENet architecture was proposed to enhance the global inter-channel relationships within a network, thereby improving its performance.

Defining the Problem

Past research observations from VGG and ResNet indicate that increasing depth can improve model quality, and residual connections help the network learn better features. Another line of work uses grouped and multi-branch convolutions, as in Inception. This is a relationship between design ideas, not a claim that all grouped-convolution research preceded Inception.

These studies reflect an assumption:

  • Channel relationships are usually modeled implicitly by local-receptive-field functions whose learned parameters do not change with each input instance.

However, the authors believe this assumption is problematic:

  • Can information from the whole feature map dynamically adjust channels for each input?

Stacked CNNs can already have global receptive fields. SE explicitly generates input-dependent channel weights.

Solving the Problem

The authors aim to introduce global information into the network channels without significantly increasing computational load and parameters. Thus, they proposed the Squeeze-and-Excitation (SE) module to introduce global information with minimal additional computational cost.

Model Architecture

Squeeze-and-Excitation Networks, Figure 1

This is another architecture that is easier to understand through code rather than text.

Let’s implement a simplified version. It assumes positive integer in_channels and reduction_ratio, with in_channels >= reduction_ratio. The C//16 comment refers to the default ratio:

python
import torch
import torch.nn as nn

class SEBlock(nn.Module):

    def __init__(self, in_channels, reduction_ratio=16):
        super(SEBlock, self).__init__()
        self.squeeze = nn.AdaptiveAvgPool2d(1)
        self.excitation = nn.Sequential(
            nn.Conv2d(in_channels, in_channels // reduction_ratio, kernel_size=1, stride=1, padding=0, bias=False),
            nn.ReLU(),
            nn.Conv2d(in_channels // reduction_ratio, in_channels, kernel_size=1, stride=1, padding=0, bias=False),
            nn.Sigmoid()
        )

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        # x: (B, C, H, W)

        # (B, C, H, W) -> (B, C, 1, 1)
        out = self.squeeze(x)

        # (B, C, 1, 1) -> (B, C//16, 1, 1) -> (B, C, 1, 1)
        out = self.excitation(out)

        # (B, C, H, W) * (B, C, 1, 1) -> (B, C, H, W)
        return x * out

This code only illustrates the SE block itself. In SE-ResNet, the block acts on the residual branch before that branch is added to the shortcut.

Isn't it easier to understand through code?

In SENet, global information is introduced through Global Average Pooling. This involves averaging the feature values of each channel's feature map and passing them through two fully connected transformations with ReLU between them and sigmoid at the end to obtain channel scaling weights.

Although this global information isn't extensive, experiments have shown that this design can enhance network performance.

Now, after looking at the code, let's revisit the diagram provided by the authors:

Squeeze-and-Excitation Networks, Figure 2

In SENet, the steps are as follows:

  1. Global Pooling: Introduce global information.
  2. FC: Dimension reduction.
  3. ReLU: Introduce non-linearity.
  4. FC: Dimension increase.
  5. Sigmoid: Obtain importance weights for each channel.

The Sigmoid function is used because the authors wanted channels to combine rather than compete. Each channel gets its own continuous scaling weight, computed from the shared multi-channel descriptor. The weights need not sum to 1 as softmax outputs do, so several channels can receive large weights simultaneously.

Discussion

Image Classification Results

Squeeze-and-Excitation Networks, Table 2; crop includes red editorial marks

SE blocks can be integrated into many CNNs; parameter and runtime overhead depend on channel counts and placement.

For example, comparing ResNet-50 with SE-ResNet-50:

For a 224 × 224 pixel input image, ResNet-50 requires about 3.86 GFLOPs for a single forward pass. Each SE block involves a global average pooling operation during the squeeze phase and two small FC layers during the excitation phase. With a reduction ratio of 16, SE-ResNet-50 requires about 3.87 GFLOPs, only a 0.26% increase compared to the original ResNet-50.

In terms of parameter increase, the addition is about 10%, not shown in the table but mentioned in the paper.

In Table 2, under the same training setup, ResNet-50 and SE-ResNet-50 have top-5 error rates of 7.48% and 6.62%. The latter approaches ResNet-101's 6.52% at roughly half its GFLOPs. The original column reports earlier published results; the 0.86-percentage-point difference is against the authors' retrained baseline.

Results on Lightweight Models

Squeeze-and-Excitation Networks, Table 3

On MobileNet, the SE variant reduces ImageNet top-1 error from the authors' retrained 28.4% baseline to 25.3%, a 3.1-percentage-point reduction. The 29.4% original column is a separate published result.

On ShuffleNet, the retrained baseline drops from 32.6% to 31.0%, a 1.6-point reduction. Parameters nevertheless rise from 4.2M to 4.7M for MobileNet and from 1.8M to 2.4M for ShuffleNet; that matters on small models.

Importance of Squeeze

Squeeze-and-Excitation Networks, Table 16

The authors removed global pooling and used spatial 1×1 convolutions for channel mapping, calling the result NoSqueeze. It still worked! But against the same ResNet-50 baseline of 23.30% top-1 error, full SE reaches 22.28% (1.02 points lower) while NoSqueeze reaches 22.93% (only 0.37 points lower). Both use 28.1M parameters, yet NoSqueeze costs 4.27 GFLOPs versus full SE's 3.87.

Importance of Excitation

Squeeze-and-Excitation Networks, Figure 6

The authors sampled 50 ImageNet validation images each for goldfish, pug, plane, and cliff, then plotted class-mean excitation values for sampled channels at different depths. all is the mean across all 1,000 classes. This is an illustrative sample:

  • Early in the network, activations are similar across categories, but deeper layers show more category-specific activations.
  • In deeper layers (e.g., SE_4_6 and SE_5_1), specific channels exhibit strong category preferences, indicating SE modules' significant role in these layers for feature discrimination.
  • In the SE_5_2 layer, many activation values approach 1, so scaling for those channels approaches an identity mapping.
  • At the network's end, the SE_5_3 curves also have similar shapes across classes. Together with the last-stage removal experiment, this suggests room to save parameters in this configuration.

The same SE block design can play different roles at different network depths.

Other Experimental Results

Within the reported ablation setup, the authors found:

  1. Table 12: replacing the final excitation nonlinearity yields top-1 errors of 22.28% for sigmoid, 23.00% for tanh, and 23.47% for ReLU.
  2. Table 11: average pooling has 22.28% top-1 error versus 22.57% for max pooling.
  3. Table 10: reduction ratio 16 gives 22.28% top-1 error with 28.1M parameters, a useful trade-off rather than the highest accuracy.

For more detailed tables, please refer to the original paper. We believe these experiments are more relevant to engineering practices, so we won't list them all.

Conclusion

By adding SE blocks, SENet improves a CNN's ability to learn and generalize in the reported experiments. Its key idea is a simple way to bring attention into a convolutional network: use global information to help the network focus on useful channels for each input.

The ImageNet comparisons and ablations support that idea. In Table 16, global information helps more than adding the same number of parameters without a squeeze operation. There is a cost, though: ResNet-50 rises only from 3.86 to 3.87 GFLOPs, but parameters increase by about 10%.