Back to paper library
arXiv 2018CVPR 2018Feature fusionPaper notes

PANet

Give Me a Shortcut

Original paper

Path Aggregation Network for Instance Segmentation

Paper authors

Shu Liu · Lu Qi · Haifang Qin · Jianping Shi · Jiaya Jia

Publication
CVPR 2018Conference
First submitted to arXiv
arXiv:1803.01534
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{featurefusionpanet2018,
  title = {{Path Aggregation Network for Instance Segmentation}},
  author = {Shu Liu and Lu Qi and Haifang Qin and Jianping Shi and Jiaya Jia},
  year = {2018},
  eprint = {1803.01534},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1803.01534}
}
On this page

After the introduction of the classic FPN, the question of "how to achieve more efficient feature fusion" became an area of research. Let's explore another equally classic architecture — PANet.

Defining the Problem

In the PANet paper, the primary comparison is with the FPN architecture. If you're not familiar with FPN, you can refer to:

Recall a key conclusion from FPN: "Bottom-up, top-down, then add."

PANet asks how this can be improved. The authors first review the motivation for FPN’s top-down path:

The insightful point that neurons in high layers strongly respond to entire objects while other neurons are more likely to be activated by local texture and patterns manifests the necessity of augmenting a top-down path to propagate semantically strong features and enhance all features with reasonable classification capability in FPN.

Excerpted from PANet

A direct translation would lose some of that passage's beauty, so it stays in English. Put simply, neurons at the top can see whole objects, while neurons at the bottom see local textures. The quote explains FPN's top-down route; PANet asks for another route back up.

That passage already sums up the paper's central idea.

Teaching diagram reused from the earlier note; PANet Figure 1 uses FPN levels P2–P5 and the labels are not directly interchangeable

Using this image, let's break down the statement. High layers refer to Block5, where neurons have a broad receptive field and can focus on larger objects. Lower-level features tend to respond to local textures and patterns. Labels such as P1 follow the earlier teaching diagram; the PANet paper uses FPN levels P2–P5.

FPN quickly propagates high-level semantics downward, while precise low-level localization cues may traverse over a hundred backbone layers to reach the top. PANet adds a short bottom-up route across feature scales.

That's quite a difference. Shouldn't we do something about it?

Solving the Problem

PANet Model Design

Liu et al., Path Aggregation Network for Instance Segmentation, Figure 1(a)–(e), arXiv:1803.01534v4

The image shows the proposal. Panel (a) is the FPN we discussed before; panel (b) adds a bottom-up shortcut (green) beside the longer original path (red). Panel (c) pools information from multiple scales for each proposal, panel (d) is the class and box branch, and panel (e) is fully connected mask fusion. We will focus on (b) here, then briefly return to (c) and (e).

Path Aggregation FPN

Let's look at the path aggregation block. It takes a lower-level feature map (NiN_i) and a coarser FPN map (Pi+1P_{i+1}) to create Ni+1N_{i+1}.

For each feature map (Ni), a 3×3 convolution with stride=2 is first applied to halve the spatial dimensions and reduce the amount of data. Note that N2 is simply P2, with no processing.

Then, corresponding small patches from Pi+1 and the downsampled Ni are added together. Finally, another 3×3 convolution processes this combined feature map, creating the new feature map (Ni+1) for subsequent sub-networks.

This process repeats until reaching the top feature map (P5). Throughout these blocks, PANet maintains the feature map's channel count at 256, with each convolution followed by a ReLU activation. The authors believe this helps the model learn more useful features.

The outputs [N2, N3, N4, N5] are multi-scale feature maps. Adaptive feature pooling subsequently extracts aligned regions for each proposal from these scales and combines those features for the prediction heads; it does not directly sum entire maps of different sizes.

Discussion

How Does It Compare to Basic FPN?

Liu et al., PANet, annotated crop of Table 3, arXiv:1803.01534v4; colored Chinese callouts were added for this article

As we look more closely at the architecture, we can see that the authors ran quite a few analyses and experiments. Let's look at a few highlights.

First, consider Multi-Scale Training (MST). Starting from the reimplemented baseline RBL, it raises mask AP from 33.4 to 35.3: 1.9 points, or about 2 points. This is a training-strategy change, not a replacement of the FPN architecture.

Next, adding multi-GPU synchronized BatchNorm (MBN) to MST raises mask AP from 35.3 to 35.7, or 0.4 points. Keeping MST+MBN fixed, adding Bottom-Up Path Augmentation (BPA) raises it from 35.7 to 36.4, another 0.7 points. BPA gives lower-level localization cues a shorter route to coarser features.

I find this ablation a little unfortunate: MBN sits between MST and BPA, so we cannot see a direct comparison of MST with MST+BPA alone. The 0.7-point BPA gain is measured on top of MST+MBN.

Besides "Bottom-Up Path Augmentation," the authors introduced other key elements, although they fall within the Model Head section and are not the focus here. Nonetheless, since the table is provided, let's briefly cover some information.

One element is "Adaptive Feature Pooling," which connects the feature grid with all feature levels, allowing useful information from each level to propagate directly to subsequent sub-networks. This enhances cooperation between different feature levels, improving the model's performance. Fully Connected Fusion adds a branch complementary to the convolutional mask branch and combines their predictions to improve mask quality.

Conclusion

PANet carries FPN's design forward by exploring a more effective way to fuse multi-scale features. Its path augmentation, adaptive feature pooling and prediction branches show how the different levels can work together in the paper's instance-segmentation setting.

PANet conveys two critical messages:

First, when facing a multi-scale problem, we need to consider how to fuse the features. PANet pays close attention to features at different scales, then uses path augmentation, feature pooling and fusion to bring their information together.

Second, unlike FPN's "Bottom-up, top-down, then add," PANet proposes "Bottom-up, top-down, bottom-up, then add."

There are still unresolved issues in feature fusion, such as comparing additive and concatenative fusion, improving fusion efficiency, and adjusting fusion weights. These topics will be further explored in future research.