Back to paper library
arXiv 2016CVPR 2017Feature fusionPaper notes

FPN

The Pyramid Structure

Original paper

Feature Pyramid Networks for Object Detection

Paper authors

Tsung-Yi Lin · Piotr Dollár · Ross Girshick · Kaiming He · Bharath Hariharan · Serge Belongie

Publication
CVPR 2017Conference
First submitted to arXiv
arXiv:1612.03144
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{featurefusionfpn2016,
  title = {{Feature Pyramid Networks for Object Detection}},
  author = {Tsung-Yi Lin and Piotr Dollár and Ross Girshick and Kaiming He and Bharath Hariharan and Serge Belongie},
  year = {2016},
  eprint = {1612.03144},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1612.03144}
}
On this page

Imagine a scenario.

Given a typical convolutional network model's operation flow: first, an input image, possibly of size 3 x 224 x 224, is passed through successive layers of downsampling, ultimately producing a high-dimensional semantic feature map, potentially 256 x 7 x 7. Conventionally, the final output of such a model is 1/32 the original size, meaning that for an input image of 224 x 224, the final feature map is 7 x 7.

This feature map has various descriptions:

  • Low-resolution feature map (since it's 1/32 of the original width and height)
  • High-level semantic features (typically with a large receptive field; semantic level and channel dimension are different concepts)
  • Top-level features (bottom | original image -> C1 -> C2 -> ... -> C5 | top)

This design poses no issue for classification tasks, where the goal is to reference the entire input image content to output one or several possible classification results.

However, using only this layer can make small-object localization difficult. As mentioned, this feature map is 1/32 the scale. If an object we want to detect is smaller than 32 x 32 pixels, its details may become difficult to distinguish after downsampling. The 32×32 size is not a hard threshold below which detection becomes impossible.

Okay, saying it "disappears" might be an exaggeration. In reality, if the model uses convolution operations for downsampling, the object might still leave some trace in the nearby pixels.

This indicates that to improve object detection performance, we need to do something to preserve these features.

Around the end of 2015, SSD was proposed:

Another important direction was to combine speed with multi-scale detection.

SSD and YOLO both pursue single-stage detection, but SSD is not YOLO v1 with feature fusion added. It predicts independently from several feature scales and combines detections; the original version does not perform FPN-style top-down fusion. This is an important precursor, but shallow and deep features still differ in semantic strength.

Defining the Problem

In this paper, the authors explicitly pointed out these main issues:

  1. Limitations of Traditional Feature Pyramids

    Extracting features independently at each image-pyramid scale can provide strong multi-scale representations. The main drawbacks are repeated computation and memory use. Applying such pyramids only at test time also creates a training/inference mismatch.

  2. Limitations of SSD

    SSD begins its prediction pyramid at relatively late features such as VGG conv4_3, then adds lower-resolution layers. Earlier high-resolution features are not fully reused. They matter for small objects, but need stronger semantics to become effective detection features.

Solving the Problem

FPN Model Design

Adapted from Lin et al., Feature Pyramid Networks for Object Detection, Figures 2 and 3, arXiv:1612.03144v2

FPN aims to provide multi-scale features reusable across detection systems. The authors proposed a structure, shown above, to create a feature pyramid with strong semantics at all scales while maintaining the pyramid shape of the convolutional network's feature hierarchy.

To achieve this, the authors designed a structure that:

  • Combines low-resolution features with high-resolution features through top-down pathways and lateral connections.

This sentence essentially concludes the paper, leaving the implementation and testing parts.

However, let's look at some implementation details provided by the authors.

  1. Bottom-up Pathway

    The first part is the data pathway from low-level to high-level features. I drew an additional ResNet18 diagram with concrete dimensions to illustrate this flow; it is a teaching example, not the paper’s experimental configuration.

    Taking ResNet18 as an example, an input image of 3 x 224 x 224. Based on PyTorch syntax, the channel count is at the front. After passing through ResNet18, we get five different resolution feature maps. The P1–P5 labels below follow the handwritten diagram. The paper instead names backbone features C2–C5 and fused outputs P2–P5; keep these notations separate:

    • P1: 1/2 size feature map, 64 x 112 x 112.
    • P2: 1/4 size feature map, 64 x 56 x 56.
    • P3: 1/8 size feature map, 128 x 28 x 28.
    • P4: 1/16 size feature map, 256 x 14 x 14.
    • P5: 1/32 size feature map, 512 x 7 x 7.

    This image passes through the model's downsampling process, termed as the bottom-up pathway in the paper.

    High-resolution features cost more resources, but the original FPN explicitly uses stride-4 C2 to produce P2 for small-object representation. It omits conv1, not C2.

  2. Top-down Pathway and Lateral Connections

    Recall the previous steps, where input data flows from left to right.

    The second stage of feature pyramid design involves merging features from right to left.

    Let's zoom in on the P4 - P5 section:

    During this fusion process, we first deal with the high-dimensional feature map due to its smaller size.

    First, we use a 1×1 convolution on the top feature to align its channels, then upsample it by a factor of two to match the next feature map's spatial size. In the original FPN, a 1×1 projection of C5 produces P5, followed by nearest-neighbor upsampling in the top-down path.

    Next, we aim to add these different dimension feature maps. However, this introduces a challenge: channel mismatch.

    As shown above, the feature map from Block5 has 512 channels, while the feature map from Block4 has only 256 channels. Due to the channel mismatch, direct addition is impossible. To resolve this, we use a 1×1 convolution to adjust the feature map's channel count. Note that there's no fixed rule for adjusting channel counts; you can predefine a suitable channel count based on practical needs, such as 64 in a teaching example; the paper uses 256.

    In the original paper, the selected backbone maps C2–C5 receive 1×1 projections so that the fused maps have matching channels. The half-resolution P1 in the hand-drawn illustration is not part of this pyramid.

    After aligning the channels and sizes, we can directly add feature maps from different layers, completing one round of fusion.

    The original paper also applies a 3×3 convolution to each fused map to reduce aliasing from upsampling.

    To further illustrate, let's look at the fusion from P4 to P3:

    Finally, here's a more practical architecture design, which might look like this:

Discussion

Is It Really Better?

To summarize the results: Yes.

The authors conducted several experiments to illustrate this, and we'll highlight some key charts from the paper. First, let's look at the ablation study provided in the paper:

Lin et al., Feature Pyramid Networks for Object Detection, Table 1, arXiv:1612.03144v2; proposal AR, not detection AP

This table reports RPN proposal average recall (AR), not detection AP, using up to 100 or 1000 proposals per image. At 1000 proposals, full FPN (c) reaches 56.3 AR; bottom-up only, without the top-down path (d), reaches 49.5; no lateral connections (e) reaches 46.1; and only the finest level P2 (f) reaches 51.3.

All three simplified variants trail full FPN, but by different amounts. These AR differences must not be reported as detection AP gains. Variant (f) also stays above the C4-only baseline (a) at 48.3, so this is more specific than saying every change merely returns to baseline.

Next, let's look at another table:

Lin et al., Feature Pyramid Networks for Object Detection, Table 4, arXiv:1612.03144v2

Table 4 then compares single-model detection AP with COCO competition entries, including G-RMI from 2016 and Faster R-CNN+++ from 2015. On test-dev, FPN's 36.2 AP exceeds the best prior entry in the table, AttractioNet at 35.7 AP, by 0.5 points. This is a different task and metric from the RPN AR table above, not a controlled ablation of one fusion block.

Conclusion

FPN introduces a simple framework that can be combined with various backbone networks to build powerful feature pyramids. This method shows significant improvements over multiple strong benchmark models and competition winners.

FPN makes two points clear:

  • First, when addressing problems with multi-scale characteristics, consider techniques for merging features from different scales.
  • Second, feature fusion can be summarized in three steps: "Bottom-up, top-down, then add."

Following this paper, many discussions emerged, such as how to design better feature fusion strategies, improve fusion efficiency, or adjust fusion weights, among other scenarios.

There are many subsequent papers, and we’ll explore them together when we have time.