Back to paper library
arXiv 2019CVPR 2019Feature fusionPaper notes

NAS-FPN

Money Talks

Original paper

NAS-FPN: Learning Scalable Feature Pyramid Architecture for Object Detection

Paper authors

Golnaz Ghiasi · Tsung-Yi Lin · Ruoming Pang · Quoc V. Le

Publication
CVPR 2019Conference
First submitted to arXiv
arXiv:1904.07392
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{featurefusionnasfpn2019,
  title = {{NAS-FPN: Learning Scalable Feature Pyramid Architecture for Object Detection}},
  author = {Golnaz Ghiasi and Tsung-Yi Lin and Ruoming Pang and Quoc V. Le},
  year = {2019},
  eprint = {1904.07392},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1904.07392}
}
On this page

After the introduction of FPN, feature fusion has been a hot topic of discussion. Here, we list some papers chronologically:

Among these, PANet is quite well-known. The other papers also explore useful approaches to multi-scale fusion and are worth reading.

So, which one should we choose?

Google wanted to know this too, so they proposed NAS-FPN.

The idea is: if we do not know which connections work best, let an algorithm search for them.

  • Let’s conduct a massive search!

What is NAS?

NAS stands for Neural Architecture Search, an important research direction in deep learning. Its goal is to find useful architectures for a task within a specified search space and resource budget. Neural network architectures typically consist of multiple layers, neurons, and connections, and the design of these architectures can significantly impact model performance.

Traditionally, neural network design is manually conducted by experts, requiring extensive experimentation and adjustment. This is a time-consuming process that demands professional knowledge. NAS aims to simplify this process through automation, allowing machines to explore candidate architectures, without guaranteeing a global optimum.

In NAS, a search space is defined, containing variants allowed by researcher-defined operations and connection rules, not every possible neural network. Different search strategies, such as genetic algorithms, reinforcement learning, and evolutionary algorithms, are used to automatically generate, evaluate, and select these architectures to find the best-performing one for a specific task.

Generally speaking, the advantages and disadvantages of NAS are:

Advantages

  • Automation: NAS reduces manual design of individual candidates; total time and resource use still depend on the search method.
  • Optimization: NAS compares candidates against task, dataset and cost objectives, potentially outperforming a manually designed baseline.
  • Flexibility: NAS is not limited to specific tasks or architectures and can adapt to different application scenarios, generating models that suit specific needs.
  • Innovation: NAS helps discover new neural network structures, potentially leading to innovative model architectures and further advancement in deep learning.

Disadvantages

  • Computational Resource Consumption: The search process may require massive computational resources, including GPUs or TPUs, and significant time, which may limit practical application.
  • Complexity: The size of the search space and the number of possible combinations can make the search process very complex, requiring advanced algorithms and techniques for effective searching.
  • Data Dependency: The best architecture found may be highly dependent on the specific dataset used for the search, with no guarantee of similar performance on other datasets.
  • Randomness: The search process may have a degree of randomness, leading to different results in different runs, challenging the stability of the outcomes.

More Disadvantages

NAS sounds flexible and inventive, but most of us do not have the compute to build our own search system. We usually borrow the result, which immediately raises another question:

  • Does my use case match the paper’s?

The relevant conditions include training and inference data distributions, hardware and compute budget. A mismatch does not automatically make the architecture unsuitable:

  • Its transfer performance needs validation.

NAS-FPN itself tests different backbones and input resolutions. This demonstrates transfer in those settings.

So, why discuss this paper? Perhaps we will be among those who can run a search one day. It shows how to define the search space and evaluation process for our own feature-fusion problem. Even without rerunning it, the discovered connections can give us design ideas.

Solving the Problem

NAS-FPN Model Design

Ghiasi et al., NAS-FPN, Figure 2, arXiv:1904.07392v1

The primary goal of this study is to find a better FPN architecture. In academic terms, the initial part of a model is called the backbone, which can be replaced with structures like ResNet, MobileNet, etc.

Next, the FPN part is usually called the neck, responsible for multi-scale feature connections, which is the focus here.

After the neck comes the head, which predicts classes and bounding boxes. Engineers commonly compare and adapt these three parts.

In this study, the authors used a structure called "RetinaNet" as the base, with ResNet as the backbone and FPN as the neck.

Merging Cells

Ghiasi et al., NAS-FPN, Figure 3, arXiv:1904.07392v1

In NAS-FPN, based on the original FPN design, the authors introduced a new concept called "Merging Cells."

A merging cell is a small module responsible for merging two different input feature layers into a new output feature layer. The merging process involves the following steps:

  1. Select the first feature layer: Choose a candidate, called hi. The backbone inputs are C3–C7, aligned to a common channel count; P3–P7 are the pyramid outputs. Intermediate results join the candidate pool later.
  2. Select the second feature layer: Choose another candidate, called hj, without replacement within this cell.
  3. Determine the output feature size: Choose a resolution size, which will be the size of the new feature layer.
  4. Select the merging operation: Use a specific mathematical operation (such as addition or global pooling) to merge hi and hj.

In step 4, as shown below, two binary operations were designed in the search space: sum and global pooling. In the global-pooling operation, one input is pooled to form a channel gate that modulates the other input before fusion. These two fusion operations introduce no trainable parameters, but the following convolution and BatchNorm do.

If hi and hj differ in resolution, nearest-neighbor upsampling or max pooling aligns them before fusion. The new feature layer then passes through ReLU, a 3×3 convolution and BatchNorm. The cells eventually produce P3–P7; because input and output scales match, the entire pyramid network can be stacked.

Discussion

An RNN controller is trained with PPO, using validation AP as the reward. The proxy task uses ResNet-10, 512×512 inputs and 10 training epochs; 100 TPU workers evaluate candidates in parallel.

The number of newly sampled unique architectures levels off after roughly 8,000 controller steps. These are controller updates, not the detector’s training steps.

The authors select the highest proxy-task validation AP among sampled architectures for full training and evaluation.

The selected architecture was first sampled at around controller step 8,000 and was sampled repeatedly afterwards.

Then, the authors presented the FPN architecture obtained through NAS, as shown below:

Ghiasi et al., NAS-FPN, Figure 7(f) crop, arXiv:1904.07392v1; not final COCO test-dev AP

This diagram looks complex, giving a feeling of being impressive but incomprehensible.

But it isn’t. Let’s annotate it:

The annotation helps us follow the connections. Its green “C1–C5” labels were added by this note; the actual paper uses C3–C7 as inputs and P3–P7 as outputs. Those labels are not model specifications.

The paper labels (a) as the baseline FPN. This diagram expands feature levels and operations; the connections and resolutions matter more than counting nodes and calling it a “pseudo FPN.”

In full Figure 7, the 7.5, 9.9, 15.0, 16.0 and 16.8 AP values from (b) to (f) are ResNet-10 proxy-task search rewards, not final COCO test-dev detector scores. Higher-reward architectures contain bidirectional cross-scale connections, echoing part of PANet’s design intuition:

  • High-level, low-resolution features flow toward higher-resolution levels.
  • Higher-resolution features also flow toward lower-resolution levels.
  • The details differ from PANet.
Ghiasi et al., NAS-FPN, Figure 7, arXiv:1904.07392v1

For fully trained COCO test-dev results, Table 1 gives 37.0 AP/37.5 ms for FPN and 39.9 AP/56.1 ms for seven-layer NAS-FPN, both with ResNet-50 and 640×640 inputs. Accuracy rises with latency. The 48.3 AP configuration instead uses AmoebaNet, seven NAS-FPN layers and DropBlock; it must not be confused with the 16.8 proxy reward.

Conclusion

Earlier feature-fusion architectures were mostly designed by hand and tested experimentally. We can accept a conclusion based on experiments without a complete theory, but how do we decide whether those experiments explored enough designs?

NAS-FPN answers with a defined search space and an unusually large compute budget. Its selected connections include bidirectional paths reminiscent of PANet, though the connection details differ. That resemblance supports looking more closely at PANet's design idea.

Money can make a search much bigger. We still need to define the right problem and test whether its result works for us.