Back to paper library
arXiv 2019CVPR Workshops 2020Classic CNNsPaper notes

CSPNet

Half-Half-Half Plugin

Original paper

CSPNet: A New Backbone that can Enhance Learning Capability of CNN

Paper authors

Chien-Yao Wang · Hong-Yuan Mark Liao · I-Hau Yeh · Yueh-Hua Wu · Ping-Yang Chen · Jun-Wei Hsieh

Publication
CVPR Workshops 2020Workshop
First submitted to arXiv
arXiv:1911.11929
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{classiccnnscspnet2019,
  title = {{CSPNet: A New Backbone that can Enhance Learning Capability of CNN}},
  author = {Chien-Yao Wang and Hong-Yuan Mark Liao and I-Hau Yeh and Yueh-Hua Wu and Ping-Yang Chen and Jun-Wei Hsieh},
  year = {2019},
  eprint = {1911.11929},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1911.11929}
}
On this page

CSPNet stands for "Cross Stage Partial Network."

This paper has been widely used in industry. There are also many related implementations in Timm, such as cspdarknet53, cspresnet50 and cspresnext50.

Problem Definition

The authors examine existing network architectures and take an interest in DenseNet's feature reuse. They identify two problems to improve:

  1. Computational bottlenecks: a bottleneck occurs when parts of a computation cannot use resources such as processing units, memory or data transfer effectively, limiting overall performance. In a CNN, one layer may do much more work than others, leaving some units busy while others wait. The authors therefore consider per-layer load as well as total FLOPs. Their Figure 7 measures backbone and detection-head bottlenecks against different baselines; those percentages are not interchangeable.

  2. High memory use: DenseNet improves feature reuse, but its many concatenations can increase reads, writes and intermediate storage. Later designs such as PeleeNet and VoVNet still face related costs. The paper uses convolutional input/output (CIO) as a proxy for traffic; CIO is neither measured peak GPU memory nor measured DRAM bandwidth.

Solution

Model Architecture

Paper Figure 2: (a) passes the entire input through the dense block and a transition; (b) splits input channels, sends one branch through the dense block and the other around it, then merges them at the stage end and applies a second transition. This shows a stage, not the complete detector.

The upper panel shows the original DenseNet stage. In simple terms, follow its arrows like this:

  1. A convolution takes input x0x_0 and produces a new feature x1x_1.
  2. Concatenate x1x_1 with x0x_0 for the next layer.
  3. That layer produces x2x_2; concatenate it with x0x_0 and x1x_1.
  4. Repeat as new features enter the later layers, then use the transition layer.

The authors interpret these repeated connections through backpropagated gradient combinations: several layers repeatedly receive related information. This motivates their design.


To reduce memory use, the authors' change is simple: do not send every feature map through the dense block. In panel (b), they split the stage's input channels into two paths. One passes through the dense block and its first transition; the other bypasses the block. They concatenate the paths and apply a second transition. The bypass features are retained, and no branch is detached from backpropagation.

Key differences in the new design include:

  1. Change gradient combinations: two branches rejoin at different points, so gradients reaching the dense block and bypass differ. This is not a universal doubling of every path count.
  2. Reduce local compute peaks: for example, only half the base channels enter a dense block. The paper reports roughly halving a PeleeNet backbone bottleneck and roughly 80% for a particular YOLOv3-head comparison; they use different baselines.
  3. Lower estimated memory traffic: when base channels greatly outnumber the growth rate, partial-block CIO approaches half the original estimate.

Thus CSP keeps feature reuse within the dense path while reserving some channels until the stage end. Whether it saves time and memory on a device is a separate measurement.

Transition Layer

Paper Figure 3: (a) is DenseNet; (b) uses transition → concatenation → transition in full CSPDenseNet; (c) Fusion First concatenates before transition; (d) Fusion Last applies transition before concatenation. The last two are order-of-fusion ablations.

Splitting feature maps is easy, but how do you merge them back together?

To answer this, the authors proposed two fusion strategies:

  1. Fusion First: panel (c) concatenates both paths and then applies one transition; the paper argues this still mixes related gradient information repeatedly.
  2. Fusion Last: panel (d) applies a transition to the dense path before joining the bypass, reducing the repetition of gradient information between the paths. Gradients return along both paths; the design does not call detach on a branch.

The full CSPDenseNet in panel (b) also applies another transition after concatenation.

The experimental results of the two strategies are as follows:

Paper Figure 4: the x-axis counts billions of FLOPs and the y-axis is ILSVRC2012 top-1 accuracy. Red, green and blue arrows show different cost and accuracy changes relative to PeleeNet; this is not a latency plot.

Using PeleeNet's 70.7%/1.017B FLOPs as the baseline, at the same γ=0.5\gamma=0.5:

  1. Fusion First: 69.2%/0.755B, about 26% fewer FLOPs but 1.5 percentage points lower top-1.
  2. Fusion Last: 70.6%/0.804B, about 21% fewer FLOPs and 0.1 point lower top-1.
  3. Full CSPPeleeNet: 70.9%/0.888B, about 13% fewer FLOPs and 0.2 point higher top-1.

The first two rows isolate fusion order; the third is the full partial-transition model. Panel (d)'s Fusion Last is not the same architecture as panel (b).

The comparison explains why the authors favor Fusion Last's order over Fusion First's.

Exact Fusion Model

Paper Figure 6: (a) FPN fuses neighboring scales, (b) GFM gathers all scales, and (c) EFM selects scales according to anchor field of view. Colored arrows show backbone, fusion, prediction and output paths, not measured accuracy or speed.

After discussing the construction of the CSP module, the core concept of the paper is essentially covered.

The final section focuses on optimizing the FPN structure.


EFM in (c) merges the three scales around an anchor's assigned scale, providing a suitable field of view without always fusing every scale. It is a feature-pyramid change for a YOLOv3-style detector, distinct from CSP backbone partitioning. Like PANet, it handles multiple scales, but it is not simply PANet with more global information. Related reading:

To reduce the channels after pyramid concatenation, the authors use cross-channel Maxout. The paper argues it helps memory traffic and compute; measured savings depend on the detector and hardware.

Discussion

Effectiveness of the CSP Architecture

Paper Table 1: γ\gamma controls the channel split; higher top-1 and lower BFLOPs are preferable. This table does not list parameter counts or measured memory, so those costs cannot be inferred from its columns.

The first question to verify is whether the CSP architecture is indeed effective.

The authors took a previous backbone model, PeleeNet, and modified it to create a CSP-based version for comparison.

Here γ\gamma controls the channel split; γ=0.5\gamma=0.5 gives two equal paths. Compare γ=0.25\gamma=0.25, 0.50.5 and 0.750.75 within the same fusion strategy. Combining one row's accuracy with another row's FLOPs would describe no actual model. The table reports no parameter counts.

At γ=0.5\gamma=0.5, Fusion First scores 69.2%, below SPeleeNet's 69.6% but above PeleeNeXt's 68.9%. Fusion Last scores 70.6%/0.804B versus PeleeNet's 70.7%/1.017B: roughly 21% fewer FLOPs and 0.1 percentage point lower top-1. Those are the green arrow's values, not full CSPPeleeNet's.

At γ=0.25\gamma=0.25, Fusion Last reaches 70.8%/0.902B, roughly 11% fewer FLOPs and 0.1 point higher top-1 than PeleeNet. Full CSPPeleeNet at that ratio is a different row: 71.5%/0.986B.

Full CSPPeleeNet at γ=0.5\gamma=0.5 uses about 13% fewer FLOPs than PeleeNet and improves top-1 by 0.2 point. At γ=0.25\gamma=0.25, it uses about 3% fewer FLOPs and improves top-1 by 0.8 point. These are ImageNet ablations, not device latency measurements.

Effectiveness of the EFM Architecture

Paper Table 2: all rows use a CSPPeleeNet backbone. GFM reports 134 fps/24.3 AP/46.2 AP50; EFM 132/26.4/48.6; EFM(SAM) 129/26.8/49.0.

The authors performed an ablation study on the EFM using the MS COCO dataset, with results shown above.

Plain EFM is 2 fps slower than GFM but improves AP 24.3→26.4 and AP50AP_{50} 46.2→48.6, gains of 2.1/2.4 points. EFM(SAM) versus EFM(SPP) yields 129/128 fps and 26.8/26.2 AP in this ablation, so the authors choose EFM(SAM).

Performance on ImageNet

Paper Table 3: the PeleeNet, ResNet-10, ResNeXt-50 and DenseNet-201-Elastic pairs are separate comparisons. Its final three rows use ten crops and should not be combined with the single-result rows above.

Across four paired configurations in the table, adding CSP reduces BFLOPs by about 13–22% while top-1 holds steady or rises slightly. These are classification results under the paper's training setup; parameter count need not fall.

For lightweight models, CSPNet proved particularly effective, such as:

  • ResNet-10→CSPResNet-10: 63.5→65.3% top-1 and 2.273→1.905B FLOPs, a 1.8-point accuracy gain.
  • PeleeNet→CSPPeleeNet: 70.7→70.9% and 1.017→0.888B. DenseNet-201-Elastic and its CSP version show 77.9→77.9% and 8.77→7.13B.
  • ResNeXt-50→CSPResNeXt-50: 77.8→77.9% and 10.11→7.93B. The CSP model's top-5 is 94.0%, slightly below the original's 94.2%, so not every metric improves.

Performance on MS COCO

Paper Table 4: rows are grouped by GTX 1080 Ti FPS. Changing a model's input size also changes FPS and AP. Most rows use COCO test-dev, but the two TTFNet rows use minval5k. This is not a CPU or Jetson TX2 speed table.

Among the 30–100 fps rows, CSPResNeXt50 + PANet(SPP) at 608×608 records 35 fps/38.4 AP/60.6 AP50AP_{50}/41.6 AP75AP_{75}. YOLOv3(SPP) also reaches 60.6 AP50AP_{50}, so the CSP row does not uniquely lead that column. COCO AP is not a per-image “detection rate.”

In the 100–200 fps group, Pelee at 304×304 reports 106 fps/38.3 AP50AP_{50}; CSPPeleeNet + EFM(SAM) at 512×512 reports 109 fps/50.4 AP50AP_{50}. The 12.1-point gap includes changes in backbone, head and input size, so it cannot all be assigned to CSP. At 129 fps, EFM(SAM) at 416×416 reports 49.0 AP50AP_{50} versus CenterNet at 512×512 and 44.9, a 4.1-point difference.

Among rows above 300 fps, the precise model name is CSPDenseNetb Reference + PRN. At 320×320 it gives 400 fps/34.2 AP50AP_{50} versus SNet49-ThunderNet's 267 fps/33.7, a 133-fps and 0.5-point difference. These are GTX 1080 Ti results with batch size one where possible, not mobile GPU results.

Inference Speed

Paper Table 5: GPU, CPU and mGPU refer to different hardware. A dash means no measurement. The final column is COCO AP50AP_{50}, not speed; comparison models use input sizes of both 320 and 416.

The authors separate desktop GTX 1080 Ti, Intel Core i9-9900K CPU and Jetson TX2 mobile-GPU FPS. CPU inference uses OpenCV DNN. They apply no compression or quantization, but architecture and input size are not fully controlled across rows.

In Table 5's CPU column, CSPDenseNetb Ref.-PRN scores 102 fps/34.2 AP50AP_{50}. SNet49-ThunderNet gives 47/33.7, YOLOv3-tiny 54/33.1, and YOLOv3-tiny-PRN 71/33.1. Thus its CPU FPS exceeds them by 55/48/31; those gaps are not GPU or TX2 results.

On the mobile GPU, CSPPeleeNet Ref.-EFM(SAM) gives 41 fps/44.6 AP50AP_{50} versus YOLOv3-tiny's 37 fps/33.1. The former uses 320×320 input and the latter 416×416. The paper also claims EFM reduces PeleeNet feature-pyramid memory use; Table 5 lists FPS and AP50AP_{50}, not memory, so it cannot verify a savings ratio.

Conclusion

CSPNet uses channel splitting, feature fusion and different gradient paths to improve the compute efficiency of several architectures. The paper's ImageNet classification and COCO detection results support these designs in the configurations tested, including edge-device inference experiments.

Although the architecture becomes more complex, the reported configurations reduce FLOPs and estimated memory traffic while keeping accuracy high, making the design promising for object detection and other real-time tasks. Its measured speed still depends on the hardware and the rest of the detector.

Rather than being a backbone, CSPNet behaves more like a plugin. Next time you design a model, consider integrating the CSP structure—it might deliver some surprising benefits!