UNet++: Redesigning Skip Connections to Exploit Multiscale Features in Image Segmentation
- Publication
- IEEE Transactions on Medical Imaging 2020Journal
- First submitted to arXiv
- arXiv:1912.05074
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{featurefusionunetpp2019,
title = {{UNet++: Redesigning Skip Connections to Exploit Multiscale Features in Image Segmentation}},
author = {Zongwei Zhou and Md Mahfuzur Rahman Siddiquee and Nima Tajbakhsh and Jianming Liang},
year = {2019},
eprint = {1912.05074},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/1912.05074}
}
The authors of this paper use U-Net as a starting point to explore design issues in the original U-Net architecture.
The earlier note quotes a pointed question from “研习 U-Net”:
"Is there really nothing wrong with this topology that hasn't changed in three years?"
Excerpted from 研习 U-Net
The authors think U-Net has more than one problem to explore.
Problem Definition
Cropped from Figure 1(a–d) of the UNet++ paper. The four outputs run from to ; panel (d) is the deepest , not a fifth downsampling operation.
A common U-Net has five resolution levels and four downsampling operations, as in (d). Convolutional depth, resolution levels and the number of downsamplings are different quantities.
Why not three layers? Why not seven layers? How deep should the network be designed?
Network depth is often an important design choice. It affects what the model can represent, how much work it requires and how we train it. Let's look at the question from several angles:
-
Feature representation: Deeper networks can learn more complex and abstract features, useful in segmentation, detection and classification. Shallower ones may capture simpler, more local patterns. Adding downsampling levels also reduces spatial resolution, so segmentation needs to balance semantics and boundary detail.
-
Computational complexity: More depth often brings more parameters and computation, increasing training and inference costs. Width and connections matter too, so depth alone does not determine memory use or latency. Finding an appropriate depth is harder when resources are limited.
-
Overfitting and generalization: A larger model may overfit a small dataset. A shallower one might generalize better, but could lose representational capacity; neither outcome is automatic.
-
Optimization difficulty: You could downsample 100 times—if the image were large enough. As a network grows, vanishing or exploding gradients may make training harder, requiring attention to initialization and optimization.
-
Theory and practice: Depth can make some functions economical to represent, but this does not tell us the best depth for a particular task. Too deep or too shallow can both be poor choices.
-
Dataset and task: Different datasets and tasks need different levels of abstraction. One fixed depth may not serve them all equally well.
-
Interpretability and debugging: With more stages, tracing intermediate features and errors can take more work.
Thinking through how deep? helps us make a better design choice for the actual task. In the paper's depth experiments, the shallower U-Net beats the deepest version on Cell and Brain Tumor; on EM, the deeper models do better, but only marginally. UNet++ therefore asks whether U-Nets of different depths can share computation, learn together and offer a choice of outputs. It also asks whether directly joining encoder and decoder features at the same resolution imposes an unnecessary semantic gap on fusion.
Solution
UNet++ Model Design
Cropped from Figure 1(e–g). From left to right: share an encoder, replace long skips with short ones, and densely connect decoder nodes at the same resolution. Dotted lines are skip connections; these are not three separately trained networks.
For image segmentation, we would like the network to learn features at different depths and bring them together. The authors propose several structures to explore how that fusion can work. They did not draw the web in panel (g) out of thin air; panels (e) and (f) show the steps that led to it:
-
Unified Architecture (U-Netᵉ)
- Objective: Define a unified architecture for nested U-Nets
- U-Nets of varying depths share at least part of the encoder, while each has its own decoder and output. One training run can compare depths; training each from scratch and then ensembling predictions would share neither encoder computation nor learning.
-
Upgraded U-Net (UNet+)
- Objective: Replace the original long skips with short ones between adjacent nodes and test what connecting the decoders alone contributes.
- Deeper decoders can pass training signals to shallower ones. Comparing this stage with UNet++ then shows what dense connections add.
-
Advanced U-Net (UNet++)
- Objective: Test whether dense skips deliver features from multiple semantic stages to a decoder, beyond fusing just an encoder and decoder map.
- In panel (g), receives both an upsampled map from below and multiple preceding maps along its own row. Features accumulate horizontally and are upsampled vertically for gradual fusion.
Together, these designs try to retain U-Net's strengths while making depth choices and multi-depth feature fusion more flexible. The shared sub-networks and selectable outputs address the first question; dense connections change feature fusion for the second. Of course, the architecture did not suddenly grow into this shape—the authors' U-Net discussion describes more of their thinking along the way.
Discussion
Let's discuss several questions based on the above content:
Is It Just About More Parameters?
Cropped from the 2D columns of paper Table IV, with a red reading annotation. Values are mean IoU ± standard deviation (%); DS means deep supervision. This crop omits the 3D Lung Nodule column and some significance markings.
The authors widened the original U-Net to nearly match UNet++ in parameter count. A little crude, perhaps, but it addresses “is this just more parameters?” The table gives 7.8M for U-Net, 9.1M for wide U-Net and 9.0M for UNet++:
- Widening is not useless, but parameter count alone cannot explain UNet++'s results.
On Cell, wide U-Net scores 88.91 versus the original 88.73, a gain of 0.18 IoU points; UNet++ scores 91.03 without DS and 91.21 with it. EM reads 88.30 → 88.37 → 88.92/89.33. The size of the difference depends on the dataset, and DS does not raise every row: the UNet++ Nuclei result moves from 92.44 to 92.37. More parameters can improve expressiveness, but they can also increase compute and storage needs or overfitting risk on small datasets. This comparison suggests that connection design matters; it does not isolate every effect of UNet++'s design and training.
Deep Supervision and Model Pruning
Deep supervision and model pruning matter here because they let the network learn from several depths and then select a smaller branch for inference. That can be useful where resources are limited; mobile-device inference requires measurement.
-
Deep Supervision
Zhou et al., UNet++, Figure 1(g) crop, arXiv:1912.05074v2 This is a crop of paper Figure 1(g), not a separate architecture. Four lines from the top row to the loss symbol identify the supervised outputs.
Auxiliary losses supervise the full-resolution top-row outputs through , rather than one output at every resolution level. The paper combines pixel-wise cross-entropy and soft Dice loss for each output, then sums the four losses with equal weights. Sub-networks of different depths receive segmentation supervision while sharing encoder features. Gradients from later branches also update the shared earlier layers; this helps training.
-
Model Pruning
Zhou et al., UNet++, Figure 2, arXiv:1912.05074v2 Cropped from paper Figure 2. From left to right are through . Gray nodes are unnecessary for the chosen output; pruning selects the branch needed for the output.
Inference can average multiple branch predictions or use just one output. Evaluate each branch on a validation set, then choose how much of the network can be removed within an acceptable accuracy loss. Selecting removes the deeper branches on which that output does not depend; this is branch pruning, not arbitrary weight removal.
After considering deep supervision and pruning, we can ask whether this is feasible and why it matters:
- Feasibility: Deep supervision gives several top-row outputs, each producing a segmentation result. We can evaluate those sub-networks separately. During training, later branches still send gradients through shared earlier layers. At inference, removing branches that the chosen output does not use preserves that branch's own output, although its accuracy can differ from the full model or an average of outputs. The paper also compares joint training followed by pruning with training a shallow branch alone.
- Compute and resources: If an output is good enough, much of the deeper network can be removed, reducing work and storage. This may matter on resource-limited devices; actual mobile inference was not tested in this paper.
- Speed: In the paper's Figure 5 experiments, averages 32.2% less inference time and a 75.6% smaller memory footprint than unpruned , with 0.6 IoU points lower performance on average. Pruning more deeply loses more IoU. The result depends on the benchmark setup; to is not universally “three times faster.”
- Flexibility: Different tasks or datasets may justify different branch depths. A simpler dataset might tolerate a smaller model, but validation is needed to find the accuracy–size balance.
Deep supervision helps the branches learn together during training; pruning chooses the branch used for inference. In this paper, inference time was measured on a GPU, not directly on a mobile device.
Conclusion
The deep supervision and pruning experiments show UNet++'s potential for image segmentation. Supervision gives the different-depth outputs clearer training signals, while pruning can reduce computation and storage at an acceptable loss of accuracy. That trade-off could matter on resource-limited hardware.
From an engineering perspective, these methods also bring challenges. The chosen pruning depth depends on validation performance. A branch that works on one dataset might fail on another or in a real deployment—one careless cut and the model could drop dead. This is a risk to check, not a cross-dataset experiment reported in the paper.
Possible directions include adaptive pruning that adjusts the chosen branch, multi-objective optimization to balance accuracy and efficiency, and cross-dataset validation or transfer learning to test generalization. These are ideas to explore, not solutions established by this paper.
In practice, adding deep supervision and branch selection makes the model and training procedure more complex. Engineers may need more time to tune the pruning strategy and verify that it generalizes, lengthening the development cycle.
The paper offers another way to think about feature fusion, but turning it into a reliable system still takes work. I hope this note gives researchers a useful reference and some ideas to try.