{"notes":[{"id":"segmentation/fcn","category":"segmentation","legacyPath":"segmentation/1411-fcn","legacySlug":"1411-fcn","legacyPaths":[],"title":"FCN","subtitle":"Fully Convolutional Networks","author":"Z. Yuan","paperDate":"2014-11-14","publishedOn":"2024-09-11","paperTitle":"Fully Convolutional Networks for Semantic Segmentation","paperUrl":"https://arxiv.org/abs/1411.4038","bibliography":{"title":"Fully Convolutional Networks for Semantic Segmentation","authors":["Jonathan Long","Evan Shelhamer","Trevor Darrell"],"url":"https://arxiv.org/abs/1411.4038","pdfUrl":"https://arxiv.org/pdf/1411.4038","codeUrl":null,"preprint":{"id":"1411.4038","submittedOn":"2014-11-14","url":"https://arxiv.org/abs/1411.4038"},"publication":{"venue":"CVPR","year":2015,"kind":"conference","url":"https://arxiv.org/abs/1411.4038"},"citation":"@misc{segmentationfcn2014,\n  title = {{Fully Convolutional Networks for Semantic Segmentation}},\n  author = {Jonathan Long and Evan Shelhamer and Trevor Darrell},\n  year = {2014},\n  eprint = {1411.4038},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1411.4038}\n}\n","citationPath":"/content/papers/citations/segmentation/fcn.1c65fe1c14c6.bib"},"thumbnail":{"src":"/content/papers/assets/1a8a4d2936086814cc5bd26819653959fe4cc29c64c5cba2009f8b8e43c87842.jpg","width":1224,"height":684},"locale":"en","path":"/content/papers/en/segmentation/fcn.05e555100992.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"model-architecture","text":"Model Architecture","depth":3},{"id":"upsampling","text":"Upsampling","depth":3},{"id":"multi-scale-fusion","text":"Multi-Scale Fusion","depth":3},{"id":"training-dataset-and-strategy","text":"Training Dataset and Strategy","depth":3},{"id":"evaluation-metrics","text":"Evaluation Metrics","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"pascal-voc-results","text":"PASCAL VOC Results","depth":3},{"id":"nyudv2-results","text":"NYUDv2 Results","depth":3},{"id":"sift-flow-results","text":"SIFT Flow Results","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"feature-fusion/unet","relation":"next","topic":"Dense prediction and encoder-decoder skip connections"}]},{"id":"feature-fusion/unet","category":"feature-fusion","legacyPath":"feature-fusion/1505-unet","legacySlug":"1505-unet","legacyPaths":[],"title":"U-Net","subtitle":"The Dawn of Integration","author":"Z. Yuan","paperDate":"2015-05-18","publishedOn":"2024-08-15","paperTitle":"U-Net: Convolutional Networks for Biomedical Image Segmentation","paperUrl":"https://arxiv.org/abs/1505.04597","bibliography":{"title":"U-Net: Convolutional Networks for Biomedical Image Segmentation","authors":["Olaf Ronneberger","Philipp Fischer","Thomas Brox"],"url":"https://arxiv.org/abs/1505.04597","pdfUrl":"https://arxiv.org/pdf/1505.04597","codeUrl":null,"preprint":{"id":"1505.04597","submittedOn":"2015-05-18","url":"https://arxiv.org/abs/1505.04597"},"publication":{"venue":"MICCAI","year":2015,"kind":"conference","url":"https://doi.org/10.1007/978-3-319-24574-4_28"},"citation":"@misc{featurefusionunet2015,\n  title = {{U-Net: Convolutional Networks for Biomedical Image Segmentation}},\n  author = {Olaf Ronneberger and Philipp Fischer and Thomas Brox},\n  year = {2015},\n  eprint = {1505.04597},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1505.04597}\n}\n","citationPath":"/content/papers/citations/feature-fusion/unet.b015405a1c46.bib"},"thumbnail":{"src":"/content/papers/assets/0b302bf8ed079d85b3a07562463380875eb3d9dfc631684549000c38d16a5f38.jpg","width":1224,"height":824},"locale":"en","path":"/content/papers/en/feature-fusion/unet.cd213004394b.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"discussion","text":"Discussion","depth":2},{"id":"isbi-cell-tracking-challenge-2015","text":"ISBI Cell Tracking Challenge 2015","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"segmentation/fcn","relation":"background","topic":"Dense prediction and encoder-decoder skip connections"},{"id":"feature-fusion/unetpp","relation":"next","topic":"Dense prediction and encoder-decoder skip connections"},{"id":"feature-fusion/trans-unet","relation":"comparison","topic":"Encoder-decoder features across scales"}]}],"text":"This is one of the most influential papers in deep learning, especially for image segmentation, and it's definitely a must-read.\n\n## Defining the Problem\n\nImage segmentation has been an evolving field for several years. With the rise of convolutional networks (CNNs), more and more approaches have shifted towards using CNNs to solve the segmentation problem. Traditional methods would often break the system into separate stages, making the whole process disjointed and heavily reliant on manual design.\n\nFully convolutional computation had precedents. This paper adapted pretrained classification networks into end-to-end pixelwise segmentation models, combining in-network upsampling and skip fusion without separate region-proposal or refinement stages.\n\n## Solving the Problem\n\n### Model Architecture\n\n![modify predict head](/content/papers/assets/3587e78aedea003ec99142a99bb30bbfcd31b25cd0ef278d59402a289c66b087.jpg)\n\nAt the time, some of the most popular CNN architectures were AlexNet, VGG16, and GoogLeNet, mainly used for image classification tasks. To adapt these models for image segmentation, the authors modified the final \"fully connected layers,\" replacing them with convolutional layers, as shown in the diagram above.\n\nWhen using fully connected layers, the input image size is restricted, because although convolutional layers can handle images of arbitrary sizes, fully connected layers cannot. The authors solved this issue by replacing fully connected layers with convolutional ones. This permits variable spatial input sizes, subject to minimum-size, padding and output-alignment requirements.\n\nThe equivalent convolution depends on the full input shape. A fully connected layer mapping $C\\times H\\times W$ inputs to 4096 units becomes an $H\\times W$ convolution with 4096 output channels. It is a 1×1 convolution only when the original spatial input is 1×1. For example, VGG16 fc6 becomes a 7×7 convolution and fc7 becomes a 1×1 convolution; kernel size is not an arbitrary choice.\n\nThe authors use a 1×1 convolution to produce a score map for each class, then upsample those maps to the input image size. Upsampling restores spatial resolution and alignment.\n\nIn this paper, the final output contains 21 channels because they used the PASCAL VOC 2011 dataset, which has 20 classes plus one for the background, totaling 21 classes.\n\n### Upsampling\n\nIn the paper, the authors mention a technique called **Shift-and-Stitch**, which allows dense predictions from coarse outputs without interpolation. This method was originally introduced in OverFeat:\n\n- [**[13.12] Overfeat: Integrated recognition, localization and detection using convolutional networks**](https://arxiv.org/abs/1312.6229)\n\nWhen the output is downsampled by a factor of $f$, the input can be shifted by $x$ pixels to the right and $y$ pixels down, and predictions are made for each pair $(x, y) \\in \\{0, \\dots, f-1\\} \\times \\{0, \\dots, f-1\\}$. The $f^2$ input sets are processed through the network, and the outputs are interleaved to align predictions with the center of the receptive field of each pixel.\n\nAlthough the authors conducted preliminary experiments with Shift-and-Stitch, they ultimately chose **deconvolution** (transposed convolution, not the mathematical inverse of convolution) for upsampling, finding it more efficient, especially when combined with the skip connection technique mentioned later.\n\nThe complete network architecture is shown below:\n\n![fcn arch](/content/papers/assets/1a8a4d2936086814cc5bd26819653959fe4cc29c64c5cba2009f8b8e43c87842.jpg)\n\n### Multi-Scale Fusion\n\n![multi-scale](/content/papers/assets/fa92fb0b6367ed1945047cfa0bf617325b09e7b0c8d88df3b62e661fd2419980.jpg)\n\nUsing only the deepest, lowest-resolution features results in coarse outputs when scaled back to the original image size. To improve this, the authors proposed **multi-scale fusion**.\n\nAs shown above, features from different resolutions were first projected to class-score maps with 1×1 convolutions, then aligned and summed. The smallest feature map is $1/32$ of the original size, and the largest is $1/8$. The fusion process involves upsampling the $1/32$ feature map to match the $1/16$ feature map, then summing them, followed by upsampling the result to match the $1/8$ feature map and summing them as well, before upsampling to the original size.\n\nThis technique is referred to as a \"skip layer,\" and it helps the model achieve better results. Below is a comparison:\n\n![multi-scale](/content/papers/assets/e3e1a52a66fadf3bba12585d10d863d3eb2214036f6db495b84b2f807dec14d6.jpg)\n\nThe left image shows FCN-32s directly upsampled to the input size. The two middle images show FCN-16s after adding pool4 scores, then FCN-8s after also adding pool3 scores. The right image is the annotation. The fused predictions retain more detail.\n\n\n> [!TIP]\n> \n> This anticipates aspects of Feature Pyramid Networks (FPN)! The distinction is that FCN fuses class-score maps for a single segmentation output, while FPN constructs a multilevel feature pyramid.\n\n\n### Training Dataset and Strategy\n\nThe authors used the PASCAL VOC 2011 Segmentation Challenge dataset for training and validation. They trained using per-pixel multinomial logistic loss, and the evaluation metric was mean intersection-over-union (mean IU). All models were trained and tested using the Caffe framework on a single NVIDIA Tesla K40c.\n\n1. **Learning Rate**: The model was trained with stochastic gradient descent (SGD) with momentum set to 0.9, and weight decay set to $5 \\times 10^{-4}$ or $2 \\times 10^{-4}$. The learning rates were set to $10^{-3}$, $10^{-4}$, and $5 \\times 10^{-5}$ for FCN-AlexNet, FCN-VGG16, and FCN-GoogLeNet, respectively. These values were determined through a linear search. The class-score convolution was zero-initialized; random initialization improved neither performance nor convergence. Dropout was retained where present in the original classification network.\n\n2. **Training Data**: Whole-image training effectively batches overlapping receptive fields without physically cropping each patch. Sampling spatial loss locations did not improve convergence and took longer, so the authors used the full image.\n\n3. **Class Balancing**: Class balancing could be done through weighting or sampling, but results showed no need for balancing, even with some label imbalance (approximately 3/4 of the data was background).\n\n4. **Dense Prediction**: Outputs were upsampled via learned deconvolution layers. The final upsampling filters were fixed to bilinear interpolation; intermediate upsampling filters were initialized as bilinear and then learned. Shift-and-stitch and sparse filters were not used.\n\n5. **Data Augmentation**: Random flipping and shifting (up to 32 pixels) were tried, but no significant improvements were observed.\n\n6. **Additional Training Data**: Initially, 1,112 images from PASCAL VOC 2011 were used for training, but later, the authors used the larger set of 8,498 annotated PASCAL training images collected by Hariharan et al., improving FCN-VGG16's mean IU by 3.4 points to 59.4. This was measured on the non-overlapping subset of 736 validation images, excluding images shared with the expanded training set.\n\n### Evaluation Metrics\n\nThe paper reports four commonly used evaluation metrics from semantic segmentation and scene parsing tasks, including pixel accuracy and variations of Intersection over Union (IoU).\n\nLet $n_{ij}$ count pixels of true class $i$ predicted as $j$, $t_i=\\sum_j n_{ij}$, and $n_{cl}$ be the number of evaluated classes. Accumulate the confusion matrix over the evaluation set and follow its rules for absent classes and ignored pixels. The metrics are:\n\n- **Pixel Accuracy**:\n\n  $$\n  \\frac{\\sum_i n_{ii}}{\\sum_i t_i}\n  $$\n\n  Where $n_{ii}$ represents the number of pixels correctly predicted for class $i$ (i.e., the number of pixels that belong to class $i$ and are predicted as class $i$), and $\\sum_i t_i$ denotes the total number of pixels in the image.\n\n  Pixel accuracy is the most basic evaluation metric. It directly reflects the proportion of correctly predicted pixels across the entire image. However, this metric can obscure prediction errors for smaller objects when certain categories (like background) dominate the pixel count in the dataset.\n\n  ***\n\n- **Mean Accuracy**:\n\n  $$\n  \\frac{1}{n_{cl}} \\sum_i \\frac{n_{ii}}{t_i}\n  $$\n\n  Here, $t_i$ represents the total number of pixels that belong to class $i$, and $n_{ii}$ is the number of pixels correctly predicted as class $i$.\n\n  Mean accuracy considers the prediction performance of each class and averages the accuracy across all classes. The advantage of this metric is that it evaluates the prediction performance for each class fairly, even for classes with fewer pixels. It is less affected by class imbalance and better reflects the consistency of the model's performance across different classes.\n\n  ***\n\n- **Mean Intersection over Union (Mean IoU)**:\n\n  $$\n   \\frac{1}{n_{cl}} \\sum_i \\frac{n_{ii}}{t_i + \\sum_j n_{ji} - n_{ii}}\n  $$\n\n  The numerator $n_{ii}$ represents the number of correctly predicted pixels, while the denominator $t_i + \\sum_j n_{ji} - n_{ii}$ includes both the correctly predicted pixels and the incorrectly predicted pixels, forming the union set for the class.\n\n  Mean IoU measures the classification performance by evaluating the overlap between predicted and actual class regions. It calculates the IoU for each class and averages the results to reflect the overall performance of the model. Compared to pixel accuracy, Mean IoU is more balanced in handling class imbalance, distinguishing between small and large objects in predictions.\n\n  ***\n\n- **Frequency Weighted Intersection over Union (Frequency Weighted IoU)**:\n\n  $$\n  \\frac{1}{\\sum_k t_k} \\sum_i \\frac{t_i n_{ii}}{t_i + \\sum_j n_{ji} - n_{ii}}\n  $$\n\n  Frequency Weighted IoU considers the frequency of each class and uses the total number of pixels for each class $t_i$ as a weight to calculate the weighted average IoU for each class. The denominator $\\sum_k t_k$ is the total number of pixels in the dataset, and each term $t_i n_{ii}$ represents the number of correctly predicted pixels for a given class multiplied by its total pixel count. This emphasizes classes with larger pixel counts (like background).\n\n  This metric not only evaluates the prediction accuracy for each class but also weights the classes based on their proportions in the dataset. This gives more influence to frequent classes such as background; it does not remove class-imbalance bias. Frequency Weighted IoU is particularly sensitive to performance in real-world applications, where common classes are more influential.\n\n## Discussion\n\n### PASCAL VOC Results\n\n![pascal voc](/content/papers/assets/593340ef7aa7c5fc59400b0932e7b3f05f9001bd02a0bcc9fee3d4418ba0df38.jpg)\n\nThe table above shows the performance of FCN-8s on the PASCAL VOC 2011 and 2012 test sets, compared to previous state-of-the-art systems SDS and the well-known R-CNN.\n\nOn VOC2012, mean IoU rises from SDS’s 51.6 to FCN-8s’s 62.2: 10.6 percentage points, or about 20% relative improvement. Inference time has also significantly reduced, with the convolutional network computation being 114 times faster, and overall throughput about 286 times higher (roughly 50 s versus 175 ms per image). These are comparisons of the paper’s implementations, not universal hardware-independent ratios.\n\n### NYUDv2 Results\n\n![nyudv2](/content/papers/assets/a778ed33b698e98bb3693786bf67d6d8b47e46ff7e5c223569e04882f33af2b7.jpg)\n\nInitially, the authors trained the unmodified coarse model (FCN-32s) using RGB images, then incorporated depth information to upgrade the model to handle four-channel RGB-D input (early fusion), but the improvements were limited.\n\nBuilding on the work by Gupta et al., the authors employed HHA depth encoding and performed \"late fusion\" of RGB and HHA, eventually upgrading to the 16-stride version.\n\n\n> [!TIP]\n> \n> The NYUDv2 dataset consists of 1,449 RGB-D images, each with pixel-level annotations, consolidated into a 40-class semantic segmentation task.\n\n\n### SIFT Flow Results\n\n![sift flow](/content/papers/assets/cf0b23e62bdcd7a1b92b3cc7cee1ffad34f03b00b9af6c0b9595eabcbb374c4c.jpg)\n\nThe authors trained a dual-headed FCN-16s model that simultaneously predicted semantic and geometric labels. The model's performance on both tasks was comparable to models trained separately, but with nearly the same learning and inference speed as a single model.\n\nThe table above shows the results on the standard data split (2,488 training images and 200 test images), achieving state-of-the-art performance in both tasks at publication time.\n\n\n> [!TIP]\n> \n> The SIFT Flow dataset consists of 2,688 images with 33 semantic categories (such as \"bridge,\" \"mountain,\" \"sun\") and three geometric categories (\"horizontal,\" \"vertical,\" \"sky\").\n\n\n## Conclusion\n\nThe introduction of FCN represents a milestone contribution, extending the advantages of convolutional networks to semantic segmentation tasks and laying the groundwork for many subsequent image segmentation techniques. Its innovations, such as skip connections and a fully convolutional architecture, make semantic segmentation feasible as an end-to-end task.\n\nWhile there is still room for improvement in output precision, resource consumption, and multi-scale feature fusion, the ideas and techniques proposed by FCN have significant implications for future research in image segmentation.\n","imageSizes":{"/content/papers/assets/3587e78aedea003ec99142a99bb30bbfcd31b25cd0ef278d59402a289c66b087.jpg":{"width":1224,"height":716},"/content/papers/assets/1a8a4d2936086814cc5bd26819653959fe4cc29c64c5cba2009f8b8e43c87842.jpg":{"width":1224,"height":684},"/content/papers/assets/fa92fb0b6367ed1945047cfa0bf617325b09e7b0c8d88df3b62e661fd2419980.jpg":{"width":2020,"height":478},"/content/papers/assets/e3e1a52a66fadf3bba12585d10d863d3eb2214036f6db495b84b2f807dec14d6.jpg":{"width":1148,"height":496},"/content/papers/assets/593340ef7aa7c5fc59400b0932e7b3f05f9001bd02a0bcc9fee3d4418ba0df38.jpg":{"width":1224,"height":294},"/content/papers/assets/a778ed33b698e98bb3693786bf67d6d8b47e46ff7e5c223569e04882f33af2b7.jpg":{"width":1224,"height":520},"/content/papers/assets/cf0b23e62bdcd7a1b92b3cc7cee1ffad34f03b00b9af6c0b9595eabcbb374c4c.jpg":{"width":1224,"height":600}}}
