{"notes":[{"id":"feature-fusion/hourglass","category":"feature-fusion","legacyPath":"feature-fusion/1603-hourglass","legacySlug":"1603-hourglass","legacyPaths":[],"title":"Hourglass","subtitle":"The Forgotten Elder","author":"Z. Yuan","paperDate":"2016-03-22","publishedOn":"2024-04-06","paperTitle":"Stacked Hourglass Networks for Human Pose Estimation","paperUrl":"https://arxiv.org/abs/1603.06937","bibliography":{"title":"Stacked Hourglass Networks for Human Pose Estimation","authors":["Alejandro Newell","Kaiyu Yang","Jia Deng"],"url":"https://arxiv.org/abs/1603.06937","pdfUrl":"https://arxiv.org/pdf/1603.06937","codeUrl":null,"preprint":{"id":"1603.06937","submittedOn":"2016-03-22","url":"https://arxiv.org/abs/1603.06937"},"publication":{"venue":"ECCV","year":2016,"kind":"conference","url":"https://doi.org/10.1007/978-3-319-46484-8_29"},"citation":"@misc{featurefusionhourglass2016,\n  title = {{Stacked Hourglass Networks for Human Pose Estimation}},\n  author = {Alejandro Newell and Kaiyu Yang and Jia Deng},\n  year = {2016},\n  eprint = {1603.06937},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1603.06937}\n}\n","citationPath":"/content/papers/citations/feature-fusion/hourglass.300cfcc3e581.bib"},"thumbnail":{"src":"/content/papers/assets/ad4d473578cc1a03387cd17083539bba81f4eea08875d248c057e128cd1fce71.jpg","width":1024,"height":468},"locale":"en","path":"/content/papers/en/feature-fusion/hourglass.f532443524ef.md","headings":[{"id":"model-architecture","text":"Model Architecture","depth":2},{"id":"the-origin-of-hourglass","text":"The Origin of Hourglass?","depth":2},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"feature-fusion/trans-unet","relation":"comparison","topic":"Encoder-decoder features across scales"},{"id":"feature-fusion/unet","relation":"comparison","topic":"Encoder-decoder features across scales"}]},{"id":"feature-fusion/trans-unet","category":"feature-fusion","legacyPath":"feature-fusion/2102-trans-unet","legacySlug":"2102-trans-unet","legacyPaths":[],"title":"TransUNet","subtitle":"Convolutions Are Not Enough","author":"Z. Yuan","paperDate":"2021-02-08","publishedOn":"2025-02-06","paperTitle":"TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation","paperUrl":"https://arxiv.org/abs/2102.04306","bibliography":{"title":"TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation","authors":["Jieneng Chen","Yongyi Lu","Qihang Yu","Xiangde Luo","Ehsan Adeli","Yan Wang","Le Lu","Alan L. Yuille","Yuyin Zhou"],"url":"https://arxiv.org/abs/2102.04306","pdfUrl":"https://arxiv.org/pdf/2102.04306","codeUrl":null,"preprint":{"id":"2102.04306","submittedOn":"2021-02-08","url":"https://arxiv.org/abs/2102.04306"},"publication":null,"citation":"@misc{featurefusiontransunet2021,\n  title = {{TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation}},\n  author = {Jieneng Chen and Yongyi Lu and Qihang Yu and Xiangde Luo and Ehsan Adeli and Yan Wang and Le Lu and Alan L. Yuille and Yuyin Zhou},\n  year = {2021},\n  eprint = {2102.04306},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2102.04306}\n}\n","citationPath":"/content/papers/citations/feature-fusion/trans-unet.a4dac5dc037b.bib"},"thumbnail":{"src":"/content/papers/assets/14209e3562b4a87f468b64e95c4cdb775196c8c6b7bca43c935875959165e49e.jpg","width":1366,"height":678},"locale":"en","path":"/content/papers/en/feature-fusion/trans-unet.40fcdd669141.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"discussion","text":"Discussion","depth":2},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"feature-fusion/unet","relation":"comparison","topic":"Encoder-decoder features across scales"},{"id":"feature-fusion/hourglass","relation":"comparison","topic":"Encoder-decoder features across scales"}]},{"id":"feature-fusion/unet","category":"feature-fusion","legacyPath":"feature-fusion/1505-unet","legacySlug":"1505-unet","legacyPaths":[],"title":"U-Net","subtitle":"The Dawn of Integration","author":"Z. Yuan","paperDate":"2015-05-18","publishedOn":"2024-08-15","paperTitle":"U-Net: Convolutional Networks for Biomedical Image Segmentation","paperUrl":"https://arxiv.org/abs/1505.04597","bibliography":{"title":"U-Net: Convolutional Networks for Biomedical Image Segmentation","authors":["Olaf Ronneberger","Philipp Fischer","Thomas Brox"],"url":"https://arxiv.org/abs/1505.04597","pdfUrl":"https://arxiv.org/pdf/1505.04597","codeUrl":null,"preprint":{"id":"1505.04597","submittedOn":"2015-05-18","url":"https://arxiv.org/abs/1505.04597"},"publication":{"venue":"MICCAI","year":2015,"kind":"conference","url":"https://doi.org/10.1007/978-3-319-24574-4_28"},"citation":"@misc{featurefusionunet2015,\n  title = {{U-Net: Convolutional Networks for Biomedical Image Segmentation}},\n  author = {Olaf Ronneberger and Philipp Fischer and Thomas Brox},\n  year = {2015},\n  eprint = {1505.04597},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1505.04597}\n}\n","citationPath":"/content/papers/citations/feature-fusion/unet.b015405a1c46.bib"},"thumbnail":{"src":"/content/papers/assets/0b302bf8ed079d85b3a07562463380875eb3d9dfc631684549000c38d16a5f38.jpg","width":1224,"height":824},"locale":"en","path":"/content/papers/en/feature-fusion/unet.cd213004394b.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"discussion","text":"Discussion","depth":2},{"id":"isbi-cell-tracking-challenge-2015","text":"ISBI Cell Tracking Challenge 2015","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"segmentation/fcn","relation":"background","topic":"Dense prediction and encoder-decoder skip connections"},{"id":"feature-fusion/unetpp","relation":"next","topic":"Dense prediction and encoder-decoder skip connections"},{"id":"feature-fusion/trans-unet","relation":"comparison","topic":"Encoder-decoder features across scales"}]}],"text":"If you are a developer who frequently explores various papers, the Hourglass architecture may not be entirely unfamiliar to you. Let's quickly introduce this architecture:\n\nThe Hourglass architecture is designed specifically for human pose estimation tasks. By comprehensively processing and integrating features at all scales, it aims to optimally capture various spatial relationships related to the human body. The research team demonstrated that by repeatedly applying bottom-up and top-down processing and incorporating intermediate supervision, the network's performance could be significantly enhanced. Therefore, they named this new architecture the \"Stacked Hourglass\" network. This network, based on successive pooling and upsampling steps, is used to generate the final prediction.\n\nThis paper was published more than six months before the FPN paper, but why is there such a significant difference in their fame? The most intuitive indicator is the number of citations. In August 2023, a rough count put FPN at 20,816 and Hourglass at 5,365. The counting source is unavailable, so treat these as a historical observation rather than current figures. How can two architectures with similar designs have such a vast difference in citations?\n\n- **Wait, you said they are similar?**\n\n## Model Architecture\n\nWe have discussed FPN before, and you are likely familiar with its architecture. Now, let's look at the Hourglass architecture:\n\n![Figure 3 from the Hourglass paper: one module's downsampling, same-resolution skip paths, and upsampling](/content/papers/assets/ad4d473578cc1a03387cd17083539bba81f4eea08875d248c057e128cd1fce71.jpg)\n\nThis is **one module** from Figure 3, not the whole stacked network. The full architecture connects multiple such modules and applies intermediate supervision.\n\nAt first glance, they seem like entirely different architectures, but let's walk through the Hourglass model according to the paper's description:\n\n1. **Capturing Information at Different Scales**\n\n   Imagine observing an image with a person's full body. To accurately understand the person's pose, we need to focus on both the entire body and detailed parts like the face and hands. These pieces of information may be at different scales. The Hourglass design aims to capture these different scales simultaneously.\n\n2. **Design of the Hourglass Structure**\n\n   This model's structure resembles an hourglass. It consists of a series of convolutional layers (for feature extraction) and max-pooling layers (to reduce image resolution). Skip branches at each resolution preserve some spatial details; pooling itself still discards information.\n\n3. **Upsampling and Feature Combination**\n\n   After processing the lower resolutions, the network upsamples with nearest-neighbor interpolation, then adds features elementwise from the skip branch at the same resolution. This brings global body cues and local details together.\n\n4. **Final Prediction**\n\n   Once the module returns to its output resolution, two consecutive 1×1 convolutions produce heatmaps for joints such as elbows and knees. That output head is omitted from the single-module diagram above; it is distinct from the 1×1 convolutions inside residual bottlenecks.\n\nThe last part is specific to pose estimation, so let's set it aside for our architectural comparison.\n\nLet's redraw the model architecture:\n\n![Annotated copy of Hourglass Figure 3, with a green Backbone box and red P1–P5 labels added for this note](/content/papers/assets/d82447fa497ca9d71e7708093b719b04d9166ffe8ff3b3ef6fd1eabb5332e827.jpg)\n\nThe green box is this note's illustration of the backbone, not a division labeled by the original paper. Downsampling here involves residual modules and max pooling. Upsampling then restores resolution stage by stage and adds the corresponding skip features. The original Hourglass largely keeps the same channel count internally; 1×1 convolutions also appear in residual bottlenecks. This annotated image simplifies the information flow.\n\nSee it now? In their upsampling and addition of same-resolution features, Hourglass and FPN really do look alike! Sharing that operation does not make their full architectures identical.\n\nHourglass stacks complete, relatively symmetric encoder–decoder modules and supervises intermediate joint heatmaps at the highest output resolution of each module (64×64 in the paper). FPN builds multi-scale outputs on an existing backbone, including P2–P5 in the original paper and P6 for the RPN, for tasks such as detection. P1 in the redrawing is an illustrative label, not a level convention shared by both papers.\n\nSo why is there such a discrepancy in citations between these two papers?\n\nWe can only speculate: object detection is more popular, which could be one reason. Another potential reason is the narrative structure of the papers. The Hourglass paper mainly emphasizes \"stacking many layers\" and \"intermediate supervision.\"\n\n## The Origin of Hourglass?\n\nA closer look at the Hourglass paper reveals an earlier reference with a similar structure:\n\n**[Bottom-Up and Top-Down Reasoning with Hierarchical Rectified Gaussians (July 2015)](https://arxiv.org/abs/1507.05699)**\n\nThis earlier paper doesn't refer to the architecture as \"Hourglass,\" but rather as a \"Bottom-Up\" and \"Top-Down\" structure. Here is a paraphrase of its abstract, not a verbatim quotation:\n\n> The authors model nonnegative latent variables with a hierarchical Rectified Gaussian model. A quadratic energy describes interactions between levels. Inference becomes a quadratic program, implemented through iterative updates in a recurrent network with ReLUs, allowing bottom-up evidence and top-down feedback to influence the result.\n\nWait, what does that mean?\n\nTo simplify:\n\nIn plain terms, local evidence and a broader interpretation of the pose repeatedly refine one another. “Rectified” refers to the nonnegative constraint on the variables, not to fine-tuning neurons.\n\nIf it's still unclear, don't worry; it's not the main focus here. We're looking for an earlier related architecture. Below is their QP2 pose-estimation model (Figure 4 in that paper):\n\n![QP2 pose-estimation architecture from Hu and Ramanan's paper, showing hierarchical latent variables and joint output](/content/papers/assets/f991a2dc780690258c586d4bdd650a79660949d55066b892cda1030224268298.jpg)\n\nSeeing this familiar structure, one might agree that it is a related predecessor explicitly discussed in the Hourglass paper. Its bottom-up and top-down iterations share weights, however, so it is not the same module. But is it the earliest such design?\n\nNot necessarily.\n\nAbout two months before that July 2015 preprint:\n\n- **[U-Net: Convolutional Networks for Biomedical Image Segmentation](https://arxiv.org/abs/1505.04597)**\n\nwas proposed. Its multiscale connections look similar. One clear difference is U-Net's use of concatenation; the modules, channel configuration, outputs, and supervision differ too. The Hourglass paper does not cite U-Net.\n\n## Conclusion\n\nHourglass and FPN share top-down multiscale fusion and skip connections. Recognizing that common idea is useful; treating the architectures as identical would obscure differences in module stacking, outputs, and supervision.\n\nOn the same foundational architecture, different application processes have evolved to solve different problems. FPN handles multi-scale information through a feature pyramid, mainly used for object detection and segmentation. In contrast, Hourglass networks extract features at various detail levels, particularly suitable for dense prediction tasks like pose estimation.\n\nIf you've ever been confused about the fundamental nature of these two architectures, hopefully, this article has provided some clarity.\n","imageSizes":{"/content/papers/assets/ad4d473578cc1a03387cd17083539bba81f4eea08875d248c057e128cd1fce71.jpg":{"width":1024,"height":468},"/content/papers/assets/d82447fa497ca9d71e7708093b719b04d9166ffe8ff3b3ef6fd1eabb5332e827.jpg":{"width":1024,"height":561},"/content/papers/assets/f991a2dc780690258c586d4bdd650a79660949d55066b892cda1030224268298.jpg":{"width":1024,"height":351}}}
