{"notes":[{"id":"feature-fusion/trans-unet","category":"feature-fusion","legacyPath":"feature-fusion/2102-trans-unet","legacySlug":"2102-trans-unet","legacyPaths":[],"title":"TransUNet","subtitle":"不能只有卷积","author":"Z. Yuan","paperDate":"2021-02-08","publishedOn":"2025-02-06","paperTitle":"TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation","paperUrl":"https://arxiv.org/abs/2102.04306","bibliography":{"title":"TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation","authors":["Jieneng Chen","Yongyi Lu","Qihang Yu","Xiangde Luo","Ehsan Adeli","Yan Wang","Le Lu","Alan L. Yuille","Yuyin Zhou"],"url":"https://arxiv.org/abs/2102.04306","pdfUrl":"https://arxiv.org/pdf/2102.04306","codeUrl":null,"preprint":{"id":"2102.04306","submittedOn":"2021-02-08","url":"https://arxiv.org/abs/2102.04306"},"publication":null,"citation":"@misc{featurefusiontransunet2021,\n  title = {{TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation}},\n  author = {Jieneng Chen and Yongyi Lu and Qihang Yu and Xiangde Luo and Ehsan Adeli and Yan Wang and Le Lu and Alan L. Yuille and Yuyin Zhou},\n  year = {2021},\n  eprint = {2102.04306},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2102.04306}\n}\n","citationPath":"/content/papers/citations/feature-fusion/trans-unet.a4dac5dc037b.bib"},"thumbnail":{"src":"/content/papers/assets/14209e3562b4a87f468b64e95c4cdb775196c8c6b7bca43c935875959165e49e.jpg","width":1366,"height":678},"locale":"zh-hans","path":"/content/papers/zh-hans/feature-fusion/trans-unet.495195dcefa7.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"讨论","text":"讨论","depth":2},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"feature-fusion/unet","relation":"comparison","topic":"编解码器如何跨尺度传递特征"},{"id":"feature-fusion/hourglass","relation":"comparison","topic":"编解码器如何跨尺度传递特征"}]},{"id":"feature-fusion/unet","category":"feature-fusion","legacyPath":"feature-fusion/1505-unet","legacySlug":"1505-unet","legacyPaths":[],"title":"U-Net","subtitle":"融合的序幕","author":"Z. Yuan","paperDate":"2015-05-18","publishedOn":"2024-08-15","paperTitle":"U-Net: Convolutional Networks for Biomedical Image Segmentation","paperUrl":"https://arxiv.org/abs/1505.04597","bibliography":{"title":"U-Net: Convolutional Networks for Biomedical Image Segmentation","authors":["Olaf Ronneberger","Philipp Fischer","Thomas Brox"],"url":"https://arxiv.org/abs/1505.04597","pdfUrl":"https://arxiv.org/pdf/1505.04597","codeUrl":null,"preprint":{"id":"1505.04597","submittedOn":"2015-05-18","url":"https://arxiv.org/abs/1505.04597"},"publication":{"venue":"MICCAI","year":2015,"kind":"conference","url":"https://doi.org/10.1007/978-3-319-24574-4_28"},"citation":"@misc{featurefusionunet2015,\n  title = {{U-Net: Convolutional Networks for Biomedical Image Segmentation}},\n  author = {Olaf Ronneberger and Philipp Fischer and Thomas Brox},\n  year = {2015},\n  eprint = {1505.04597},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1505.04597}\n}\n","citationPath":"/content/papers/citations/feature-fusion/unet.b015405a1c46.bib"},"thumbnail":{"src":"/content/papers/assets/0b302bf8ed079d85b3a07562463380875eb3d9dfc631684549000c38d16a5f38.jpg","width":1224,"height":824},"locale":"zh-hans","path":"/content/papers/zh-hans/feature-fusion/unet.29e27ec799f9.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"讨论","text":"讨论","depth":2},{"id":"isbi-cell-tracking-challenge-2015","text":"ISBI Cell Tracking Challenge 2015","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"segmentation/fcn","relation":"background","topic":"密集预测与编解码器的跳连"},{"id":"feature-fusion/unetpp","relation":"next","topic":"密集预测与编解码器的跳连"},{"id":"feature-fusion/trans-unet","relation":"comparison","topic":"编解码器如何跨尺度传递特征"}]},{"id":"feature-fusion/hourglass","category":"feature-fusion","legacyPath":"feature-fusion/1603-hourglass","legacySlug":"1603-hourglass","legacyPaths":[],"title":"Hourglass","subtitle":"被遗忘的长者","author":"Z. Yuan","paperDate":"2016-03-22","publishedOn":"2024-04-06","paperTitle":"Stacked Hourglass Networks for Human Pose Estimation","paperUrl":"https://arxiv.org/abs/1603.06937","bibliography":{"title":"Stacked Hourglass Networks for Human Pose Estimation","authors":["Alejandro Newell","Kaiyu Yang","Jia Deng"],"url":"https://arxiv.org/abs/1603.06937","pdfUrl":"https://arxiv.org/pdf/1603.06937","codeUrl":null,"preprint":{"id":"1603.06937","submittedOn":"2016-03-22","url":"https://arxiv.org/abs/1603.06937"},"publication":{"venue":"ECCV","year":2016,"kind":"conference","url":"https://doi.org/10.1007/978-3-319-46484-8_29"},"citation":"@misc{featurefusionhourglass2016,\n  title = {{Stacked Hourglass Networks for Human Pose Estimation}},\n  author = {Alejandro Newell and Kaiyu Yang and Jia Deng},\n  year = {2016},\n  eprint = {1603.06937},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1603.06937}\n}\n","citationPath":"/content/papers/citations/feature-fusion/hourglass.300cfcc3e581.bib"},"thumbnail":{"src":"/content/papers/assets/ad4d473578cc1a03387cd17083539bba81f4eea08875d248c057e128cd1fce71.jpg","width":1024,"height":468},"locale":"zh-hans","path":"/content/papers/zh-hans/feature-fusion/hourglass.9961b5da0e3a.md","headings":[{"id":"模型架构","text":"模型架构","depth":2},{"id":"hourglass-的最初","text":"Hourglass 的最初？","depth":2},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"feature-fusion/trans-unet","relation":"comparison","topic":"编解码器如何跨尺度传递特征"},{"id":"feature-fusion/unet","relation":"comparison","topic":"编解码器如何跨尺度传递特征"}]}],"text":"ViT 是在 2020 年底提出来的，随后相关应用就快速地扩展到大大小小的领域上。\n\n这次来看一篇结合 U-Net 和 ViT 的论文。文章本身不困难，只要你略懂 U-Net 和 ViT 的概念，就能够轻松上手。\n\n\n> [!TIP]\n> \n> 有需要前情提要的读者，可以参考我们之前的文章：\n> \n> - [**[15.05] U-Net: 融合的序幕**](/blog/papers/feature-fusion/unet)\n> - [**[20.10] ViT: 新世界拓荒者**](/blog/papers/vision-transformers/vit)\n\n\n## 定义问题\n\n在 CNN 中引入注意力机制不是什么新鲜事，只是放到医学图像分割领域，目前还很少见。\n\n没人做？那就是最大的问题！\n\n作者希望能像 ViT 一样，以「序列到序列」的思维，把图像切成许多 patches 后直接喂给 Transformer，并借此学到图像的全局特征。\n\n然而，要是我们仅仅用简单的上采样，把 Transformer 产生的低层特征拉回到原始大小（套用 Upsample 之类的），最后只会得到粗糙的分割结果。毕竟 Transformer 虽能擅长处理全局信息，但对于局部边界和细节并没有做特别的强化。\n\n在 ViT 之中，patch size 通常定在 16×16，对于图像分类已经够用。但分割任务可是需要精确到每个像素的位置，也就是说我们必须缩小 patch size 到 8×8，甚至 4×4 才行。可是越小的 patch size，计算量就越高，导致 Transformer 的计算量爆表，根本无法训练。\n\n所以，我们得先处理「多尺度降采样」以及「分辨率」的议题。\n\n## 解决问题\n\n显然地，直接套用 ViT 到图像分割任务上不可行，因此作者提出了混合式的架构，如下图：\n\n\n\n![architecture](/content/papers/assets/14209e3562b4a87f468b64e95c4cdb775196c8c6b7bca43c935875959165e49e.jpg)\n\n\n\n首先使用 CNN 进行特征提取，就像一般我们常见的 Backbone 这样，可以得到 1/2、1/4、1/8 和 1/16 分辨率的特征图。\n\n考量到 Transformer 计算复杂度，这里只将 1/16 分辨率的特征图，搭配 $1 \\times 1$ 的 patch size，把特征图转换成序列，最后通过 Transformer 进行全局特征提取。\n\n当 Transformer 完成自注意力运算后，就能拿到较为「全局」的特征表达，再把序列转回 1/16 分辨率的特征图，最后通过 CNN 的逐级上采样（就像 U-Net 的解码器）来融合其他分辨率的特征图。这样，我们既保留了全局信息的优势，又能通过跳跃连接 (skip connections) 来补足局部细节，最后得到较精细的分割结果。\n\n\n> [!TIP]\n> \n> 这里我们省略 ViT 和 U-Net 的详细运行细节，之前都讲过了。\n\n\n## 讨论\n\n![results](/content/papers/assets/eeb3cea755d0cb5b7fe057cf0918bf1ff93e5afcf3daa3d260a89cd6bf5ad1a9.jpg)\n\n- 上表中的 CUP 指的是 Cascaded Upsampler，用来对比直接上采样的效果。\n\n---\n\n在 **Synapse 多器官分割数据集** 上，作者与四种先前的 SOTA 方法进行比较，包含 V-Net、DARR、U-Net 和 AttnUNet。\n\n此外，为了验证 **CUP**与 **混合编码器** 的性能，实验也对比了以下变体：\n\n- **ViT-None**：ViT 作为编码器，使用简单的上采样进行解码\n- **ViT-CUP**：ViT 作为编码器，搭配 CUP 解码器\n- **R50-ViT-CUP**：ResNet-50 + ViT 作为编码器，搭配 CUP 解码器\n- **TransUNet**：R50-ViT-CUP + U-Net 风格的跳跃连接（Skip Connections）\n\n为了公平比较，U-Net 和 AttnUNet 的编码器也替换为 **ImageNet 预训练的 ResNet-50**，以确保与 ViT-Hybrid 版本一致。\n\n- **直接上采样 vs. CUP**：参考上表中的 **ViT-None vs. ViT-CUP**，可以看到 Dice Similarity Coefficient（DSC）提升了 6.36 个百分点，Hausdorff 距离降低了 3.50 mm。这显示 CUP 解码器在此实验中比简单上采样更能恢复边界细节。\n- **ViT vs. R50-ViT**：参考上表中的 **ViT-CUP vs. R50-ViT-CUP**，DSC 再提升 3.43 个百分点，Hausdorff 距离进一步降低 3.24 mm。结论是纯 ViT 虽能捕捉高层语义，但难以保留医学图像的边界与细节，而混合 CNN + ViT 编码器能有效补足低层特征的缺失。\n\n最后是 TransUNet 对比于其他方法，可以看到 DSC 提升范围在 1.91 ～ 8.67 个百分点之间，相较于最佳 CNN 方法 R50-AttnUNet，TransUNet 仍能额外提升 1.91 个百分点；相比 R50-ViT-CUP，TransUNet 更是提升 6.19 个百分点。\n\n作者分析这是由于单纯的 CNN 虽然能够捕捉丰富的局部细节，但缺乏全局视野；而纯 ViT 虽然能够捕捉全局语意，但难以保留医学图像的边界与细节。而 TransUNet 通过跳跃连接结合全局与局部信息，在此 Synapse 实验中超越表列的其他方法。\n\n## 结论\n\nTransUNet 通过 Transformer 强大的全局自注意力机制，结合 CNN 的低层细节表征，成功突破传统 FCN 架构的限制，在医学图像分割上展现卓越表现。\n\n通过 U 型混合架构，TransUNet 优于现有 CNN 为主的自注意力方法，为医学图像分割提供了一种全新的解决方案，证明了 Transformer 在此领域的潜力与实用性。\n","imageSizes":{"/content/papers/assets/14209e3562b4a87f468b64e95c4cdb775196c8c6b7bca43c935875959165e49e.jpg":{"width":1366,"height":678},"/content/papers/assets/eeb3cea755d0cb5b7fe057cf0918bf1ff93e5afcf3daa3d260a89cd6bf5ad1a9.jpg":{"width":1624,"height":442}}}
