{"notes":[{"id":"classic-cnns/fcdm","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"FCDM","subtitle":"只能用 Transformer 吗？","author":"Z. Yuan","paperDate":"2026-03-10","publishedOn":"2026-04-07","paperTitle":"Reviving ConvNeXt for Efficient Convolutional Diffusion Models","paperUrl":"https://arxiv.org/abs/2603.09408","bibliography":{"title":"Reviving ConvNeXt for Efficient Convolutional Diffusion Models","authors":["Taesung Kwon","Lorenzo Bianchi","Lennart Wittke","Felix Watine","Fabio Carrara","Jong Chul Ye","Romann Weber","Vinicius Azevedo"],"url":"https://arxiv.org/abs/2603.09408","pdfUrl":"https://arxiv.org/pdf/2603.09408v1","codeUrl":"https://github.com/star-kwon/FCDM","preprint":{"id":"2603.09408","submittedOn":"2026-03-10","url":"https://arxiv.org/abs/2603.09408"},"publication":{"venue":"CVPR","year":2026,"kind":"conference","url":"https://openaccess.thecvf.com/content/CVPR2026/html/Kwon_Reviving_ConvNeXt_for_Efficient_Convolutional_Diffusion_Models_CVPR_2026_paper.html"},"citation":"@misc{classiccnnsfcdm2026,\n  title = {{Reviving ConvNeXt for Efficient Convolutional Diffusion Models}},\n  author = {Taesung Kwon and Lorenzo Bianchi and Lennart Wittke and Felix Watine and Fabio Carrara and Jong Chul Ye and Romann Weber and Vinicius Azevedo},\n  year = {2026},\n  eprint = {2603.09408},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2603.09408}\n}\n","citationPath":"/content/papers/citations/classic-cnns/fcdm.7438c4ca3414.bib"},"thumbnail":{"src":"/content/papers/assets/2a98fdd408b56d724e2b57d3c4a4a2e3c8f3d1ffc8384c400f1ffceff0f84aad.png","width":1960,"height":1280},"locale":"zh-hans","path":"/content/papers/zh-hans/classic-cnns/fcdm.958c40268623.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"扩散模型的骨干在做什么","text":"扩散模型的骨干在做什么","depth":3},{"id":"卷积能不能也这样放大","text":"卷积能不能也这样放大","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"fcdm-区块","text":"FCDM 区块","depth":3},{"id":"u-形架构只有两个旋钮","text":"U 形架构，只有两个旋钮","depth":3},{"id":"和-dico-差在哪","text":"和 DiCo 差在哪","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"实验设置","text":"实验设置","depth":3},{"id":"每个尺寸都赢-dit","text":"每个尺寸都赢 DiT","depth":3},{"id":"用上-guidance-之后","text":"用上 guidance 之后","depth":3},{"id":"分辨率加倍","text":"分辨率加倍","depth":3},{"id":"消融","text":"消融","depth":3},{"id":"四张显卡","text":"四张显卡","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/convnext","relation":"referenced","topic":""},{"id":"classic-cnns/convnext-v2","relation":"referenced","topic":""},{"id":"feature-fusion/unet","relation":"referenced","topic":""}]},{"id":"classic-cnns/convnext","category":"classic-cnns","legacyPath":"classic-cnns/2201-convnext","legacySlug":"2201-convnext","legacyPaths":[],"title":"ConvNeXt","subtitle":"让卷积再次伟大","author":"Z. Yuan","paperDate":"2022-01-10","publishedOn":"2024-06-30","paperTitle":"A ConvNet for the 2020s","paperUrl":"https://arxiv.org/abs/2201.03545","bibliography":{"title":"A ConvNet for the 2020s","authors":["Zhuang Liu","Hanzi Mao","Chao-Yuan Wu","Christoph Feichtenhofer","Trevor Darrell","Saining Xie"],"url":"https://arxiv.org/abs/2201.03545","pdfUrl":"https://arxiv.org/pdf/2201.03545","codeUrl":"https://github.com/facebookresearch/ConvNeXt","preprint":{"id":"2201.03545","submittedOn":"2022-01-10","url":"https://arxiv.org/abs/2201.03545"},"publication":{"venue":"CVPR","year":2022,"kind":"conference","url":"https://arxiv.org/abs/2201.03545"},"citation":"@misc{classiccnnsconvnext2022,\n  title = {{A ConvNet for the 2020s}},\n  author = {Zhuang Liu and Hanzi Mao and Chao-Yuan Wu and Christoph Feichtenhofer and Trevor Darrell and Saining Xie},\n  year = {2022},\n  eprint = {2201.03545},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2201.03545}\n}\n","citationPath":"/content/papers/citations/classic-cnns/convnext.801b4060b807.bib"},"thumbnail":{"src":"/content/papers/assets/37882616c5c72df33699003256d14bf7c58c8ea5d1e9e4fb5b1c33268c9a6923.jpg","width":1224,"height":432},"locale":"zh-hans","path":"/content/papers/zh-hans/classic-cnns/convnext.db5ced84300e.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"重新设计网络架构","text":"重新设计网络架构","depth":3},{"id":"引入现代训练技术","text":"引入现代训练技术","depth":3},{"id":"宏观设计","text":"宏观设计","depth":3},{"id":"resnext-ify","text":"ResNeXt-ify","depth":3},{"id":"inverted-bottleneck","text":"Inverted Bottleneck","depth":3},{"id":"大核卷积","text":"大核卷积","depth":3},{"id":"微观设计","text":"微观设计","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"总结模型配置","text":"总结模型配置","depth":3},{"id":"imagenet-上的性能","text":"ImageNet 上的性能","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"vision-transformers/convmixer","relation":"comparison","topic":"用卷积混合图像分块"},{"id":"vision-transformers/caformer","relation":"comparison","topic":"用卷积混合图像分块"},{"id":"reparameterization/replknet","relation":"comparison","topic":"大卷积核与简化骨架"}]},{"id":"classic-cnns/convnext-v2","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"ConvNeXt V2","subtitle":"卷积也会填空","author":"Z. Yuan","paperDate":"2023-01-02","publishedOn":"2025-09-27","world":"external/convnext-v2","paperTitle":"ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders","paperUrl":"https://arxiv.org/abs/2301.00808","bibliography":{"title":"ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders","authors":["Sanghyun Woo","Shoubhik Debnath","Ronghang Hu","Xinlei Chen","Zhuang Liu","In So Kweon","Saining Xie"],"url":"https://arxiv.org/abs/2301.00808","pdfUrl":"https://arxiv.org/pdf/2301.00808v1","codeUrl":null,"preprint":{"id":"2301.00808","submittedOn":"2023-01-02","url":"https://arxiv.org/abs/2301.00808"},"publication":{"venue":"CVPR","year":2023,"kind":"conference","url":"https://openaccess.thecvf.com/content/CVPR2023/html/Woo_ConvNeXt_V2_Co-Designing_and_Scaling_ConvNets_With_Masked_Autoencoders_CVPR_2023_paper.html"},"citation":"@misc{classiccnnsconvnextv22023,\n  title = {{ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders}},\n  author = {Sanghyun Woo and Shoubhik Debnath and Ronghang Hu and Xinlei Chen and Zhuang Liu and In So Kweon and Saining Xie},\n  year = {2023},\n  eprint = {2301.00808},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2301.00808}\n}\n","citationPath":"/content/papers/citations/classic-cnns/convnext-v2.065fc6e2e545.bib"},"thumbnail":{"src":"/content/papers/assets/0af11b7bcef637ee0205fe47b31901b1b775a0336ccab0042e73e8d9643589c0.png","width":1640,"height":600},"locale":"zh-hans","path":"/content/papers/zh-hans/classic-cnns/convnext-v2.e960e9de238c.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"mae-为什么不好套","text":"MAE 为什么不好套","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"fcmae全卷积的-mae","text":"FCMAE：全卷积的 MAE","depth":3},{"id":"只换预训练不够","text":"只换预训练，不够","depth":3},{"id":"特征坍塌","text":"特征坍塌","depth":3},{"id":"grn让通道互相竞争","text":"GRN：让通道互相竞争","depth":3},{"id":"convnext-v2-块","text":"ConvNeXt V2 块","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"架构和预训练要一起改","text":"架构和预训练要一起改","depth":3},{"id":"和-transformer-的-mae-比","text":"和 Transformer 的 MAE 比","depth":3},{"id":"检测与分割","text":"检测与分割","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/convnext","relation":"referenced","topic":""},{"id":"vision-transformers/mae","relation":"referenced","topic":""}]},{"id":"feature-fusion/unet","category":"feature-fusion","legacyPath":"feature-fusion/1505-unet","legacySlug":"1505-unet","legacyPaths":[],"title":"U-Net","subtitle":"融合的序幕","author":"Z. Yuan","paperDate":"2015-05-18","publishedOn":"2024-08-15","paperTitle":"U-Net: Convolutional Networks for Biomedical Image Segmentation","paperUrl":"https://arxiv.org/abs/1505.04597","bibliography":{"title":"U-Net: Convolutional Networks for Biomedical Image Segmentation","authors":["Olaf Ronneberger","Philipp Fischer","Thomas Brox"],"url":"https://arxiv.org/abs/1505.04597","pdfUrl":"https://arxiv.org/pdf/1505.04597","codeUrl":null,"preprint":{"id":"1505.04597","submittedOn":"2015-05-18","url":"https://arxiv.org/abs/1505.04597"},"publication":{"venue":"MICCAI","year":2015,"kind":"conference","url":"https://doi.org/10.1007/978-3-319-24574-4_28"},"citation":"@misc{featurefusionunet2015,\n  title = {{U-Net: Convolutional Networks for Biomedical Image Segmentation}},\n  author = {Olaf Ronneberger and Philipp Fischer and Thomas Brox},\n  year = {2015},\n  eprint = {1505.04597},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1505.04597}\n}\n","citationPath":"/content/papers/citations/feature-fusion/unet.b015405a1c46.bib"},"thumbnail":{"src":"/content/papers/assets/0b302bf8ed079d85b3a07562463380875eb3d9dfc631684549000c38d16a5f38.jpg","width":1224,"height":824},"locale":"zh-hans","path":"/content/papers/zh-hans/feature-fusion/unet.29e27ec799f9.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"讨论","text":"讨论","depth":2},{"id":"isbi-cell-tracking-challenge-2015","text":"ISBI Cell Tracking Challenge 2015","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"segmentation/fcn","relation":"background","topic":"密集预测与编解码器的跳连"},{"id":"feature-fusion/unetpp","relation":"next","topic":"密集预测与编解码器的跳连"},{"id":"feature-fusion/trans-unet","relation":"comparison","topic":"编解码器如何跨尺度传递特征"}]}],"text":"扩散模型的骨干，这几年走了一条很清楚的路。\n\n最早的 DDPM、ADM，还有 Stable Diffusion 背后的潜在扩散模型（LDM），用的都是卷积加注意力的 U-Net。后来的 DiT 把卷积全部拿掉，换成一叠 Transformer 区块，结果放大得很漂亮。后来的 SD3、FLUX 这些文生图模型，骨干也都是 Transformer。\n\n可是分类那边，ConvNeXt 早就证明了卷积网络调得好，可以和 Transformer 打平。那扩散模型为什么不能用纯卷积？\n\n这篇来自 KAIST、苏黎世联邦理工学院（ETH Zürich）和意大利的研究者。他们直接把 ConvNeXt 改成扩散模型的骨干，用 DiT 的训练和评估流程，从小模型一路比到大模型。\n\n\n> [!TIP]\n> \n> ConvNeXt 的设计、ConvNeXt V2 的 GRN，以及 U-Net 的结构，可以先看这几篇：\n> \n> - **[[22.01] ConvNeXt：让卷积再次伟大](/blog/papers/classic-cnns/convnext)**\n> - **[[23.01] ConvNeXt V2：卷积也会填空](/blog/papers/classic-cnns/convnext-v2)**\n> - **[[15.05] U-Net：融合的序幕](/blog/papers/feature-fusion/unet)**\n\n\n## 定义问题\n\n### 扩散模型的骨干在做什么\n\n先交代一下背景。扩散模型从一张纯噪声开始，一步步去噪，最后得到一张图。每一步都要调用一次骨干网络，输入是目前带噪声的图、时间步 $t$ 和条件（这篇用的是 ImageNet 类别 $y$），输出是预测的噪声。\n\n为了省计算，现在多半在潜在空间里做。256×256 的图先用 VAE 压成 32×32×4 的潜在表示，扩散和去噪都在这个小张量上进行，最后再解码回图像。\n\nDiT 把潜在表示切成小块当 token，送进 Transformer。它被广泛采用，是因为只要调区块数、通道数、头数和切块大小几个参数就能放大，时间和类别的条件也用 adaLN-Zero 干净地注入每个区块。缺点是贵，DiT-XL/2 每次前向传播要 118.6 GFLOPs，DiT 最好的那组成绩训练了 700 万步。\n\n### 卷积能不能也这样放大\n\n之前已经有人回头试卷积骨干，例如 DiC 用 3×3 卷积的 U-Net，DiCo 用 3×3 可分离卷积加上一种通道注意力，都比 DiT 省算力。这篇想问的是更直接的版本。ConvNeXt 那套在分类上验证过的设计，原封不动地搬过来，能不能像 DiT 一样好放大，又比 DiT 省？\n\n## 解决问题\n\n### FCDM 区块\n\n[![本文整理的示意图。左边是 FCDM 区块，由上到下是 7×7 depthwise 卷积、不带仿射参数的 LayerNorm、蓝色的「乘上 (1 + scale) 再加 shift」、把通道从 C 放大到 rC 的 1×1 卷积、GELU、GRN、把通道缩回 C 的 1×1 卷积、蓝色的「乘上 gate」，最后和输入相加。右侧一条蓝色的条件分支：类别 y 和时间 t 的嵌入 c，经过 SiLU 和线性层，产生 shift、scale 和 gate，线性层标注初始化为零；下方注明初始化为零时 gate 等于 0，整个区块一开始就是恒等函数。右边是 U 形架构：输入潜在表示经过 L 个区块、C 个通道，降采样两倍后是 2L 个区块、2C 个通道，再降一次是中间的 4L 个区块、4C 个通道，然后对称地升回去，同层之间用虚线的跳接相连。](/content/papers/assets/2a98fdd408b56d724e2b57d3c4a4a2e3c8f3d1ffc8384c400f1ffceff0f84aad.png)](/content/papers/assets/2a98fdd408b56d724e2b57d3c4a4a2e3c8f3d1ffc8384c400f1ffceff0f84aad.png)\n\n*图：本文整理。依原论文第 3.1 节与官方代码绘制。*\n\n区块几乎就是 ConvNeXt V2 的区块。7×7 的 depthwise 卷积，接 LayerNorm，再用两个 1×1 卷积先放大、再缩回，中间夹着 GELU 和 GRN，最后加上残差。官方代码的放大倍数 $r$ 是 3。\n\nConvNeXt 本来是做分类的，没有地方放时间和类别的条件。作者学 DiT 的 adaLN-Zero，让类别和时间的嵌入经过一个小的线性层，产生 shift、scale、gate 三组参数。LayerNorm 拿掉自己的仿射参数，改用 $h\\cdot(1+\\text{scale})+\\text{shift}$；区块输出再乘上 gate，然后才和残差相加。这个线性层初始化为零，所以刚开始训练时 gate 是 0，每个区块都是恒等函数，网络再深也能稳定起步。\n\n我们照这个描述写一个区块，看看初始化和条件的效果：\n\n```python\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\n\nclass GRN(nn.Module):                                  # ConvNeXt V2 的全局响应归一化\n    def __init__(self, c):\n        super().__init__()\n        self.gamma = nn.Parameter(torch.zeros(1, c, 1, 1))\n        self.beta = nn.Parameter(torch.zeros(1, c, 1, 1))\n\n    def forward(self, x):\n        g = x.norm(p=2, dim=(2, 3), keepdim=True)\n        return self.gamma * (x * g / (g.mean(1, keepdim=True) + 1e-6)) + self.beta + x\n\nclass FCDMBlock(nn.Module):\n    def __init__(self, c, r=3):\n        super().__init__()\n        self.dw = nn.Conv2d(c, c, 7, padding=3, groups=c)\n        self.pw1, self.pw2 = nn.Conv2d(c, r * c, 1), nn.Conv2d(r * c, c, 1)\n        self.grn = GRN(r * c)\n        self.ada = nn.Sequential(nn.SiLU(), nn.Linear(c, 3 * c))\n        nn.init.zeros_(self.ada[1].weight)              # 条件分支从 0 开始\n        nn.init.zeros_(self.ada[1].bias)\n\n    def forward(self, x, cond):\n        shift, scale, gate = self.ada(cond)[:, :, None, None].chunk(3, dim=1)\n        h = F.layer_norm(self.dw(x).permute(0, 2, 3, 1), (x.shape[1],)).permute(0, 3, 1, 2)\n        h = h * (1 + scale) + shift                     # 条件决定每个通道怎么缩放、平移\n        h = self.pw2(self.grn(F.gelu(self.pw1(h))))\n        return x + gate * h                             # gate 也由条件决定\n\ntorch.manual_seed(0)\nc = 32\nblock = FCDMBlock(c)\nx = torch.randn(2, c, 16, 16)\ncond_a, cond_b = torch.randn(1, c).expand(2, c), torch.randn(1, c).expand(2, c)   # 两组不同的类别＋时间嵌入\n\nwith torch.no_grad():\n    print(\"刚初始化：输出和输入的最大差距\", f\"{(block(x, cond_a) - x).abs().max():.1e}\")\n    nn.init.normal_(block.ada[1].weight, std=0.1)    # 模拟训练一阵子之后\n    ya, yb = block(x, cond_a), block(x, cond_b)\n    print(\"训练后：同一张输入、不同条件的输出差距\", f\"{(ya - yb).abs().max():.2f}\")\n```\n\n```text\n刚初始化：输出和输入的最大差距 0.0e+00\n训练后：同一张输入、不同条件的输出差距 0.85\n```\n\n刚初始化时，区块完全不改变输入。条件分支的权重长出来之后，同一张输入在不同的类别和时间下，会被处理成不同的样子。条件就是从这三组参数注入的。\n\n### U 形架构，只有两个旋钮\n\n区块排成 U 形。往下走三层，通道数是 $C$、$2C$、$4C$，区块数是 $L$、$2L$、$4L$，每降一次采样就两者都加倍；再对称地升回来，同层用跳接相连。整个架构只有 $L$ 和 $C$ 两个超参数。\n\n为了和 DiT 公平比较，四个尺寸的参数量都对齐 DiT：\n\n| 模型 | $L$ | $C$ | 参数量 | FLOPs | 相对 DiT 的 FLOPs |\n| --- | ---: | ---: | ---: | ---: | ---: |\n| FCDM-S | 2 | 128 | 32.7M | 3.1G | 50.8% |\n| FCDM-B | 2 | 256 | 127.7M | 12.2G | 53.0% |\n| FCDM-L | 2 | 512 | 504.5M | 48.3G | 59.9% |\n| FCDM-XL | 3 | 512 | 698.8M | 64.6G | 54.5% |\n\n*表：摘自原论文 Table 2，FLOPs 在 256×256 分辨率下计算。*\n\n参数量一样，FLOPs 大约只有 DiT 的一半。\n\n### 和 DiCo 差在哪\n\nDiCo 是之前最好的卷积扩散模型，结构和 FCDM 很像，作者花了一节说明差在哪里。\n\nDiCo 的卷积模块从头到尾维持原本的通道数。FCDM 用 ConvNeXt 的倒置瓶颈，先在原本的通道数上做 depthwise 卷积，再放大通道，所以 depthwise 卷积的成本不受放大倍数影响。DiCo 用一个额外的 1×1 卷积做通道注意力，让各通道的响应更多样；FCDM 用 GRN 达到类似的效果，几乎不用额外参数。DiCo 还有一个独立的前馈模块，FCDM 的倒置瓶颈已经做了这件事，就省掉了。\n\n结果是同样参数量下，FCDM 的 FLOPs 大约是 DiCo 的四分之三。\n\n## 讨论\n\n### 实验设置\n\n训练完全照 DiT 的流程。AdamW，学习率固定 $10^{-4}$，没有权重衰减，批量 256，数据增强只有水平翻转，权重取 EMA。评估用 FID，从 5 万张生成图和真实图的特征分布算距离，越低越好。\n\n### 每个尺寸都赢 DiT\n\n先看训练 40 万步、不用 classifier-free guidance 的结果：\n\n| 模型 | FLOPs | 推理吞吐量 | FID |\n| --- | ---: | ---: | ---: |\n| DiT-S/2 | 6.1G | 1234 | 68.4 |\n| FCDM-S | 3.1G | 2687 | **48.5** |\n| DiT-B/2 | 23.0G | 380 | 43.5 |\n| FCDM-B | 12.2G | 1002 | **26.2** |\n| DiT-L/2 | 80.7G | 115 | 23.3 |\n| FCDM-L | 48.3G | 381 | **13.8** |\n| DiT-XL/2 | 118.6G | 81 | 19.5 |\n| FCDM-XL | 64.6G | 273 | **10.7** |\n| DiT-XL/2（700 万步） | 118.6G | 81 | 9.6 |\n| FCDM-XL（100 万步） | 64.6G | 273 | **7.9** |\n\n*表：摘自原论文 Table 1 与 Table 3，吞吐量的单位是论文报告的 it/s。*\n\n同样的步数，FCDM 每个尺寸的 FID 都比 DiT 低得多，推理吞吐量是 DiT 的 2.2 到 3.4 倍。最后两行最能说明收敛速度，FCDM-XL 训练 100 万步，FID 就比训练 700 万步的 DiT-XL/2 还低。\n\n和其他卷积骨干比，差距就小多了。同样 40 万步，DiCo-L 的 FID 是 13.66，比 FCDM-L 的 13.83 还好一点；XL 这一级 FCDM 是 10.72，DiCo-XL 是 11.67，FCDM 的 FLOPs 少了四分之一。小模型的吞吐量则是 DiC 最高，它用的是硬件优化得最好的标准 3×3 卷积。\n\n### 用上 guidance 之后\n\n把 FCDM-XL 训练到 200 万步（400 个 epoch），加上 classifier-free guidance，和各家已发表的结果比：\n\n| 模型 | 训练 epoch | FLOPs | FID | IS |\n| --- | ---: | ---: | ---: | ---: |\n| DiT-XL/2 | 1400 | 118.6G | 2.27 | 278.2 |\n| SiT-XL/2 | 1400 | 118.6G | 2.06 | 277.5 |\n| DiCo-XL | 750 | 87.3G | 2.05 | 282.2 |\n| FCDM-XL | 400 | 64.6G | **2.03** | **285.7** |\n\n*表：摘自原论文 Table 4 的部分行，ImageNet 256×256。*\n\nFID 2.03 比 DiT-XL/2 的 2.27 好，训练 epoch 不到它的三分之一。不过和 SiT、DiCo 比，差距只在小数点后第二位。作者自己也写明，FCDM 在 DiT 这套实验设置下胜过同级对手，但还没超过 EDM-2、Simpler Diffusion 这些最新的顶尖结果。\n\n### 分辨率加倍\n\n在 512×512 上，FCDM-XL 训练 100 万步的 FID 是 7.46，DiT-XL/2 训练 300 万步是 12.03，DiCo-XL 训练 130 万步是 8.10。\n\n更有意思的是速度的变化。分辨率从 256 升到 512，DiT-XL/2 的吞吐量从 80.5 掉到 18.6，少了四倍多；FCDM-XL 从 272.7 掉到 129.6，只少了一半左右。注意力的计算量随 token 数平方增长，卷积只随像素数线性增长，分辨率越高，差距越大。\n\n### 消融\n\n用 FCDM-L 训练 20 万步，每次改一个地方，看 FID：\n\n| 改动 | FID |\n| --- | ---: |\n| FCDM-L（7×7 depthwise、GRN、倒置瓶颈） | **19.97** |\n| depthwise 卷积改成 5×5 | 20.48 |\n| depthwise 卷积改成 3×3 | 21.28 |\n| GRN 换成 DiCo 的通道注意力 | 23.85 |\n| 加上 DiCo 的前馈模块 | 28.52 |\n| 拿掉倒置瓶颈 | 28.76 |\n| 换成 ResNet 区块 | 31.14 |\n\n*表：摘自原论文 Table 6。后四行调整了通道数，让 FLOPs 一致。*\n\n卷积核越小越差，大卷积核在生成任务上一样有用。把 DiCo 的设计搬进来都会变差，拿掉倒置瓶颈或换回 ResNet 区块更是差了一大截。ConvNeXt 这套设计，在生成任务上几乎每一项都还站得住。\n\n### 四张显卡\n\n论文特别强调训练成本。FCDM-XL 在四张 RTX 4090 上，开梯度检查点、批量 256，每秒大约跑 0.9 步；批量 256 也放得进一张 40GB 的 A100。\n\n照这个速度推算，40 万步大约 5 天，200 万步大约 26 天。这是本文根据论文给出的速度所作的估算。对一般实验室来说，这是摸得到的规模。\n\n官方 repo 提供了训练和采样的代码，授权是 CC BY-NC，目前没有附上预训练权重。\n\n## 结论\n\nFCDM 几乎没有发明新东西。ConvNeXt V2 的区块、DiT 的 adaLN-Zero 条件注入、一个只有两个旋钮的 U 形架构，三样都是现成的。组起来之后，FLOPs 只有 DiT 的一半，收敛快了好几倍，分辨率越高优势越明显。\n\n比 2.03 更有说服力的是消融表。把 DiCo 的改良搬进来会变差，把 ConvNeXt 的设计拿掉也会变差，一套为分类设计的区块，在生成任务上几乎原封不动地成立。\n\n所以标题那个问题的答案，至少在 ImageNet 的类别条件生成上，是「不一定」。\n\n文生图就还早。论文附录只在 MS-COCO 上训练了 10 万步、看了几张生成图，条件也只用了池化过的 CLIP 文本向量；要像 SD3 那样接上完整的文本序列，作者自己也列为之后的工作。\n","imageSizes":{"/content/papers/assets/2a98fdd408b56d724e2b57d3c4a4a2e3c8f3d1ffc8384c400f1ffceff0f84aad.png":{"width":1960,"height":1280}}}