{"notes":[{"id":"object-detection/deformable-detr","category":"object-detection","legacyPath":"object-detection/2010-deformable-detr","legacySlug":"2010-deformable-detr","legacyPaths":[],"title":"Deformable DETR","subtitle":"可变形的注意力机制","author":"Z. Yuan","paperDate":"2020-10-08","publishedOn":"2024-09-20","paperTitle":"Deformable DETR: Deformable Transformers for End-to-End Object Detection","paperUrl":"https://arxiv.org/abs/2010.04159","bibliography":{"title":"Deformable DETR: Deformable Transformers for End-to-End Object Detection","authors":["Xizhou Zhu","Weijie Su","Lewei Lu","Bin Li","Xiaogang Wang","Jifeng Dai"],"url":"https://arxiv.org/abs/2010.04159","pdfUrl":"https://arxiv.org/pdf/2010.04159","codeUrl":null,"preprint":{"id":"2010.04159","submittedOn":"2020-10-08","url":"https://arxiv.org/abs/2010.04159"},"publication":{"venue":"ICLR","year":2021,"kind":"conference","url":"https://arxiv.org/abs/2010.04159"},"citation":"@misc{objectdetectiondeformabledetr2020,\n  title = {{Deformable DETR: Deformable Transformers for End-to-End Object Detection}},\n  author = {Xizhou Zhu and Weijie Su and Lewei Lu and Bin Li and Xiaogang Wang and Jifeng Dai},\n  year = {2020},\n  eprint = {2010.04159},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2010.04159}\n}\n","citationPath":"/content/papers/citations/object-detection/deformable-detr.df721a30472e.bib"},"thumbnail":{"src":"/content/papers/assets/36edfc5595a85c73fcdc9a4bbb9e1ca512e1f100676eb7c37203d71c84905c05.jpg","width":1224,"height":752},"locale":"zh-hans","path":"/content/papers/zh-hans/object-detection/deformable-detr.d2987152d10d.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"可变形注意力","text":"可变形注意力","depth":3},{"id":"多尺度计算","text":"多尺度计算","depth":3},{"id":"模型架构","text":"模型架构","depth":3},{"id":"程式码拆解","text":"程式码拆解","depth":3},{"id":"训练策略","text":"训练策略","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"与-detr-的比较","text":"与 DETR 的比较","depth":3},{"id":"消融实验","text":"消融实验","depth":3},{"id":"sota-比较","text":"SOTA 比较","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"object-detection/detr","relation":"background","topic":"集合预测与注意力检测器"},{"id":"object-detection/h-detr","relation":"next","topic":"集合预测与注意力检测器"},{"id":"object-detection/smca-detr","relation":"comparison","topic":"让 Transformer 检测器更快收敛"}]},{"id":"object-detection/detr","category":"object-detection","legacyPath":"object-detection/2005-detr","legacySlug":"2005-detr","legacyPaths":[],"title":"DETR","subtitle":"跨领域的奠基者","author":"Z. Yuan","paperDate":"2020-05-26","publishedOn":"2024-06-04","paperTitle":"End-to-End Object Detection with Transformers","paperUrl":"https://www.ecva.net/papers/eccv_2020/papers_ECCV/html/832_ECCV_2020_paper.php","bibliography":{"title":"End-to-End Object Detection with Transformers","authors":["Nicolas Carion","Francisco Massa","Gabriel Synnaeve","Nicolas Usunier","Alexander Kirillov","Sergey Zagoruyko"],"url":"https://www.ecva.net/papers/eccv_2020/papers_ECCV/html/832_ECCV_2020_paper.php","pdfUrl":"https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123460205.pdf","codeUrl":"https://github.com/facebookresearch/detr","preprint":{"id":"2005.12872","submittedOn":"2020-05-26","url":"https://arxiv.org/abs/2005.12872"},"publication":{"venue":"ECCV","year":2020,"kind":"conference","url":"https://www.ecva.net/papers/eccv_2020/papers_ECCV/html/832_ECCV_2020_paper.php"},"citation":"@misc{objectdetectiondetr2020,\n  title = {{End-to-End Object Detection with Transformers}},\n  author = {Nicolas Carion and Francisco Massa and Gabriel Synnaeve and Nicolas Usunier and Alexander Kirillov and Sergey Zagoruyko},\n  year = {2020},\n  eprint = {2005.12872},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2005.12872}\n}\n","citationPath":"/content/papers/citations/object-detection/detr.5258ffe0b9a0.bib"},"thumbnail":{"src":"/content/papers/assets/dd7e8441e6460ff562cf558374789924518fafaed7ecde0e3e35ee778c316330.jpg","width":1024,"height":266},"locale":"zh-hans","path":"/content/papers/zh-hans/object-detection/detr.99d0f3b81385.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"模型架构","text":"模型架构","depth":3},{"id":"损失设计","text":"损失设计","depth":3},{"id":"数据集","text":"数据集","depth":3},{"id":"技术细节","text":"技术细节","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"这次行吗","text":"这次行吗？","depth":3},{"id":"模型组件分析","text":"模型组件分析","depth":3},{"id":"模型行为分析","text":"模型行为分析","depth":3},{"id":"对未见过目标数量的泛化能力","text":"对未见过目标数量的泛化能力","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"object-detection/deformable-detr","relation":"next","topic":"集合预测与注意力检测器"},{"id":"object-detection/smca-detr","relation":"comparison","topic":"让 Transformer 检测器更快收敛"}]},{"id":"object-detection/h-detr","category":"object-detection","legacyPath":"object-detection/2207-h-detr","legacySlug":"2207-h-detr","legacyPaths":[],"title":"H-DETR","subtitle":"静默查询的反叛","author":"Z. Yuan","paperDate":"2022-07-26","publishedOn":"2025-07-06","paperTitle":"DETRs with Hybrid Matching","paperUrl":"https://arxiv.org/abs/2207.13080","bibliography":{"title":"DETRs with Hybrid Matching","authors":["Ding Jia","Yuhui Yuan","Haodi He","Xiaopei Wu","Haojun Yu","Weihong Lin","Lei Sun","Chao Zhang","Han Hu"],"url":"https://arxiv.org/abs/2207.13080","pdfUrl":"https://openaccess.thecvf.com/content/CVPR2023/papers/Jia_DETRs_With_Hybrid_Matching_CVPR_2023_paper.pdf","codeUrl":"https://github.com/HDETR/H-Deformable-DETR","preprint":{"id":"2207.13080","submittedOn":"2022-07-26","url":"https://arxiv.org/abs/2207.13080"},"publication":{"venue":"CVPR","year":2023,"kind":"conference","url":"https://openaccess.thecvf.com/content/CVPR2023/html/Jia_DETRs_With_Hybrid_Matching_CVPR_2023_paper.html"},"citation":"@misc{objectdetectionhdetr2022,\n  title = {{DETRs with Hybrid Matching}},\n  author = {Ding Jia and Yuhui Yuan and Haodi He and Xiaopei Wu and Haojun Yu and Weihong Lin and Lei Sun and Chao Zhang and Han Hu},\n  year = {2022},\n  eprint = {2207.13080},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2207.13080}\n}\n","citationPath":"/content/papers/citations/object-detection/h-detr.822184101757.bib"},"thumbnail":{"src":"/content/papers/assets/63f1bc60b92f58d360457c0e3bbef0a3d4b3bd2a07377983b2dc5f5591ce28c2.jpg","width":1784,"height":350},"locale":"zh-hans","path":"/content/papers/zh-hans/object-detection/h-detr.a4dc10ce4de4.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"主要分支一对一监督","text":"主要分支：一对一监督","depth":3},{"id":"辅助分支一对多监督","text":"辅助分支：一对多监督","depth":3},{"id":"总损失与实现细节","text":"总损失与实现细节","depth":3},{"id":"更多形式的混合监督","text":"更多形式的混合监督","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"coco-benchmark","text":"COCO Benchmark","depth":3},{"id":"lvis-benchmark","text":"LVIS Benchmark","depth":3},{"id":"sota-benchmark","text":"SOTA Benchmark","depth":3},{"id":"消融实验","text":"消融实验","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"object-detection/deformable-detr","relation":"background","topic":"集合预测与注意力检测器"}]},{"id":"object-detection/smca-detr","category":"object-detection","legacyPath":"object-detection/2101-smca-detr","legacySlug":"2101-smca-detr","legacyPaths":[],"title":"SMCA DETR","subtitle":"掺点高斯吧！","author":"Z. Yuan","paperDate":"2021-01-19","publishedOn":"2024-09-21","paperTitle":"Fast Convergence of DETR with Spatially Modulated Co-Attention","paperUrl":"https://arxiv.org/abs/2101.07448","bibliography":{"title":"Fast Convergence of DETR with Spatially Modulated Co-Attention","authors":["Peng Gao","Minghang Zheng","Xiaogang Wang","Jifeng Dai","Hongsheng Li"],"url":"https://arxiv.org/abs/2101.07448","pdfUrl":"https://openaccess.thecvf.com/content/ICCV2021/papers/Gao_Fast_Convergence_of_DETR_With_Spatially_Modulated_Co-Attention_ICCV_2021_paper.pdf","codeUrl":"https://github.com/gaopengcuhk/SMCA-DETR","preprint":{"id":"2101.07448","submittedOn":"2021-01-19","url":"https://arxiv.org/abs/2101.07448"},"publication":{"venue":"ICCV","year":2021,"kind":"conference","url":"https://openaccess.thecvf.com/content/ICCV2021/html/Gao_Fast_Convergence_of_DETR_With_Spatially_Modulated_Co-Attention_ICCV_2021_paper.html"},"citation":"@misc{objectdetectionsmcadetr2021,\n  title = {{Fast Convergence of DETR with Spatially Modulated Co-Attention}},\n  author = {Peng Gao and Minghang Zheng and Xiaogang Wang and Jifeng Dai and Hongsheng Li},\n  year = {2021},\n  eprint = {2101.07448},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2101.07448}\n}\n","citationPath":"/content/papers/citations/object-detection/smca-detr.bd7ba32cd46b.bib"},"thumbnail":{"src":"/content/papers/assets/203dc068f589d0931e83b6156954b5b803b20c0d178d5b7c557b1dc6f8a8aa13.jpg","width":1210,"height":740},"locale":"zh-hans","path":"/content/papers/zh-hans/object-detection/smca-detr.0874e23d60bf.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"空间调制共注意力","text":"空间调制共注意力","depth":3},{"id":"初步预测","text":"初步预测","depth":3},{"id":"高斯权重图","text":"高斯权重图","depth":3},{"id":"多头-smca","text":"多头 SMCA","depth":3},{"id":"训练策略","text":"训练策略","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"和-detr-的比较","text":"和 DETR 的比较","depth":3},{"id":"消融实验","text":"消融实验","depth":3},{"id":"head-shared-smca","text":"Head-shared SMCA","depth":3},{"id":"multi-head-smca","text":"Multi-head SMCA","depth":3},{"id":"对比其他-sota-架构","text":"对比其他 SOTA 架构","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"object-detection/detr","relation":"comparison","topic":"让 Transformer 检测器更快收敛"},{"id":"object-detection/deformable-detr","relation":"comparison","topic":"让 Transformer 检测器更快收敛"}]}],"text":"DETR 留给研究者们一个很好发挥的空间。\n\n它只用了最基本的 Transformer 架构，做最朴素的目标检测。里面也没有用到什么提升分数的技巧，只是单纯的把图片丢进去，然后就可以得到目标的位置和类别。\n\n多么美好的论文！\n\n往后的研究者们只要基于这个思路改进模型，论文还不是刷刷刷一大把？\n\n## 定义问题\n\n和常见的目标检测器相比，DETR 的某些配置实在太慢了。\n\n如果按论文表 1 比较 DETR-DC5 与 Faster R-CNN + FPN，推理速度约慢两倍：12 FPS 对 26 FPS！\n\n那又怎样？感觉好像也不是什么大问题？\n\n没错，更大的问题是训练收敛速度：同一组对照的训练耗时为 7000 对 380 GPU 小时，接近 20 倍！\n\n按相同计算资源粗略换算，原本只需 1 天的训练，现在可能接近 20 天，这是多么可怕的事情？\n\n韶光易逝，年华易老。等模型收敛，等到头发都白了。\n\n这不行，肯定要改。\n\n## 解决问题\n\n作者认为问题出在 Transformer 的注意力机制上。\n\nDETR 编码器在卷积特征图的所有空间位置之间计算注意力，特征图分辨率提高时，计算与内存成本会快速增加；初始注意力接近均匀，也需要较长训练才会聚焦。\n\n所以这里不能再使用原本的 Transformer 注意力机制，而是借鉴了「可变形卷积」的思路，将原本的注意力机制改成一种「可变形的注意力机制」。\n\n\n> [!TIP]\n> \n> 这里的注意力作用在 CNN 特征图的位置，不是原始图像的每个像素。\n\n\n### 可变形注意力\n\n\n\n![deformable attention](/content/papers/assets/afd16d6c3cafc26ce43e498bd396a22b1965610ad3fbe96ae88e3668e9f8bdd3.jpg)\n\n\n\n在图像特征图中，给定每个查询元素（Query Element），作者以参考点为基准，从查询特征预测少量采样偏移与权重，再对采样点的特征做加权求和；偏移量没有固定的局部范围限制。\n\n假设输入的特征图为：\n\n$$\nx\n\\in \\mathbb{R}^{C \\times H \\times W}\n$$\n\n其中 $C$ 表示通道数，$H$ 和 $W$ 分别是特征图的高度和宽度。\n\n查询元素 $q$ 包含一个内容特征 $z_q$ 和一个 2D 参考点 $p_q$，可变形注意力特征的计算公式如下：\n\n$$\n\\text{DeformAttn}(z_q, p_q, x) = \\sum_{m=1}^{M} W_m \\sum_{k=1}^{K} A_{mqk} \\cdot W_m^{\\prime} x(p_q + \\Delta p_{mqk})\n$$\n\n其中：\n\n- $M$ 表示注意力头的数量。\n- $K$ 表示每个注意力头对每个查询所采样的点数，远少于整张特征图的位置数。\n- $A_{mqk}$ 是第 $m$ 个注意力头中第 $k$ 个采样点的注意力权重，范围在 $[0, 1]$ 之间；同一头的 $K$ 个权重总和为 1。\n- $\\Delta p_{mqk}$ 是第 $m$ 个注意力头中第 $k$ 个采样点的偏移量，这些偏移量可以是任意的实数。\n- $W_m$ 和 $W_m^{\\prime}$ 是可学习的权重矩阵，负责将输入特征进行线性变换。\n- $x(p_q + \\Delta p_{mqk})$ 表示在位置 $p_q + \\Delta p_{mqk}$ 处的特征值，因为该位置是分数值（即非整数点），因此使用双线性插值来计算。\n\n查询特征 $z_q$ 经过一个线性投影操作，该操作会输出一个 $3MK$ 通道的张量：\n\n- 前 $2MK$ 个通道用来编码每个采样点的偏移量 $\\Delta p_{mqk}$。\n- 剩余的 $MK$ 个通道会经过 softmax 操作，计算出对应的注意力权重 $A_{mqk}$。\n\n这种设计方式，可以保证「偏移量」与「注意力权重」都是从查询元素的特征中学习得到的，而不是基于固定的规则。\n\n### 多尺度计算\n\n现代目标检测框架通常会使用多尺度特征图来进行目标检测。\n\n可变形注意力模块当然也得支持多尺度版本，允许同时在多个特征图层上进行采样和操作。\n\n假设输入的多尺度特征图为 $\\{x_l\\}_{l=1}^L$，每个特征图 $x_l \\in \\mathbb{R}^{C \\times H_l \\times W_l}$。\n\n查询元素的参考点用归一化坐标 $\\hat{p}_q \\in [0, 1]^2$ 表示，则多尺度的计算公式为：\n\n$$\n\\text{MSDeformAttn}(z_q, \\hat{p}_q, \\{x_l\\}_{l=1}^L) =\n$$\n\n$$\n\\sum_{m=1}^{M} W_m \\sum_{l=1}^{L} \\sum_{k=1}^{K} A_{mlqk} \\cdot W_m^{\\prime} x_l(\\phi_l(\\hat{p}_q) + \\Delta p_{mlqk})\n$$\n\n- $L$ 表示输入的特征图层数量。\n- $\\phi_l(\\hat{p}_q)$ 是一个缩放函数，用来将归一化坐标转换到第 $l$ 层特征图上的实际坐标。\n- 其他符号的意义与单尺度的情况类似；此时每个注意力头的权重在所有 $L\\times K$ 个采样点上一起归一化。\n\n### 模型架构\n\n\n\n![model](/content/papers/assets/36edfc5595a85c73fcdc9a4bbb9e1ca512e1f100676eb7c37203d71c84905c05.jpg)\n\n\n\n如上图，编码器中处理多尺度特征图的自注意力、以及解码器读取特征图的交叉注意力改用可变形注意力；**解码器内目标查询彼此互动的自注意力仍保留**。编码器输入使用 ResNet 的 C3–C5 特征与从 C5 下采样得到的 C6，不经 FPN 的自上而下融合。\n\n\n> [!NOTE]\n> \n> 如果理论知识你不感兴趣，也可以直接到官方的 Github 上取得他们的实作。\n> \n> - [**fundamentalvision/Deformable-DETR**](https://github.com/fundamentalvision/Deformable-DETR)\n> \n\n\n### 程式码拆解\n\n我们单独把作者的实作拿出来看一下，到底 MSDeformAttn 和一般的 Attn 有什么差异：\n\n```python\nclass MSDeformAttn(nn.Module):\n\n    def __init__(self, d_model=256, n_levels=4, n_heads=8, n_points=4):\n        \"\"\"\n        Multi-Scale Deformable Attention Module\n        :param d_model      hidden dimension\n        :param n_levels     number of feature levels\n        :param n_heads      number of attention heads\n        :param n_points     number of sampling points per attention head per feature level\n        \"\"\"\n        super().__init__()\n        if d_model % n_heads != 0:\n            raise ValueError('d_model must be divisible by n_heads, but got {} and {}'.format(d_model, n_heads))\n        _d_per_head = d_model // n_heads\n        # you'd better set _d_per_head to a power of 2 which is more efficient in our CUDA implementation\n        if not _is_power_of_2(_d_per_head):\n            warnings.warn(\"You'd better set d_model in MSDeformAttn to make the dimension of each attention head a power of 2 \"\n                          \"which is more efficient in our CUDA implementation.\")\n\n        self.im2col_step = 64\n\n        self.d_model = d_model\n        self.n_levels = n_levels\n        self.n_heads = n_heads\n        self.n_points = n_points\n\n        self.sampling_offsets = nn.Linear(d_model, n_heads * n_levels * n_points * 2)\n        self.attention_weights = nn.Linear(d_model, n_heads * n_levels * n_points)\n        self.value_proj = nn.Linear(d_model, d_model)\n        self.output_proj = nn.Linear(d_model, d_model)\n\n        self._reset_parameters()\n\n    def _reset_parameters(self):\n        constant_(self.sampling_offsets.weight.data, 0.)\n        thetas = torch.arange(self.n_heads, dtype=torch.float32) * (2.0 * math.pi / self.n_heads)\n        grid_init = torch.stack([thetas.cos(), thetas.sin()], -1)\n        grid_init = (grid_init / grid_init.abs().max(-1, keepdim=True)[0]).view(self.n_heads, 1, 1, 2).repeat(1, self.n_levels, self.n_points, 1)\n        for i in range(self.n_points):\n            grid_init[:, :, i, :] *= i + 1\n        with torch.no_grad():\n            self.sampling_offsets.bias = nn.Parameter(grid_init.view(-1))\n        constant_(self.attention_weights.weight.data, 0.)\n        constant_(self.attention_weights.bias.data, 0.)\n        xavier_uniform_(self.value_proj.weight.data)\n        constant_(self.value_proj.bias.data, 0.)\n        xavier_uniform_(self.output_proj.weight.data)\n        constant_(self.output_proj.bias.data, 0.)\n\n    def forward(self, query, reference_points, input_flatten, input_spatial_shapes, input_level_start_index, input_padding_mask=None):\n        \"\"\"\n        :param query                       (N, Length_{query}, C)\n        :param reference_points            (N, Length_{query}, n_levels, 2), range in [0, 1], top-left (0,0), bottom-right (1, 1), including padding area\n                                        or (N, Length_{query}, n_levels, 4), add additional (w, h) to form reference boxes\n        :param input_flatten               (N, \\sum_{l=0}^{L-1} H_l \\cdot W_l, C)\n        :param input_spatial_shapes        (n_levels, 2), [(H_0, W_0), (H_1, W_1), ..., (H_{L-1}, W_{L-1})]\n        :param input_level_start_index     (n_levels, ), [0, H_0*W_0, H_0*W_0+H_1*W_1, H_0*W_0+H_1*W_1+H_2*W_2, ..., H_0*W_0+H_1*W_1+...+H_{L-1}*W_{L-1}]\n        :param input_padding_mask          (N, \\sum_{l=0}^{L-1} H_l \\cdot W_l), True for padding elements, False for non-padding elements\n\n        :return output                     (N, Length_{query}, C)\n        \"\"\"\n        N, Len_q, _ = query.shape\n        N, Len_in, _ = input_flatten.shape\n        assert (input_spatial_shapes[:, 0] * input_spatial_shapes[:, 1]).sum() == Len_in\n\n        value = self.value_proj(input_flatten)\n        if input_padding_mask is not None:\n            value = value.masked_fill(input_padding_mask[..., None], float(0))\n        value = value.view(N, Len_in, self.n_heads, self.d_model // self.n_heads)\n        sampling_offsets = self.sampling_offsets(query).view(N, Len_q, self.n_heads, self.n_levels, self.n_points, 2)\n        attention_weights = self.attention_weights(query).view(N, Len_q, self.n_heads, self.n_levels * self.n_points)\n        attention_weights = F.softmax(attention_weights, -1).view(N, Len_q, self.n_heads, self.n_levels, self.n_points)\n        # N, Len_q, n_heads, n_levels, n_points, 2\n        if reference_points.shape[-1] == 2:\n            offset_normalizer = torch.stack([input_spatial_shapes[..., 1], input_spatial_shapes[..., 0]], -1)\n            sampling_locations = reference_points[:, :, None, :, None, :] \\\n                                 + sampling_offsets / offset_normalizer[None, None, None, :, None, :]\n        elif reference_points.shape[-1] == 4:\n            sampling_locations = reference_points[:, :, None, :, None, :2] \\\n                                 + sampling_offsets / self.n_points * reference_points[:, :, None, :, None, 2:] * 0.5\n        else:\n            raise ValueError(\n                'Last dim of reference_points must be 2 or 4, but get {} instead.'.format(reference_points.shape[-1]))\n        output = MSDeformAttnFunction.apply(\n            value, input_spatial_shapes, input_level_start_index, sampling_locations, attention_weights, self.im2col_step)\n        output = self.output_proj(output)\n        return output\n```\n\n从架构上来看，这个模块支持多层特征（不同分辨率）。论文使用 ResNet 的 C3–C6，经投影后展平为 `input_flatten`，再用 `input_spatial_shapes` 和 `input_level_start_index` 标记各层大小与起点；这些特征不是 FPN 融合后的输出。\n\n- **每个 Query 自行学习偏移量**\n\n  ```python\n  self.sampling_offsets = nn.Linear(d_model, n_heads * n_levels * n_points * 2)\n  ```\n\n  这行的意思是，对于每一个 Query，模型会输出多个 offset (dx, dy)，表示从 reference point 偏移的方向与距离，也就是「往哪看」。这些偏移点是学出来的，而非预先定义的固定格点。\n\n  ***\n\n- **Reference Points 决定「基准位置」**\n\n  ```python\n  if reference_points.shape[-1] == 2:\n      ...\n  elif reference_points.shape[-1] == 4:\n      ...\n  ```\n\n  注意力不再让 Query 对整张特征图逐点计算，而是以指定的 `reference_points` 为基准，在学得的偏移点取样并加权；偏移不限定在固定视窗内。Encoder 以各特征位置自身为参考点，Decoder 的参考点则由目标查询预测。\n\n  ***\n\n关键差异是采样位置与权重都由 Query 决定；上方程式也在多尺度间共同归一化权重。Decoder 的查询间自注意力仍使用原本的形式。\n\n### 训练策略\n\n作者在 **COCO 2017** 数据集上进行实验。\n\n首先 Backbone 使用 **ResNet-50**，并且在 **ImageNet** 上进行预训练。Neck 的部分直接从多尺度特征图中提取特征，并且没有使用 FPN。\n\n可变形注意力设定：\n\n- 注意力头的数量：**M = 8**\n- 每个查询在每个注意力头、每个特征层的采样点数量：**K = 4**\n\n可变形变换器编码器中的参数在不同的特征层之间共享。\n\n超参数和训练策略主要遵循 DETR 的设置，除了以下几点：\n\n- 边界框分类采用 Focal Loss，损失权重设为 2。\n- 将目标查询数量从 100 增加至 300。\n\n模型预设训练 50 个 epoch，第 40 个 epoch 时学习率会降低为原来的 0.1 倍。使用 Adam 优化器来进行训练，基本学习率为 $2 \\times 10^{-4}$，且 $\\beta_1 = 0.9$，$\\beta_2 = 0.999$，权重衰减为 $10^{-4}$。\n\n预测目标查询参考点与采样偏移量的线性投影的学习率会额外乘以 0.1 的缩放因子。\n\n## 讨论\n\n### 与 DETR 的比较\n\n\n\n![table1](/content/papers/assets/feaf8114181a3f1d27dffefb49b933a7122bc0d0658e677d6fa5fe3e409f085a.jpg)\n\n\n\n根据上表，Faster R-CNN + FPN 在 **109 epoch／380 GPU 小时**时为 **42.0 AP、26.6 AP_S**；DETR 在 **500 epoch／2000 GPU 小时**时同为 **42.0 AP**，但 AP_S 为 **20.5**。Deformable DETR 用 **50 epoch／325 GPU 小时**达 **43.8 AP、26.4 AP_S**；「少 10 倍」指相对 DETR 的训练 epoch，不能直接当成 GPU 时数的倍率。\n\n详细的收敛曲线见下图：\n\n\n\n![convergence](/content/papers/assets/44941305e93a2230c58aea591583189a58e0df43d7ee9a5a3a3591cbc1f424c2.jpg)\n\n\n\n表中同样训练 50 epoch 的迭代框修正使 AP 从 **43.8 升至 45.4**；再加两阶段机制升至 **46.2**。上图的不同收敛曲线也包含改变学习率下降时点的实验，并非全是同一个 50-epoch 设定。\n\n在论文的 **Tesla V100** 测试下，Deformable DETR 为 **173G FLOPs／19 FPS**，与 Faster R-CNN + FPN 的 **180G／26 FPS**、DETR-DC5 的 **187G／12 FPS** 计算量接近；其 FPS 约为 DETR-DC5 的 **1.6 倍**，比 Faster R-CNN + FPN 低约 **27%**。\n\nDETR-DC5 速度较慢的原因主要在于 Transformer 注意力机制中大量的内存存取，而 Deformable Attention 可以缓解此问题，但代价是无序的内存存取，仍稍慢于传统卷积。\n\n### 消融实验\n\n\n\n![table2](/content/papers/assets/c266c951f4c3ff4d426be7133fd7e68bb63ad205767d02d4899ac447dff11c15.jpg)\n\n\n\n上表展示了变形注意力模块中不同设计选项的消融研究。\n\n在未使用跨尺度注意力且 $K=1$ 的对照下，加入多尺度输入使 AP 从 **39.7 升至 41.4**、AP_S 从 **21.2 升至 24.1**（各高 **1.7／2.9 个百分点**）；再将 $K$ 增至 4，AP 由 **41.4 升至 42.3**（高 **0.9 个百分点**）。\n\n在多尺度输入、$K=4$ 下加入跨尺度注意力，AP 从 **42.3 升至 43.8**（高 **1.5 个百分点**）。同样配置再加 FPN 仍为 **43.8 AP**；BiFPN 为 **43.9 AP**，增益很小。当未使用跨尺度注意力且 $K=1$ 时，模块接近可变形卷积的单点取样形式，表列 AP 为 **41.4**，低于完整多尺度注意力的 **43.8**。\n\n### SOTA 比较\n\n\n\n![table3](/content/papers/assets/5ca60e40907572afd7904b78a8570b21437fc12906d6f1967135019ce6b91db8.jpg)\n\n\n\n在上表中，Deformable DETR 均使用了迭代边界框优化与两阶段机制。\n\n在已使用迭代框修正与两阶段机制、**不使用 TTA** 的设定下，ResNet-101 和 ResNeXt-101 分别达到 **48.7／49.0 AP**；带 DCN 的 ResNeXt-101 达 **50.1 AP**。不同骨干与 TTA 条件不能视为单独的注意力模块增益。\n\n在加入测试时增强（test-time augmentations）后，该方法达到 52.3 AP。\n\n\n> [!NOTE]\n> \n> 此处的 TTA 包含测试时的水平翻转与多尺度输入，再汇合各次预测；表中的 **52.3 AP** 是带 DCN 的 ResNeXt-101 加上 TTA 的结果，不能与未使用 TTA 的模型视为相同评估条件。\n\n\n## 结论\n\nDeformable DETR 相较于传统的 DETR，大幅缩短了训练所需的时间。在小物体检测的表现上，比 DETR 展现更明显的优势。\n\n虽然 Deformable DETR 引入了可变形注意力以提升运行速度，但由于无序内存存取的特性，仍略微慢于传统的卷积神经网络架构（如 Faster R-CNN）。因此，对于某些即时性要求高的应用，速度仍是需要考量的问题。\n\nDeformable DETR 的设计为探索更高效、更实用的端对端物体检测器变体开辟了新的方向，并具有广泛的研究与应用潜力。\n","imageSizes":{"/content/papers/assets/afd16d6c3cafc26ce43e498bd396a22b1965610ad3fbe96ae88e3668e9f8bdd3.jpg":{"width":1194,"height":670},"/content/papers/assets/36edfc5595a85c73fcdc9a4bbb9e1ca512e1f100676eb7c37203d71c84905c05.jpg":{"width":1224,"height":752},"/content/papers/assets/feaf8114181a3f1d27dffefb49b933a7122bc0d0658e677d6fa5fe3e409f085a.jpg":{"width":1224,"height":360},"/content/papers/assets/44941305e93a2230c58aea591583189a58e0df43d7ee9a5a3a3591cbc1f424c2.jpg":{"width":1148,"height":596},"/content/papers/assets/c266c951f4c3ff4d426be7133fd7e68bb63ad205767d02d4899ac447dff11c15.jpg":{"width":1224,"height":332},"/content/papers/assets/5ca60e40907572afd7904b78a8570b21437fc12906d6f1967135019ce6b91db8.jpg":{"width":1224,"height":428}}}
