{"notes":[{"id":"object-detection/dn-detr","category":"object-detection","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"DN-DETR","subtitle":"Dog Today, Cat Tomorrow","author":"Z. Yuan","paperDate":"2022-03-02","publishedOn":"2024-11-27","world":"external/dn-detr","paperTitle":"DN-DETR: Accelerate DETR Training by Introducing Query DeNoising","paperUrl":"https://arxiv.org/abs/2203.01305","bibliography":{"title":"DN-DETR: Accelerate DETR Training by Introducing Query DeNoising","authors":["Feng Li","Hao Zhang","Shilong Liu","Jian Guo","Lionel M. Ni","Lei Zhang"],"url":"https://arxiv.org/abs/2203.01305","pdfUrl":"https://arxiv.org/pdf/2203.01305v3","codeUrl":"https://github.com/IDEA-Research/DN-DETR","preprint":{"id":"2203.01305","submittedOn":"2022-03-02","url":"https://arxiv.org/abs/2203.01305"},"publication":{"venue":"CVPR","year":2022,"kind":"conference","url":"https://doi.org/10.1109/CVPR52688.2022.01325"},"citation":"@misc{objectdetectiondndetr2022,\n title = {{DN-DETR: Accelerate DETR Training by Introducing Query DeNoising}},\n author = {Feng Li and Hao Zhang and Shilong Liu and Jian Guo and Lionel M. Ni and Lei Zhang},\n year = {2022},\n eprint = {2203.01305},\n archivePrefix = {arXiv},\n url = {https://arxiv.org/abs/2203.01305}\n}\n","citationPath":"/content/papers/citations/object-detection/dn-detr.a68d09a3ed54.bib"},"thumbnail":{"src":"/content/papers/assets/9bc1349cc89c66b31d56322bf37750edc42acb78407ab06b9f72c8261252ea27.png","width":1095,"height":443},"locale":"en","path":"/content/papers/en/object-detection/dn-detr.23df0821c4bb.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"rematched-every-round","text":"Rematched Every Round","depth":3},{"id":"counting-the-handoffs","text":"Counting the Handoffs","depth":3},{"id":"solution","text":"Solution","depth":2},{"id":"two-batches-of-queries-in-one-decoder","text":"Two Batches of Queries in One Decoder","depth":3},{"id":"turning-a-ground-truth-box-into-a-practice-problem","text":"Turning a Ground-Truth Box into a Practice Problem","depth":3},{"id":"several-copies-of-every-object","text":"Several Copies of Every Object","depth":3},{"id":"no-peeking-at-the-answers","text":"No Peeking at the Answers","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"experimental-setup","text":"Experimental Setup","depth":3},{"id":"gains-under-the-same-settings","text":"Gains Under the Same Settings","depth":3},{"id":"how-the-speedup-is-counted","text":"How the Speedup Is Counted","depth":3},{"id":"mask-labels-and-number-of-groups","text":"Mask, Labels, and Number of Groups","depth":3},{"id":"beyond-dab-detr","text":"Beyond DAB-DETR","depth":3},{"id":"who-did-the-practice-actually-help","text":"Who Did the Practice Actually Help?","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"object-detection/detr","relation":"referenced","topic":""},{"id":"object-detection/deformable-detr","relation":"referenced","topic":""},{"id":"object-detection/h-detr","relation":"referenced","topic":""}]},{"id":"object-detection/detr","category":"object-detection","legacyPath":"object-detection/2005-detr","legacySlug":"2005-detr","legacyPaths":[],"title":"DETR","subtitle":"A Cross-Domain Pioneer","author":"Z. Yuan","paperDate":"2020-05-26","publishedOn":"2024-06-04","paperTitle":"End-to-End Object Detection with Transformers","paperUrl":"https://www.ecva.net/papers/eccv_2020/papers_ECCV/html/832_ECCV_2020_paper.php","bibliography":{"title":"End-to-End Object Detection with Transformers","authors":["Nicolas Carion","Francisco Massa","Gabriel Synnaeve","Nicolas Usunier","Alexander Kirillov","Sergey Zagoruyko"],"url":"https://www.ecva.net/papers/eccv_2020/papers_ECCV/html/832_ECCV_2020_paper.php","pdfUrl":"https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123460205.pdf","codeUrl":"https://github.com/facebookresearch/detr","preprint":{"id":"2005.12872","submittedOn":"2020-05-26","url":"https://arxiv.org/abs/2005.12872"},"publication":{"venue":"ECCV","year":2020,"kind":"conference","url":"https://www.ecva.net/papers/eccv_2020/papers_ECCV/html/832_ECCV_2020_paper.php"},"citation":"@misc{objectdetectiondetr2020,\n title = {{End-to-End Object Detection with Transformers}},\n author = {Nicolas Carion and Francisco Massa and Gabriel Synnaeve and Nicolas Usunier and Alexander Kirillov and Sergey Zagoruyko},\n year = {2020},\n eprint = {2005.12872},\n archivePrefix = {arXiv},\n url = {https://arxiv.org/abs/2005.12872}\n}\n","citationPath":"/content/papers/citations/object-detection/detr.5258ffe0b9a0.bib"},"thumbnail":{"src":"/content/papers/assets/dd7e8441e6460ff562cf558374789924518fafaed7ecde0e3e35ee778c316330.jpg","width":1024,"height":266},"locale":"en","path":"/content/papers/en/object-detection/detr.493e749cd884.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"solution","text":"Solution","depth":2},{"id":"model-architecture","text":"Model Architecture","depth":3},{"id":"loss-design","text":"Loss Design","depth":3},{"id":"dataset","text":"Dataset","depth":3},{"id":"technical-details","text":"Technical Details","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"how-does-it-perform-this-time","text":"How does it perform this time?","depth":3},{"id":"component-analysis-of-the-model","text":"Component Analysis of the Model","depth":3},{"id":"model-behavior-analysis","text":"Model Behavior Analysis","depth":3},{"id":"generalization-to-unseen-object-counts","text":"Generalization to Unseen Object Counts","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{"defining-the-problem":"problem-definition","solving-the-problem":"solution","from-an-image-to-a-set-of-objects":"model-architecture","match-first-then-compute-the-loss":"loss-design","experimental-results-and-discussion":"discussion","how-did-it-do-start-with-the-conditions":"how-does-it-perform-this-time","which-components-matter":"component-analysis-of-the-model","what-did-the-queries-learn":"model-behavior-analysis"},"guide":null,"relatedReading":[{"id":"object-detection/deformable-detr","relation":"next","topic":"Set prediction and attention-based detection"},{"id":"object-detection/smca-detr","relation":"comparison","topic":"Faster convergence for Transformer detectors"}]},{"id":"object-detection/deformable-detr","category":"object-detection","legacyPath":"object-detection/2010-deformable-detr","legacySlug":"2010-deformable-detr","legacyPaths":[],"title":"Deformable DETR","subtitle":"Deformable Attention Mechanism","author":"Z. Yuan","paperDate":"2020-10-08","publishedOn":"2024-09-20","paperTitle":"Deformable DETR: Deformable Transformers for End-to-End Object Detection","paperUrl":"https://arxiv.org/abs/2010.04159","bibliography":{"title":"Deformable DETR: Deformable Transformers for End-to-End Object Detection","authors":["Xizhou Zhu","Weijie Su","Lewei Lu","Bin Li","Xiaogang Wang","Jifeng Dai"],"url":"https://arxiv.org/abs/2010.04159","pdfUrl":"https://arxiv.org/pdf/2010.04159","codeUrl":null,"preprint":{"id":"2010.04159","submittedOn":"2020-10-08","url":"https://arxiv.org/abs/2010.04159"},"publication":{"venue":"ICLR","year":2021,"kind":"conference","url":"https://arxiv.org/abs/2010.04159"},"citation":"@misc{objectdetectiondeformabledetr2020,\n title = {{Deformable DETR: Deformable Transformers for End-to-End Object Detection}},\n author = {Xizhou Zhu and Weijie Su and Lewei Lu and Bin Li and Xiaogang Wang and Jifeng Dai},\n year = {2020},\n eprint = {2010.04159},\n archivePrefix = {arXiv},\n url = {https://arxiv.org/abs/2010.04159}\n}\n","citationPath":"/content/papers/citations/object-detection/deformable-detr.df721a30472e.bib"},"thumbnail":{"src":"/content/papers/assets/36edfc5595a85c73fcdc9a4bbb9e1ca512e1f100676eb7c37203d71c84905c05.jpg","width":1224,"height":752},"locale":"en","path":"/content/papers/en/object-detection/deformable-detr.68972544ff9b.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"problem-solution","text":"Problem Solution","depth":2},{"id":"deformable-attention","text":"Deformable Attention","depth":3},{"id":"multi-scale-computation","text":"Multi-Scale Computation","depth":3},{"id":"model-architecture","text":"Model Architecture","depth":3},{"id":"code-breakdown","text":"Code Breakdown","depth":3},{"id":"training-strategy","text":"Training Strategy","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"comparison-with-detr","text":"Comparison with DETR","depth":3},{"id":"ablation-studies","text":"Ablation Studies","depth":3},{"id":"sota-comparison","text":"SOTA Comparison","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"object-detection/detr","relation":"background","topic":"Set prediction and attention-based detection"},{"id":"object-detection/h-detr","relation":"next","topic":"Set prediction and attention-based detection"},{"id":"object-detection/smca-detr","relation":"comparison","topic":"Faster convergence for Transformer detectors"}]},{"id":"object-detection/h-detr","category":"object-detection","legacyPath":"object-detection/2207-h-detr","legacySlug":"2207-h-detr","legacyPaths":[],"title":"H-DETR","subtitle":"The Rebellion of Silent Queries","author":"Z. Yuan","paperDate":"2022-07-26","publishedOn":"2025-07-06","paperTitle":"DETRs with Hybrid Matching","paperUrl":"https://arxiv.org/abs/2207.13080","bibliography":{"title":"DETRs with Hybrid Matching","authors":["Ding Jia","Yuhui Yuan","Haodi He","Xiaopei Wu","Haojun Yu","Weihong Lin","Lei Sun","Chao Zhang","Han Hu"],"url":"https://arxiv.org/abs/2207.13080","pdfUrl":"https://openaccess.thecvf.com/content/CVPR2023/papers/Jia_DETRs_With_Hybrid_Matching_CVPR_2023_paper.pdf","codeUrl":"https://github.com/HDETR/H-Deformable-DETR","preprint":{"id":"2207.13080","submittedOn":"2022-07-26","url":"https://arxiv.org/abs/2207.13080"},"publication":{"venue":"CVPR","year":2023,"kind":"conference","url":"https://openaccess.thecvf.com/content/CVPR2023/html/Jia_DETRs_With_Hybrid_Matching_CVPR_2023_paper.html"},"citation":"@misc{objectdetectionhdetr2022,\n title = {{DETRs with Hybrid Matching}},\n author = {Ding Jia and Yuhui Yuan and Haodi He and Xiaopei Wu and Haojun Yu and Weihong Lin and Lei Sun and Chao Zhang and Han Hu},\n year = {2022},\n eprint = {2207.13080},\n archivePrefix = {arXiv},\n url = {https://arxiv.org/abs/2207.13080}\n}\n","citationPath":"/content/papers/citations/object-detection/h-detr.822184101757.bib"},"thumbnail":{"src":"/content/papers/assets/63f1bc60b92f58d360457c0e3bbef0a3d4b3bd2a07377983b2dc5f5591ce28c2.jpg","width":1784,"height":350},"locale":"en","path":"/content/papers/en/object-detection/h-detr.005f94461932.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"solution","text":"Solution","depth":2},{"id":"main-branch-one-to-one-supervision","text":"Main Branch: One-to-One Supervision","depth":3},{"id":"auxiliary-branch-one-to-many-supervision","text":"Auxiliary Branch: One-to-Many Supervision","depth":3},{"id":"total-loss-and-implementation-details","text":"Total Loss and Implementation Details","depth":3},{"id":"more-forms-of-hybrid-supervision","text":"More Forms of Hybrid Supervision","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"coco-benchmark","text":"COCO Benchmark","depth":3},{"id":"lvis-benchmark","text":"LVIS Benchmark","depth":3},{"id":"sota-benchmark","text":"SOTA Benchmark","depth":3},{"id":"ablation-study","text":"Ablation Study","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{"auxiliary-branch-more-positive-slots":"auxiliary-branch-one-to-many-supervision","change-when-supervision-happens":"more-forms-of-hybrid-supervision","experimental-results-and-discussion":"discussion","coco-what-happens-when-more-queries-learn":"coco-benchmark","lvis-how-far-does-the-long-tail-evidence-go":"lvis-benchmark","system-comparison-the-conditions-behind-594-ap":"sota-benchmark","ap-training-time-and-inference-cost-across-the-three-schemes":"ablation-study"},"guide":null,"relatedReading":[{"id":"object-detection/deformable-detr","relation":"background","topic":"Set prediction and attention-based detection"}]}],"text":"DETR recast object detection as a matching problem and did away with anchors and NMS, but it trained astonishingly slowly. To get good results, it usually has to train for 500 epochs on COCO, while the original Faster R-CNN trains for only 12.\n\nBy early 2022, quite a few papers had tried to make DETR learn faster, most of them by changing the architecture. Deformable DETR has attention look only at a few sampled points near a reference point, Conditional DETR splits each query into a content part and a position part, and DAB-DETR writes each query directly as a box that the decoder refines layer by layer.\n\nThis paper comes from a team at the International Digital Economy Academy (IDEA) in Shenzhen, the Hong Kong University of Science and Technology, and Tsinghua University, and they took aim instead at the Hungarian matching DETR uses to hand out answers. Early in training, the same query on the same image may be matched to the dog in one epoch and to the cat in the next, so the target the model is trying to learn keeps jumping around.\n\nAlongside the original queries, they feed the decoder an extra batch of noised ground-truth boxes and ask it to restore the original boxes and classes. The answers to these problems are known in advance, so no matching is needed; at inference, these queries are removed entirely.\n\nThe conference version was an oral presentation at CVPR 2022. The extended version adds deformable attention to the decoder and applies the same trick to the original DETR, Anchor DETR, Faster R-CNN, and segmentation models; it was later published in IEEE TPAMI in 2024.\n\n## Problem Definition\n\n### Rematched Every Round\n\nFirst, recall how DETR trains. The model has a fixed number of queries, and each query outputs one class and one box. Every time the loss is computed, the Hungarian algorithm makes a one-to-one matching between predictions and ground truth, picking the assignment with the lowest total cost; a query matched to an object learns that object's box and class, and an unmatched one learns “no object.”\n\n\n> [!TIP]\n> \n> For DETR's overall architecture, Hungarian matching, and Deformable DETR, which makes attention look only at sampled points, see our notes:\n> \n> - **[[20.05] DETR: A Cross-Domain Pioneer](/blog/papers/object-detection/detr)**\n> - **[[20.10] Deformable DETR: Deformable Attention Mechanism](/blog/papers/object-detection/deformable-detr)**\n\n\nThis matching is recomputed every iteration from the current predictions. At the start of training, the predicted boxes are still a mess, and several queries have about the same cost for the same object; nudge the cost matrix slightly, and the whole matching can flip. Citing research on the stable marriage problem, the authors point out that the Hungarian algorithm only minimizes the total cost, and the matching it produces is not stable.\n\nFor a single query, the learning target therefore keeps changing. This round asks it to move its box toward the dog; the next round asks it to move toward the cat.\n\nDN-DETR is built on DAB-DETR. DAB-DETR writes each query as a 4D anchor box $(x, y, w, h)$, and each decoder layer outputs a set of corrections $(\\Delta x, \\Delta y, \\Delta w, \\Delta h)$ that pushes the box toward an object. The authors therefore split training into two tasks, learning good anchors and learning the relative offset from an anchor to its object. If the object an anchor corresponds to keeps changing, the offset can't be learned stably.\n\nAt the time, few studies suspected the matching. In the paper that introduced TSP, Sun et al. used a trained DETR as a teacher to give a student model fixed label assignments; this sped up convergence only early in training, and the final results were about the same, so they concluded that the Hungarian loss isn't the main cause of slow convergence. DN-DETR focuses on exactly that early stretch of training.\n\n### Counting the Handoffs\n\nThe authors designed a metric to measure this, called IS (instability). For an image, first record the matching result at epoch $i$:\n\n$$\nV^i_n=\\begin{cases} m, & \\text{if prediction } n \\text{ is matched to ground-truth object } m \\\\ -1, & \\text{if it is matched to no object} \\end{cases}\n$$\n\nThe number of entries in this vector that differ between two consecutive epochs is the image's IS at epoch $i$:\n\n$$\nIS^i=\\sum_{n=0}^{N-1}\\mathbf{1}\\left(V^i_n\\neq V^{i-1}_n\\right)\n$$\n\nHere $N$ is the number of predictions, and $\\mathbf{1}(\\cdot)$ is 1 when the condition holds. For the whole dataset, IS is then averaged over all images.\n\nLet's try it with five queries and two objects:\n\n```python\nimport numpy as np\n\ndef match_vector(pairs, num_queries):\n # Entry n records which object query n is matched to; -1 if it is unmatched\n v = np.full(num_queries, -1)\n for q, obj in pairs.items():\n v[q] = obj\n return v\n\nDOG, CAT = 0, 1\nprev = match_vector({0: DOG, 3: CAT}, 5) # previous epoch\nmoved = match_vector({1: DOG, 3: CAT}, 5) # query 1 takes over the dog\nswap = match_vector({0: CAT, 3: DOG}, 5) # the dog and the cat swap queries\nworst = match_vector({1: DOG, 4: CAT}, 5) # both objects move to new queries\n\nprint(\"Previous:\", prev)\nfor name, cur in [(\"Dog moved\", moved), (\"Dog-cat swap\", swap), (\"All moved\", worst)]:\n print(f\"{name}: {cur} IS = {(cur != prev).sum()}\")\n```\n\n```text\nPrevious: [ 0 -1 -1 1 -1]\nDog moved: [-1 0 -1 1 -1] IS = 2\nDog-cat swap: [ 1 -1 -1 0 -1] IS = 2\nAll moved: [-1 0 -1 -1 1] IS = 4\n```\n\nWhen another query takes over the dog, the old query goes from 0 to -1 and the new one from -1 to 0, so IS rises by 2. When the dog and the cat swap, two entries also change, one for each object. So each object that changes hands adds at most 2 to IS; when every object changes hands, IS is twice the number of objects.\n\nThe authors measured on COCO val2017, where each image has 7.36 objects on average, so the maximum IS is 7.36 × 2 = 14.72.\n\nFigure 2 of the paper trains DETR, DAB-DETR, and DN-DETR for 12 epochs under the same settings and measures IS once per epoch. The x-axis is the epoch, the y-axis is IS, and the solid red line is DN-DETR:\n\n[![Figure 2 from the paper: the x-axis runs from epoch 0 to 11, and the y-axis is IS. The solid red line for DN-DETR, the dashed blue line for DAB-DETR, and the dashed green line for DETR all start around 13 and fall; the red line is the lowest in most epochs and ends at about 5.7.](/content/papers/assets/545670c262e59825de2002719a679b732b310a49a0ecc9d8cdfbba0304278793.png)](/content/papers/assets/545670c262e59825de2002719a679b732b310a49a0ecc9d8cdfbba0304278793.png)\n\n*Figure 2 from the paper (Li et al.), excerpted from [**page 3 of the PDF**](https://arxiv.org/pdf/2203.01305v3#page=3). Matching changes between consecutive epochs, measured on COCO val2017; the original caption is not included.*\n\nAt the leftmost point, all three lines are around 13, not far from the maximum. Working backward, an IS of about 13 means that on average more than six objects per image changed the query responsible for them, which is nearly all of them.\n\nDN-DETR falls fastest, reaching about 9 by epoch 2 while DAB-DETR is still around 11; at the last epoch, DN-DETR is at about 5.7, DAB-DETR about 6.9, and the original DETR about 7.8. At epoch 7, though, DAB-DETR briefly dips below DN-DETR, then jumps back above 10 for the next two epochs.\n\nUnstable matching keeps changing the targets early in training, yet one-to-one matching can't be dropped, since it's what lets DETR skip NMS at inference. So could the matching stay, while the decoder also gets a separate assignment whose targets don't jump around?\n\n## Solution\n\n### Two Batches of Queries in One Decoder\n\nDN-DETR's decoder takes in two batches of queries at once:\n\n- **Matching part**: as in DAB-DETR, a set of learnable anchors whose outputs are scored with the Hungarian loss.\n- **Denoising part**: ground-truth boxes and classes with noise added, used as queries; each query must restore the object it came from, with no matching involved.\n\nFigure 6 of the paper draws the whole training flow in one place. On the left, the yellow and brown blocks are two denoising groups, each computing its own reconstruction loss; on the right, the green block is the matching part, which computes the Hungarian loss; the square grid on the far right is the attention mask, which we'll come back to later:\n\n[![Part of Figure 6 from the paper: in the decoder, the two denoising groups on the left take noised boxes and labels as input and feed a reconstruction loss; the matching part on the right takes learned anchors plus an unknown label as input and feeds the Hungarian loss. All three colored paths are blocked by red crosses. On the far right is the attention-mask grid, where gray cells mean blocked.](/content/papers/assets/9bc1349cc89c66b31d56322bf37750edc42acb78407ab06b9f72c8261252ea27.png)](/content/papers/assets/9bc1349cc89c66b31d56322bf37750edc42acb78407ab06b9f72c8261252ea27.png)\n\n*Part of Figure 6 from the paper (Li et al.), excerpted from [**page 5 of the PDF**](https://arxiv.org/pdf/2203.01305v3#page=5). The encoder on the left and the original caption are not included.*\n\nThe three colored paths in the figure are possible leaks, and a red cross means the mask blocks that path. The yellow and orange paths connect the two denoising groups, and the green one runs from the denoising part to the matching part.\n\nThe paper writes this as $\\mathbf{o}=D(\\mathbf{q},\\mathbf{Q},F\\mid A)$, where $\\mathbf{q}$ is the denoising part, $\\mathbf{Q}$ the matching part, $F$ the image features output by the encoder, and $A$ the attention mask. The total loss is the matching part's Hungarian loss plus the denoising part's reconstruction loss. At inference, the denoising part is removed entirely, and what remains is the same model as DAB-DETR.\n\nSince the denoising queries have to carry both a box and a class, the authors changed just one thing in DAB-DETR. Besides its anchor box, a DAB-DETR query has a content vector (the decoder embedding); DN-DETR replaces it with a class embedding followed by an indicator bit, 1 for denoising and 0 for matching. The matching part has no class to give, so all of its queries use the embedding of a single “unknown” class.\n\n\n> [!NOTE]\n> \n> In the official code, the class embedding table has 91 entries indexed by COCO category ID, plus 1 for unknown, each with 255 dimensions; append the indicator bit and it comes to exactly the decoder's 256 dimensions.\n\n\n### Turning a Ground-Truth Box into a Practice Problem\n\nLet's walk one dog through it. Its ground-truth box, in normalized coordinates, is $(c_x, c_y, w, h) = (0.40, 0.50, 0.30, 0.40)$.\n\nThere are two kinds of box noise:\n\n1. **Center shift**: add $(\\Delta x, \\Delta y)$ to the center, with $|\\Delta x| < \\lambda_1 w/2$ and $|\\Delta y| < \\lambda_1 h/2$. Since $\\lambda_1$ is less than 1, the new center always stays inside the original box.\n2. **Scaling**: the width is drawn at random from $[(1-\\lambda_2)w, (1+\\lambda_2)w]$, and the height likewise.\n\nClass noise randomly swaps the label, at a rate controlled by $\\gamma$. The paper's defaults are $\\lambda_1=\\lambda_2=0.4$ and $\\gamma=0.2$. For this dog, the center can move at most 0.06 left or right and 0.08 up or down, the width falls between 0.18 and 0.42, and the height between 0.24 and 0.56.\n\nFollowing the official code, we make five copies for this dog:\n\n```python\nimport numpy as np\n\nCLASSES = [\"person\", \"dog\", \"cat\", \"car\", \"bird\", \"chair\", \"bottle\", \"sofa\"]\ngt_box = np.array([0.40, 0.50, 0.30, 0.40]) # (cx, cy, w, h), normalized by the image width and height\ngt_label = CLASSES.index(\"dog\")\nlam, gamma, groups = 0.4, 0.2, 5 # paper defaults: λ1 = λ2 = 0.4, γ = 0.2, 5 groups\n\nrng = np.random.default_rng(2)\nw, h = gt_box[2:]\nfor g in range(groups):\n # The center moves at most λ·w/2 and λ·h/2; the width and height scale by at most ±λ\n u = rng.uniform(-1, 1, size=4)\n box = (gt_box + u * np.array([w / 2, h / 2, w, h]) * lam).clip(0, 1)\n # Each label has probability γ of being swapped for a random class\n label = gt_label\n if rng.random() < gamma:\n label = int(rng.integers(len(CLASSES)))\n print(f\"Group {g}: box {box.round(3)}, label “{CLASSES[label]}”\")\n```\n\n```text\nGroup 0: box [0.371 0.468 0.375 0.269], label “dog”\nGroup 1: box [0.427 0.45 0.193 0.328], label “dog”\nGroup 2: box [0.407 0.444 0.284 0.454], label “dog”\nGroup 3: box [0.416 0.575 0.344 0.365], label “cat”\nGroup 4: box [0.401 0.563 0.366 0.342], label “dog”\n```\n\nThis example uses only 8 classes. When the official code swaps a label, it draws uniformly from all classes, so now and then it draws the original class again.\n\nHere are the five boxes, plotted:\n\n![A dog's ground-truth box, centered at (0.40, 0.50), overlaid with five dashed noised boxes. All five centers fall inside a gray area in the middle of the ground-truth box, and each box is stretched or shrunk by a different amount; group 3's label has been swapped to cat.](/content/papers/assets/01974c337c3f7732bd2bf7101d57534d35507b06c9c6b57773347e5e9e23cbef.png)\n\n*Figure: Compiled for this note. The black box is the dog's ground-truth box, the dashed boxes are the five noised boxes generated by the code above, and the dots are their centers.*\n\nAll five centers land in the gray area, and the widths and heights vary, but every box still surrounds this dog. These five boxes then serve directly as the anchors of five queries, and their labels are looked up in the embedding table to become content vectors. To the decoder, each one is a good anchor with an object definitely nearby, and the task is to correct it back to the black box and say “dog.”\n\nGroup 3's label has been swapped to “cat,” so the model has to work out from the box's position and the image content that this is actually a dog. The authors say label swapping is there to force the model to learn the relationship between boxes and classes.\n\nThe reconstruction loss reuses DAB-DETR's combination, L1 and GIoU for boxes and focal loss for classes. The official code computes this loss at every decoder layer, with the same weights as the matching part.\n\n### Several Copies of Every Object\n\nOne set of noise gives just one version, so the authors make several copies of each object at once. Each copy contains all the annotated objects in the image and is called a denoising group. With $P$ groups of $M$ objects each, the denoising part has $P \\times M$ queries.\n\nThe default is 5 groups. With COCO val2017's average of 7.36 objects per image, the denoising part averages about 37 queries, against 300 in the matching part. Images in the same batch have different numbers of objects, so the official implementation pads every group to the image with the most objects in the batch. The authors explained this when the question came up on GitHub, and mentioned that the later DINO switched to a dynamic number of groups. [DN-DETR #8, 2022-04-19](https://github.com/IDEA-Research/DN-DETR/issues/8)\n\nThe extra compute is small. For DAB-DETR with ResNet-50, 5 groups raise the training GFLOPs from 94.4 to 94.6; inference has no extra computation at all.\n\n### No Peeking at the Answers\n\nPut ground-truth boxes straight into the decoder, and self-attention lets every query see every other one; a matching query only has to glance at a nearby denoising query to know roughly where the object is.\n\nThe authors list two kinds of leakage:\n\n1. The matching part sees the denoising part, which amounts to being handed the ground-truth boxes.\n2. Different noised versions of the same object see each other, which amounts to peeking at other versions of the same problem.\n\nThe attention mask $A=[a_{ij}]_{W\\times W}$ blocks both paths. $W=P\\times M+N$, where the first $P \\times M$ positions are the denoising part and the remaining $N$ are the matching part; $a_{ij}=1$ means query $i$ can't see query $j$:\n\n$$\na_{ij}= \\begin{cases} 1, & j= P * M:\n A[i, j] = 1 # matching queries can't see any denoising query\n\nprint(\" \" + \" \".join(f\"{n:>3}\" for n in names))\nfor n, row in zip(names, A):\n print(f\"{n:>4} \" + \" \".join(f\"{v:>3}\" for v in row))\n```\n\n```text\n g0A g0B g1A g1B m0 m1 m2\n g0A 0 0 1 1 0 0 0\n g0B 0 0 1 1 0 0 0\n g1A 1 1 0 0 0 0 0\n g1B 1 1 0 0 0 0 0\n m0 1 1 1 1 0 0 0\n m1 1 1 1 1 0 0 0\n m2 1 1 1 1 0 0 0\n```\n\nRead it row by row. g0A and g0B are the dog and the cat in group 0; they can see each other but not the two versions in group 1. In the three rows from m0 to m2, the first four entries are all 1, so the matching queries can't see any denoising query. This is the same pattern as the grid on the far right of Figure 6, where the gray cells are the 1s here and the colored cells are the 0s.\n\nThe rightmost three columns are all 0, so the denoising queries can look at the matching queries. The authors say this doesn't affect the results, because the matching queries are learned vectors and contain no ground-truth information to begin with.\n\nThis mask is used only in the decoder's self-attention. Someone on GitHub asked whether cross-attention needs masking too, and the authors replied that, in their experience, cross-attention doesn't cause leakage. [DN-DETR #35, 2022-07-26](https://github.com/IDEA-Research/DN-DETR/issues/35) In cross-attention, each query attends only to image features, with no channel between queries.\n\nThe ablation in Table 5 of the paper uses ResNet-50, 1 denoising group, and 50 epochs of training. With noise on both boxes and labels and the mask in place, it scores 43.4 AP; with the same noise but no mask, it drops to 24.0, which is 18.2 below the 42.2 with no denoising at all.\n\nThe paper explains that without the mask, the matching part can see the noised ground truth and easily predicts the annotated objects. Following that reasoning, once the denoising queries are removed at inference, the cues the matching queries leaned on during training vanish with them.\n\n## Discussion\n\n### Experimental Setup\n\nAll experiments are on COCO 2017, trained on train2017 and evaluated on val2017. The main model, DN-DETR, is DAB-DETR plus denoising training, with a 6-layer encoder, a 6-layer decoder, a hidden dimension of 256, and 300 matching queries. The noise uses $\\lambda_1=\\lambda_2=0.4$ and $\\gamma=0.2$, with 5 groups by default. The optimizer is AdamW with a learning rate of $10^{-4}$, on 8 A100s with a batch size of 16. The 50-epoch schedule lowers the learning rate at epoch 40, and the 12-epoch schedule at epoch 11.\n\nThe authors also attached the same training to Deformable DETR. DN-Deformable-DETR uses deformable attention only in the encoder, while DN-Deformable-DETR++ uses it in both the encoder and the decoder. Each has a matching DAB-Deformable-DETR baseline that also writes queries as 4D boxes but skips denoising, so that the benefit of 4D queries isn't credited to denoising.\n\n### Gains Under the Same Settings\n\nThe cleanest comparison is in Table 1. It uses single-scale features and 50 epochs of training, and DAB-DETR and DN-DETR differ only in whether they use denoising training:\n\n| Backbone | DAB-DETR | DN-DETR |\n| --- | ---: | ---: |\n| ResNet-50 | 42.2 | 44.1 (+1.9) |\n| ResNet-101 | 43.5 | 45.2 (+1.7) |\n| ResNet-50-DC5 | 44.5 | 46.3 (+1.8) |\n| ResNet-101-DC5 | 45.8 | 47.3 (+1.5) |\n\n*Table: Compiled from Table 1 of the paper, COCO val2017 AP. Both sides have the same GFLOPs and parameter count, for example 94 GFLOPs and 44M parameters for ResNet-50.*\n\nAll four backbones gain 1.5 to 1.9 AP. Cut training to 12 epochs and the gap grows. In Table 2, DC5-R50 goes from 38.0 to 41.7 (+3.7), and DC5-R101 from 40.3 to 42.8 (+2.5). The larger gains on the short schedule fit the authors' claim that the problem is concentrated in “early training stages.”\n\n### How the Speedup Is Counted\n\nThe title promises a speedup, and the authors give two kinds of evidence.\n\nOne is the convergence curves. The caption of Figure 7(a) says that for DC5-R50, before the learning rate drops, DN-DETR reaches 40 AP in 20 epochs, while DAB-DETR needs 40 epochs.\n\nThe other is to halve the schedule and compare directly:\n\n| Model | Baseline | With denoising, half the epochs |\n| --- | ---: | ---: |\n| DAB-DETR-DC5-R50 (50 vs. 25 epochs) | 44.5 | 44.4 |\n| DAB-Deformable-DETR-R50 (50 vs. 25 epochs) | 46.9 | 46.8 |\n| DAB-Deformable-DETR-R50++ (50 vs. 25 epochs) | 48.7 | 48.4 |\n| DETR-R50 (500 vs. 250 epochs) | 42.0 | 42.2 |\n\n*Table: Compiled from Table 1 on page 11 of the paper, COCO val2017 AP.*\n\nWith half the epochs, AP is only 0.1 to 0.3 lower, and the original DETR run even comes out 0.2 higher. This table is what the abstract means by “comparable performance with 50% training epochs.”\n\nHalving the epochs doesn't exactly halve the time, though. The authors measured actual ResNet-50 training time on 8 A100s, where DAB-DETR takes 51.1 minutes per epoch and 57.7 minutes with denoising, about 13% more. GFLOPs barely change, yet the wall-clock time goes up by more than a tenth; when estimating training cost, you still have to look at measured time.\n\nIn total, DAB-DETR needs 2,555 minutes for 50 epochs, while the version with denoising needs 1,443 minutes for 25 epochs, saving about 44%. This timing table measures ResNet-50, whereas the 25-epoch AP comparison above uses DC5-R50, so the two come from different models.\n\nThe abstract's 46.0 AP at 12 epochs and 49.5 AP at 50 epochs come from DN-Deformable-DETR++, which uses multi-scale features and deformable attention in both the encoder and the decoder; the “+1.9AP” is the single-scale comparison on DAB-DETR. Against a baseline with the same ++ architecture, the gap at 50 epochs is 49.5 versus 48.7, or 0.8 AP.\n\n### Mask, Labels, and Number of Groups\n\nThe remaining two rows of Table 5 separate the contributions of box noise and label noise. Box noise alone gives 43.0, 0.8 above the 42.2 without denoising; adding label swapping brings it to 43.4.\n\nFor the number of groups, see Table 6:\n\n| Backbone | No denoising | 1 group | 5 groups |\n| --- | ---: | ---: | ---: |\n| ResNet-50 | 42.2 | 43.4 | 44.1 |\n| ResNet-50-DC5 | 44.5 | 45.6 | 46.3 |\n| ResNet-101 | 43.5 | 45.0 | 45.2 |\n| ResNet-101-DC5 | 45.8 | 46.5 | 47.3 |\n\n*Table: Compiled from Table 6 of the paper, ResNet series, 50 epochs.*\n\nThe gain from none to 1 group is mostly larger than from 1 to 5 groups; only ResNet-101-DC5 goes the other way. On this basis, the authors say that more groups are better but with diminishing returns, and they default to 5.\n\nFor the noise scale, see Figure 8. This experiment uses ResNet-50, a batch size of 64, and 20 epochs of training without a learning-rate drop, fixing one kind of noise at 0.4 and varying the other. Both curves are high in the middle and low at both ends. With one kind of noise set to 0, AP is between 33.5 and 34; it is highest at 0.2 to 0.4, where the figure marks peaks of 35.5 and 35.7; pushed to 0.8, it drops back to around 34. The paper concludes that both kinds of noise help, but too much noise drags performance down.\n\n### Beyond DAB-DETR\n\nThe extended version attaches denoising training to other models (Table 3, all ResNet-50):\n\n- **Anchor DETR**: its queries have only 2D anchor points, so only the center shift is used; DC5, 12 epochs, 38.2 to 39.4 (+1.2).\n- **Original DETR**: its queries have no explicit box, so a linear layer embeds the noised box into the query; 300 epochs, 40.6 to 42.6 (+2.0).\n- **Faster R-CNN**: there's no content vector, so only box noise is added, and the noised boxes go into the detection head together with the RPN proposals; FPN, 12 epochs, 37.9 to 38.4 (+0.5).\n\nA few rows in the table carry asterisks; their numbers come from the ablations in later papers that adopted denoising training, such as DINO and Group DETR. For example, DINO scores 46.0 without denoising and 47.4 with it.\n\nFaster R-CNN gains the least. It assigns labels with an IoU threshold instead of Hungarian matching; this is consistent with the direction of the authors' diagnosis, but the two kinds of detectors differ in too many ways for this one row to pin down the cause.\n\n### Who Did the Practice Actually Help?\n\nThe matching queries can't see the denoising queries, and the two sides' losses are computed separately, so at what point does denoising actually help the matching? Someone asked exactly this on GitHub in October 2024, and the question has no reply in the repo. [DN-DETR #72, 2024-10-14](https://github.com/IDEA-Research/DN-DETR/issues/72)\n\nBoth batches of queries share the same backbone, the same encoder, and the same decoder layers and prediction heads, and the denoising gradients flow all the way back into these parameters. The paper supports its reading with two observations. One is the IS curve above. The other is Figure 3, which measures the average L1 distance between the matching queries' initial anchors and the ground-truth boxes they are matched to at the last decoder layer; over 12 epochs, DAB-DETR goes from about 0.146 to about 0.096 and DN-DETR from about 0.110 to about 0.082, staying closer throughout. The authors' view is that denoising teaches queries to look for objects nearby, so the matching jumps around less.\n\nLater studies read “where it helps” differently:\n\n- **Group DETR** (ICCV 2023) points out that in a denoising group each object has only one query, and no query corresponds to “no object”; self-attention within a group sees only other objects, never duplicate boxes. It therefore argues that denoising mainly adds positive supervision and gives no direct help with suppressing duplicate predictions. In the DAB-DETR, 12-epoch setting, adding denoising on top of Group DETR still gives another 1.5 AP, which its authors take as a sign that the two are complementary.\n- **[H-DETR](/blog/papers/object-detection/h-detr)** splits the parameters directly in a supplementary experiment. On DINO, which uses contrastive denoising, giving the denoising branch its own decoder and prediction heads moves AP from 49.1 to 49.0; separating the encoder as well drops it to 48.5. By these numbers, most of the benefit of denoising reaches the matching part through the training of the encoder.\n- **DINO**, a paper the same team put on arXiv five days later, points straight at a gap in DN-DETR. The model learned to predict from anchors that have a ground-truth box nearby, but never learned to answer “no object” for anchors with no object nearby. DINO therefore added negative samples to the practice problems.\n\nThese accounts can all hold at once. The paper's evidence shows the drop in IS and the faster convergence appearing together; the paper itself doesn't separate whether denoising first steadies the decoder's targets or feeds the encoder more gradient from positives.\n\n## Conclusion\n\nDN-DETR traces part of DETR's slow training to Hungarian matching being recomputed every round, and uses IS to turn “dog today, cat tomorrow” into a curve. During training, it adds a set of practice problems with known answers, giving the decoder noised ground-truth boxes and classes to restore, and uses an attention mask to keep the matching queries from peeking.\n\nUnder the authors' settings, it gains 1.5 to 1.9 AP with single-scale features at 50 epochs, and 2.5 to 3.7 AP at 12 epochs; it catches up with the baseline in half the epochs, at the cost of about 13% more time per epoch.\n\nIf you want to add denoising training to your own DETR-style model, get the mask right first; in the paper's ablation, AP fell from 43.4 to 24.0 without it. Images in a batch have different numbers of objects, so the groups need padding. The queries are best off carrying an explicit box, as in DAB-DETR; if they don't, you'll have to embed the box separately, as with the original DETR. Inference doesn't need to change at all.\n\nThese practice problems were later picked up by many DETR-style models. [DINO](/blog/papers/object-detection/dino-detr) added negative samples to them, and [RT-DETR](/blog/papers/object-detection/rt-detr) also carries 200 denoising queries during training.\n","imageSizes":{"/content/papers/assets/545670c262e59825de2002719a679b732b310a49a0ecc9d8cdfbba0304278793.png":{"width":646,"height":344},"/content/papers/assets/9bc1349cc89c66b31d56322bf37750edc42acb78407ab06b9f72c8261252ea27.png":{"width":1095,"height":443},"/content/papers/assets/01974c337c3f7732bd2bf7101d57534d35507b06c9c6b57773347e5e9e23cbef.png":{"width":960,"height":1212}}}