{"notes":[{"id":"object-detection/dino-detr","category":"object-detection","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"DINO","subtitle":"Close, but Not Close Enough","author":"Z. Yuan","paperDate":"2022-03-07","publishedOn":"2024-12-19","world":"external/dino-detector","paperTitle":"DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection","paperUrl":"https://arxiv.org/abs/2203.03605","bibliography":{"title":"DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection","authors":["Hao Zhang","Feng Li","Shilong Liu","Lei Zhang","Hang Su","Jun Zhu","Lionel M. Ni","Heung-Yeung Shum"],"url":"https://arxiv.org/abs/2203.03605","pdfUrl":"https://arxiv.org/pdf/2203.03605v4","codeUrl":"https://github.com/IDEA-Research/DINO","preprint":{"id":"2203.03605","submittedOn":"2022-03-07","url":"https://arxiv.org/abs/2203.03605"},"publication":{"venue":"ICLR","year":2023,"kind":"conference","url":"https://iclr.cc/virtual/2023/poster/11884"},"citation":"@misc{objectdetectiondinodetr2022,\n  title = {{DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection}},\n  author = {Hao Zhang and Feng Li and Shilong Liu and Lei Zhang and Hang Su and Jun Zhu and Lionel M. Ni and Heung-Yeung Shum},\n  year = {2022},\n  eprint = {2203.03605},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2203.03605}\n}\n","citationPath":"/content/papers/citations/object-detection/dino-detr.d8c8af775089.bib"},"thumbnail":{"src":"/content/papers/assets/6c328e7125d4762c3a4346a1c537cb4e917d47a640869bbf212de53f74ba64fc.png","width":942,"height":319},"locale":"en","path":"/content/papers/en/object-detection/dino-detr.fcb370e89c60.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"solution","text":"Solution","depth":2},{"id":"the-overall-pipeline","text":"The Overall Pipeline","depth":3},{"id":"contrastive-denoising","text":"Contrastive Denoising","depth":3},{"id":"mixed-query-selection","text":"Mixed Query Selection","depth":3},{"id":"look-forward-twice","text":"Look Forward Twice","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"12-epochs","text":"12 Epochs","depth":3},{"id":"training-longer","text":"Training Longer","depth":3},{"id":"that-spot-on-the-leaderboard","text":"That Spot on the Leaderboard","depth":3},{"id":"ablation-how-the-494-stacks-up","text":"Ablation: How the 49.4 Stacks Up","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"contrastive-learning/dino","relation":"referenced","topic":""},{"id":"object-detection/detr","relation":"referenced","topic":""},{"id":"object-detection/deformable-detr","relation":"referenced","topic":""}]},{"id":"contrastive-learning/dino","category":"contrastive-learning","legacyPath":"contrastive-learning/2104-dino","legacySlug":"2104-dino","legacyPaths":[],"title":"DINO","subtitle":"Self-Distillation With No Labels","author":"Z. Yuan","paperDate":"2021-04-29","publishedOn":"2025-03-18","paperTitle":"Emerging Properties in Self-Supervised Vision Transformers","paperUrl":"https://arxiv.org/abs/2104.14294","bibliography":{"title":"Emerging Properties in Self-Supervised Vision Transformers","authors":["Mathilde Caron","Hugo Touvron","Ishan Misra","Hervé Jégou","Julien Mairal","Piotr Bojanowski","Armand Joulin"],"url":"https://arxiv.org/abs/2104.14294","pdfUrl":"https://arxiv.org/pdf/2104.14294","codeUrl":null,"preprint":{"id":"2104.14294","submittedOn":"2021-04-29","url":"https://arxiv.org/abs/2104.14294"},"publication":{"venue":"ICCV","year":2021,"kind":"conference","url":"https://doi.org/10.1109/iccv48922.2021.00951"},"citation":"@misc{contrastivelearningdino2021,\n  title = {{Emerging Properties in Self-Supervised Vision Transformers}},\n  author = {Mathilde Caron and Hugo Touvron and Ishan Misra and Hervé Jégou and Julien Mairal and Piotr Bojanowski and Armand Joulin},\n  year = {2021},\n  eprint = {2104.14294},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2104.14294}\n}\n","citationPath":"/content/papers/citations/contrastive-learning/dino.20875c1e301c.bib"},"thumbnail":{"src":"/content/papers/assets/c2d0ab75afb02750e17183e186f29b0ef6e184eda12673a31c96f199e338188f.jpg","width":664,"height":642},"locale":"en","path":"/content/papers/en/contrastive-learning/dino.df5756884ee0.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"preventing-model-collapse","text":"Preventing Model Collapse","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"comparison-with-other-methods","text":"Comparison with Other Methods","depth":3},{"id":"importance-of-different-components","text":"Importance of Different Components","depth":3},{"id":"model-collapse-testing","text":"Model Collapse Testing","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"contrastive-learning/dinov2","relation":"next","topic":"Self-distillation for visual representations"}]},{"id":"object-detection/detr","category":"object-detection","legacyPath":"object-detection/2005-detr","legacySlug":"2005-detr","legacyPaths":[],"title":"DETR","subtitle":"A Cross-Domain Pioneer","author":"Z. Yuan","paperDate":"2020-05-26","publishedOn":"2024-06-04","paperTitle":"End-to-End Object Detection with Transformers","paperUrl":"https://www.ecva.net/papers/eccv_2020/papers_ECCV/html/832_ECCV_2020_paper.php","bibliography":{"title":"End-to-End Object Detection with Transformers","authors":["Nicolas Carion","Francisco Massa","Gabriel Synnaeve","Nicolas Usunier","Alexander Kirillov","Sergey Zagoruyko"],"url":"https://www.ecva.net/papers/eccv_2020/papers_ECCV/html/832_ECCV_2020_paper.php","pdfUrl":"https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123460205.pdf","codeUrl":"https://github.com/facebookresearch/detr","preprint":{"id":"2005.12872","submittedOn":"2020-05-26","url":"https://arxiv.org/abs/2005.12872"},"publication":{"venue":"ECCV","year":2020,"kind":"conference","url":"https://www.ecva.net/papers/eccv_2020/papers_ECCV/html/832_ECCV_2020_paper.php"},"citation":"@misc{objectdetectiondetr2020,\n  title = {{End-to-End Object Detection with Transformers}},\n  author = {Nicolas Carion and Francisco Massa and Gabriel Synnaeve and Nicolas Usunier and Alexander Kirillov and Sergey Zagoruyko},\n  year = {2020},\n  eprint = {2005.12872},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2005.12872}\n}\n","citationPath":"/content/papers/citations/object-detection/detr.5258ffe0b9a0.bib"},"thumbnail":{"src":"/content/papers/assets/dd7e8441e6460ff562cf558374789924518fafaed7ecde0e3e35ee778c316330.jpg","width":1024,"height":266},"locale":"en","path":"/content/papers/en/object-detection/detr.493e749cd884.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"solution","text":"Solution","depth":2},{"id":"model-architecture","text":"Model Architecture","depth":3},{"id":"loss-design","text":"Loss Design","depth":3},{"id":"dataset","text":"Dataset","depth":3},{"id":"technical-details","text":"Technical Details","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"how-does-it-perform-this-time","text":"How does it perform this time?","depth":3},{"id":"component-analysis-of-the-model","text":"Component Analysis of the Model","depth":3},{"id":"model-behavior-analysis","text":"Model Behavior Analysis","depth":3},{"id":"generalization-to-unseen-object-counts","text":"Generalization to Unseen Object Counts","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{"defining-the-problem":"problem-definition","solving-the-problem":"solution","from-an-image-to-a-set-of-objects":"model-architecture","match-first-then-compute-the-loss":"loss-design","experimental-results-and-discussion":"discussion","how-did-it-do-start-with-the-conditions":"how-does-it-perform-this-time","which-components-matter":"component-analysis-of-the-model","what-did-the-queries-learn":"model-behavior-analysis"},"guide":null,"relatedReading":[{"id":"object-detection/deformable-detr","relation":"next","topic":"Set prediction and attention-based detection"},{"id":"object-detection/smca-detr","relation":"comparison","topic":"Faster convergence for Transformer detectors"}]},{"id":"object-detection/deformable-detr","category":"object-detection","legacyPath":"object-detection/2010-deformable-detr","legacySlug":"2010-deformable-detr","legacyPaths":[],"title":"Deformable DETR","subtitle":"Deformable Attention Mechanism","author":"Z. Yuan","paperDate":"2020-10-08","publishedOn":"2024-09-20","paperTitle":"Deformable DETR: Deformable Transformers for End-to-End Object Detection","paperUrl":"https://arxiv.org/abs/2010.04159","bibliography":{"title":"Deformable DETR: Deformable Transformers for End-to-End Object Detection","authors":["Xizhou Zhu","Weijie Su","Lewei Lu","Bin Li","Xiaogang Wang","Jifeng Dai"],"url":"https://arxiv.org/abs/2010.04159","pdfUrl":"https://arxiv.org/pdf/2010.04159","codeUrl":null,"preprint":{"id":"2010.04159","submittedOn":"2020-10-08","url":"https://arxiv.org/abs/2010.04159"},"publication":{"venue":"ICLR","year":2021,"kind":"conference","url":"https://arxiv.org/abs/2010.04159"},"citation":"@misc{objectdetectiondeformabledetr2020,\n  title = {{Deformable DETR: Deformable Transformers for End-to-End Object Detection}},\n  author = {Xizhou Zhu and Weijie Su and Lewei Lu and Bin Li and Xiaogang Wang and Jifeng Dai},\n  year = {2020},\n  eprint = {2010.04159},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2010.04159}\n}\n","citationPath":"/content/papers/citations/object-detection/deformable-detr.df721a30472e.bib"},"thumbnail":{"src":"/content/papers/assets/36edfc5595a85c73fcdc9a4bbb9e1ca512e1f100676eb7c37203d71c84905c05.jpg","width":1224,"height":752},"locale":"en","path":"/content/papers/en/object-detection/deformable-detr.68972544ff9b.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"problem-solution","text":"Problem Solution","depth":2},{"id":"deformable-attention","text":"Deformable Attention","depth":3},{"id":"multi-scale-computation","text":"Multi-Scale Computation","depth":3},{"id":"model-architecture","text":"Model Architecture","depth":3},{"id":"code-breakdown","text":"Code Breakdown","depth":3},{"id":"training-strategy","text":"Training Strategy","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"comparison-with-detr","text":"Comparison with DETR","depth":3},{"id":"ablation-studies","text":"Ablation Studies","depth":3},{"id":"sota-comparison","text":"SOTA Comparison","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"object-detection/detr","relation":"background","topic":"Set prediction and attention-based detection"},{"id":"object-detection/h-detr","relation":"next","topic":"Set prediction and attention-based detection"},{"id":"object-detection/smca-detr","relation":"comparison","topic":"Faster convergence for Transformer detectors"}]}],"text":"The DINO here stands for DETR with Improved deNoising anchOr boxes. It has nothing to do with the self-supervised [DINO](/blog/papers/contrastive-learning/dino) in another of our notes; the names just collide.\n\nIn early 2022, the detectors at the top of the COCO test-dev leaderboard, such as SwinV2-G and Florence, all attached large backbones to traditional detection heads like HTC++ and DyHead, and still needed NMS at inference to remove duplicate boxes. Papers in the DETR camp mostly used ResNet-50 to compete on convergence speed; the authors write in their introduction that no one had yet reported what happens when a DETR-style model gets a large backbone and large-scale data.\n\nDINO filled in that report card. With a SwinL backbone and Objects365 pre-training, it reached 63.3 AP on COCO test-dev. The authors say this was the first time an end-to-end Transformer detector beat the best models of the day on the COCO leaderboard. Switch the same design back to ResNet-50 and train for only 12 epochs, and it still gets 49.4 AP.\n\nThis paper keeps DETR's overall architecture. The skeleton comes from Deformable DETR, DAB-DETR, and DN-DETR, and the authors add three changes on top, the first of which concerns denoising training. DN-DETR only teaches the model to fix boxes that are off; DINO also teaches it to answer “there's nothing here” for some boxes.\n\n## Problem Definition\n\nFirst, a bit of background. DETR prepares a fixed number of queries, and each query outputs one class and one box. During training, the Hungarian algorithm assigns each annotated object to exactly one query, and every other query's target is “no object.” Trained this way, the model needs no NMS at inference, at the cost of very slow convergence.\n\nOver the next two years, several papers each filled in a piece, and DINO stands directly on these three:\n\n- **Deformable DETR**: attention looks only at a few sampled points near a reference point, which makes it fast to compute and quick to converge. It also proposed a two-stage query initialization and a way for the decoder to refine boxes layer by layer.\n- **DAB-DETR**: splits each query into a position part and a content part, writes the position part directly as an anchor box $(x, y, w, h)$, and has every decoder layer refine it once. Wherever “anchor box” appears below, this is what it means.\n- **DN-DETR**: attributes slow convergence to unstable matching, where the same query may be matched to one object this round and to a different one the next. During training, it feeds noised ground-truth boxes into the decoder as extra queries and asks the model to restore the original boxes. The answers to these queries are known in advance, so no Hungarian matching is needed.\n\nOn this foundation, the authors saw three gaps.\n\nThe first is in denoising. DN-DETR's noised boxes all stay close to the ground-truth box, so the model learns that “an anchor box with an object nearby should be refined into that object.” The authors point out that it never learns to have an anchor box with no object nearby answer “no object.”\n\nWhat trouble does this cause? According to the authors' analysis in §3.3, when several anchor boxes are close to the same object, the model has a hard time deciding which one to use. If it chooses badly, the same object may be boxed several times, or it may pick an anchor box that's farther away. DETR originally relies on one-to-one matching and self-attention among queries to suppress duplicate boxes, and the authors admit this ability is limited.\n\nThe second is in query initialization. DN-DETR's anchor boxes are learned fixed values, so every image starts from the same set of positions. Deformable DETR's two-stage approach instead picks the highest-scoring positions from the encoder and uses both their positions and their content to initialize the queries. The authors consider the content half unreliable, since the features the encoder picks out are still rough and may cover two objects at once or only part of one.\n\nThe third is in layer-by-layer box refinement. Deformable DETR has each decoder layer refine the previous layer's boxes, but cuts the gradient between layers, so each layer learns only from its own loss. The authors conjecture that the more accurate box from a later layer should also be able to give the layer before it a hand.\n\nOf the three gaps, the first bears directly on duplicate boxes. If denoising only teaches the model to fix boxes, can it learn to let go?\n\n\n> [!TIP]\n> \n> For details on DETR, Deformable DETR, and DN-DETR, see our notes:\n> \n> - **[[20.05] DETR: A Cross-Domain Pioneer](/blog/papers/object-detection/detr)**\n> - **[[20.10] Deformable DETR: Deformable Attention Mechanism](/blog/papers/object-detection/deformable-detr)**\n> - **[[22.03] DN-DETR: Dog Today, Cat Tomorrow](/blog/papers/object-detection/dn-detr)**\n\n\n## Solution\n\n### The Overall Pipeline\n\nDINO is built like any DETR-style model, with a backbone, a six-layer Transformer encoder, a six-layer decoder, and prediction heads for classes and boxes. The paper's architecture diagram marks where the new changes go:\n\n[![DINO architecture: on the left, multi-scale features and positional encodings are flattened and sent into the encoder; Query Selection above the encoder picks positions and sends them along Init Anchors to the decoder as initial anchor boxes, next to the learnable content queries; on the right of the decoder, a dashed line separates off CDN, where noised ground-truth boxes generate positive and negative queries](/content/papers/assets/6c328e7125d4762c3a4346a1c537cb4e917d47a640869bbf212de53f74ba64fc.png)](/content/papers/assets/6c328e7125d4762c3a4346a1c537cb4e917d47a640869bbf212de53f74ba64fc.png)\n\n*Figure 2 from the paper (Zhang et al.), excerpted from [**page 6 of the v4 PDF**](https://arxiv.org/pdf/2203.03605v4#page=6). DINO's overall architecture.*\n\nStart from the left. The multi-scale features are flattened, positional encodings are added, and the result goes into the encoder. Query Selection above the encoder picks positions from its output and sends them along the Init Anchors line to the bottom of the decoder as initial anchor boxes; the Learnable Content Queries beside it are learnable content vectors, and the two together make up mixed query selection. On the right of the decoder, set off by a dashed line, is CDN, the extra denoising queries used during training; they are generated from noised ground-truth boxes (GT+Noise) and split into positives (Pos) and negatives (Neg). Look forward twice happens between decoder layers and isn't drawn in the figure.\n\nIn data-flow order, an image goes through these steps:\n\n1. The backbone extracts multi-scale features. The 4-scale version takes the outputs of backbone stages 2 to 4 and downsamples stage 4 once more to make four levels; the 5-scale version also adds stage 1, the level with the highest resolution.\n2. The encoder processes these features with deformable attention.\n3. **Mixed query selection**: pick the 900 highest-scoring positions from the encoder output and use the boxes they predict as the decoder's initial anchor boxes.\n4. The decoder refines these 900 anchor boxes layer by layer, passing gradients with **look forward twice** as it refines them. During training, the decoder also receives an extra batch of **Contrastive DeNoising** (CDN) queries.\n5. The prediction heads output classes and boxes. The 900 queries are matched to the ground truth with Hungarian matching; the denoising queries' answers are known in advance, so their losses are computed directly.\n\nAt inference, the denoising queries are removed entirely, leaving only the 900 queries. The [official code](https://github.com/IDEA-Research/DINO) flattens the 900 queries' scores over every class and takes the top 300 as output, so the same query can appear twice under two different classes. No NMS follows; the config file keeps an NMS switch (`nms_iou_threshold`), which defaults to -1, meaning off.\n\n### Contrastive Denoising\n\nDN-DETR has only one noise scale $\\lambda$, and every box whose noise doesn't exceed $\\lambda$ has to be restored to the ground-truth box. DINO uses two scales instead, $\\lambda_1 < \\lambda_2$, and splits noised boxes into two kinds:\n\n- **Positives**: noise smaller than $\\lambda_1$; they must be restored to the ground-truth box. The loss is L1 and GIoU on the box plus focal loss on the class.\n- **Negatives**: noise between $\\lambda_1$ and $\\lambda_2$; they must answer “no object.” Only the classification focal loss is computed, with no box loss.\n\nEach ground-truth box produces one positive and one negative. With $n$ ground-truth boxes in an image, one group of CDN queries has $2n$ queries; as in DN-DETR, several groups are used during training. Figure 3 of the paper draws two groups:\n\n[![CDN diagram: on the left, two CDN groups each have three yellow positive queries and three brown negative queries, which pass through the decoder together; positives map to the reconstruction loss and negatives to no object. On the right are concentric squares, with the inner square labeled positive and λ1, and the band between the inner and outer squares labeled negative and λ2](/content/papers/assets/2016ccac10bf8f058a0d252c75ab4adf790e4c28f042f38eeddd61a2c18c94e5.png)](/content/papers/assets/2016ccac10bf8f058a0d252c75ab4adf790e4c28f042f38eeddd61a2c18c94e5.png)\n\n*Figure 3 from the paper (Zhang et al.), excerpted from [**page 8 of the v4 PDF**](https://arxiv.org/pdf/2203.03605v4#page=8). The structure of a CDN group and a sketch of positive and negative samples.*\n\nStart on the left. This image has three ground-truth boxes, so each group has three yellow positives and three brown negatives, and the two groups go into the decoder together. The positives' outputs are scored with the reconstruction loss, and the negatives' target is no object. The concentric squares on the right are a schematic, with the center point standing for the ground-truth box; noise that lands inside the inner square (below $\\lambda_1$) makes a positive, and noise that lands between the inner and outer squares ($\\lambda_1$ to $\\lambda_2$) makes a negative. An anchor box is really four-dimensional, $(x, y, w, h)$; the authors draw it as a point in a plane just to keep the picture simple.\n\nThe paper sets $\\lambda_1 = 1.0$ and $\\lambda_2 = 2.0$. In the official code, the noise is added to the box's four edges, each of which moves inward or outward at random on its own. A positive moves each edge by less than half the box width (half the box height for the top and bottom edges), and a negative moves it by half to a full box width. The official training script sets `dn_box_noise_scale` to 1.0, which gives exactly these distances.\n\nThe number of groups is decided dynamically. DINO uses a fixed budget of about 100 pairs, or 200 denoising queries, and sets the number of groups from the image in the batch with the most annotations; the official code divides 200 by twice that image's object count and takes the integer part. For example, if the most crowded image in a batch has 7 objects, that makes 14 groups of 14 queries each, 196 in all. Add the original 900, and during training the decoder handles 1,096 queries per image.\n\nThese queries are kept apart by an attention mask. The original 900 queries can't see any denoising query, or the ground-truth answers would leak through self-attention; different groups can't see each other either. Positives and negatives within the same group can see each other.\n\nSo how far are the negatives from the ground-truth box, really? Following the official code's noising scheme, we draw 100,000 positives and 100,000 negatives for one ground-truth box and compute their IoU with it:\n\n```python\nimport numpy as np\n\nrng = np.random.default_rng(0)\n\ndef iou(boxes, gt):\n    # boxes: (N, 4); gt: (4,); both in (x1, y1, x2, y2) format\n    x1 = np.maximum(boxes[:, 0], gt[0])\n    y1 = np.maximum(boxes[:, 1], gt[1])\n    x2 = np.minimum(boxes[:, 2], gt[2])\n    y2 = np.minimum(boxes[:, 3], gt[3])\n    inter = np.clip(x2 - x1, 0, None) * np.clip(y2 - y1, 0, None)\n    area = np.clip(boxes[:, 2] - boxes[:, 0], 0, None) * np.clip(boxes[:, 3] - boxes[:, 1], 0, None)\n    gt_area = (gt[2] - gt[0]) * (gt[3] - gt[1])\n    return inter / (area + gt_area - inter)\n\ndef cdn_noise(gt, n, negative):\n    # Mimics the official prepare_for_cdn: each of the four edges moves inward or outward on its own\n    w, h = gt[2] - gt[0], gt[3] - gt[1]\n    half = np.array([w / 2, h / 2, w / 2, h / 2])\n    u = rng.random((n, 4)) + (1.0 if negative else 0.0)  # positives [0, 1), negatives [1, 2)\n    sign = rng.choice([-1.0, 1.0], size=(n, 4))\n    return np.clip(gt + sign * u * half, 0.0, 1.0)\n\ngt = np.array([0.40, 0.35, 0.60, 0.65])  # a ground-truth box 0.2 wide and 0.3 tall, in normalized coordinates\nfor name, negative in ((\"Positives\", False), (\"Negatives\", True)):\n    boxes = cdn_noise(gt, 100_000, negative)\n    v = iou(boxes, gt)\n    p10, p50, p90 = np.percentile(v, [10, 50, 90])\n    crossed = ((boxes[:, 2] <= boxes[:, 0]) | (boxes[:, 3] <= boxes[:, 1])).mean()\n    print(f\"{name}: median IoU {p50:.2f}, 10%–90% within {p10:.2f}–{p90:.2f}, \"\n          f\"max {v.max():.2f}, crossed edges {crossed:.0%}\")\n```\n\n```text\nPositives: median IoU 0.38, 10%–90% within 0.21–0.56, max 0.94, crossed edges 0%\nNegatives: median IoU 0.01, 10%–90% within 0.00–0.12, max 0.24, crossed edges 44%\n```\n\nThe positives are already well off target, with a median IoU of only 0.38, and the model has to fix them back to the ground-truth box. The negatives are farther still, with a median near 0 and a maximum of only 0.24. More than 40% of the negatives also have their left and right edges, or their top and bottom edges, crossed over; in the official code, such a box becomes an anchor box with almost zero width or height, and the simulation counts its IoU as 0.\n\n0.24 is already close to an upper bound you can work out by hand. When the object doesn't touch the image border, every edge of a negative has to move by at least half the box width or height. The closest a negative can get to the ground-truth box is when all four edges move outward by exactly half the box, making it twice as wide and twice as tall with the ground-truth box right in the middle, for an IoU of $\\frac{wh}{2w \\cdot 2h} = \\frac{1}{4}$.\n\nSo these negatives look nothing like duplicate boxes. By the usual NMS thresholds of 0.5 or 0.7, they are nowhere near being “duplicates.”\n\nSo what are they teaching? Go back to how the decoder uses anchor boxes. An anchor box is a starting point; the decoder samples features around it and refines it layer by layer. DN-DETR, with only positives, trains the model into the habit of refining any nearby anchor box into that object. When several anchor boxes sit near the same object and the model refines them all toward it out of habit, the output is several nearly identical boxes. The duplication appears at the output end, while the anchor boxes they started from can be quite far apart.\n\nThe negatives fill in the other half. They also stay near the object, with each edge off by at most one box width, and the decoder samples features from the same object, yet their answer is “no object.” The same group also has closer positives that must be fixed back to the ground-truth box. From the difference between these two kinds of anchor boxes, the model has to learn how far off a starting point can be before it should let go.\n\nAn issue in the official repository raised the opposite worry. Suppose one anchor box has an IoU of 0.7 with an object and another has 0.5, and CDN teaches the 0.5 one as a negative; if the next image has only the 0.5 anchor box left, will the model give up on that one too? [IDEA-Research/DINO #176, 2023-03-23](https://github.com/IDEA-Research/DINO/issues/176) By the simulation above, under the official settings an anchor box with IoU 0.5 falls within the positive range, while negatives reach at most about 0.25. Hao Zhang, one of the paper's authors, replied that every CDN group always contains both positives and negatives, so some anchor box will always be matched to the object; he believed the case in the question wouldn't arise on the matching side, depending on how the thresholds are set. As for whether CDN could make such cases worse, they hadn't studied it.\n\nDoes CDN actually help the model pick better anchor boxes? The authors define a metric, ATD(k) (Average Top-K Distance). For each ground-truth box in the validation set, find the query that ends up matched to it and measure how far its starting anchor box was from the ground-truth box (the L1 distance over the four coordinates), then average the $k$ largest of these distances. The fewer matches to distant anchor boxes, the smaller this number.\n\nIn the ResNet-50, multi-scale, 12-epoch setting, ATD(100) computed over all objects is about the same for DN and CDN; for small objects alone, CDN's distance is shorter, and small-object AP is also 1.3 higher. The only direct evidence on duplicate boxes is a single example, in Figure 8 of the paper:\n\n[![Two copies of the same group photo of children side by side, showing only person boxes. In the left image, a white arrow points to a small boy with three nearly overlapping red boxes on him; in the right image, the same spot has only one box](/content/papers/assets/4aed72dc0127639f2eb01d2a2d681c46fbb1112ac82ea7dc1b2126a3559de642.png)](/content/papers/assets/4aed72dc0127639f2eb01d2a2d681c46fbb1112ac82ea7dc1b2126a3559de642.png)\n\n*Figure 8 from the paper (Zhang et al.), excerpted from [**page 15 of the v4 PDF**](https://arxiv.org/pdf/2203.03605v4#page=15). The left image comes from a model trained with DN queries, the right from DINO; both show only boxes of the “person” class.*\n\nLook at the small boy the white arrow points to in the left image; three nearly overlapping boxes are stacked on him, while the same spot in the right image has just one. The paper doesn't report how often duplicate boxes occur.\n\nCould it just be that there are more queries? Table 7 in the appendix compares different numbers of denoising queries:\n\n| Denoising queries | AP |\n| --- | ---: |\n| No denoising | 45.1 |\n| 10 DN | 46.0 |\n| 50 DN | 46.7 |\n| 100 DN | 47.4 |\n| 200 DN | 47.4 |\n| 1,000 DN | 47.6 |\n| 100 CDN pairs (200) | 47.9 |\n\n*Table: Compiled from Table 7 of the paper, ResNet-50, COCO val2017.*\n\nGoing from 0 to 100 DN queries raises AP from 45.1 to 47.4; adding more barely moves it, and even 1,000 only reaches 47.6. With the same 200 queries, turning half of them into negatives adds 0.5 AP.\n\n### Mixed Query Selection\n\nThere are three ways to set where the decoder's 900 queries start:\n\n| Approach | Anchor boxes (position) | Content |\n| --- | --- | --- |\n| Static (DETR, DN-DETR, etc.) | Learned fixed values | All zeros, or learned fixed values |\n| Pure query selection (Deformable DETR two-stage) | Boxes predicted at the encoder's top-K positions | The same encoder features, linearly transformed |\n| Mixed query selection (DINO) | Boxes predicted at the encoder's top-K positions | Learned fixed values |\n\n*Table: Compiled from §3.4 and Figure 5 of the paper.*\n\nDINO takes only the positions from the encoder and keeps the content as learnable vectors. In the official code, K is 900; each position's score is its highest score across classes, and the selected boxes are detached from the gradient before they serve as anchor boxes.\n\nThe authors argue that the features the encoder picks out haven't been refined yet and may mix two objects or hold only half of one, so using them as content could mislead the decoder; the positions, by contrast, are a useful prior. The decoder follows these better positions to sample features from the encoder, and the content builds up layer by layer.\n\n### Look Forward Twice\n\nDeformable DETR wires its layer-by-layer box refinement like this. Layer $i$ takes the previous layer's box $b_{i-1}$, predicts an offset $\\Delta b_i$, combines the two into this layer's box, then detaches the gradient and passes the box to the next layer. So the parameters of layer $i$ are updated only by layer $i$'s own loss. The authors call this look forward once.\n\nDINO changes it to this:\n\n$$\n\\begin{aligned}\n\\Delta b_i &= \\mathrm{Layer}_i(b_{i-1}), & b_i' &= \\mathrm{Update}(b_{i-1}, \\Delta b_i), \\\\\nb_i &= \\mathrm{Detach}(b_i'), & b_i^{(\\mathrm{pred})} &= \\mathrm{Update}(b_{i-1}', \\Delta b_i)\n\\end{aligned}\n$$\n\n$b_i'$ is the box that still carries gradients, and $b_i$ is the detached version; the next layer's input is still $b_i$. What changes is the predicted box used for the loss, $b_i^{(\\mathrm{pred})}$, which is built on the previous layer's $b_{i-1}'$, still carrying its gradients. This way, the same offset $\\Delta b_i$ is used twice, once to form this layer's $b_i'$ and once, through $b_i'$, in the next layer's predicted box, hence the name look forward twice.\n\n$\\mathrm{Update}$ is the same as in Deformable DETR. It takes the inverse sigmoid of coordinates normalized to the 0–1 range, adds the offset, and converts back with a sigmoid.\n\nLet's take one coordinate and two decoder layers, and see whether layer 2's loss reaches layer 1's offset:\n\n```python\nimport torch\n\ndef update(box, delta):\n    # Footnote 3 of the paper: add in inverse-sigmoid space, then map back to 0–1\n    return torch.sigmoid(torch.logit(box) + delta)\n\ntarget = torch.tensor(0.60)  # one coordinate of the ground-truth box\nb0 = torch.tensor(0.40)      # the anchor-box coordinate layer 1 receives\n\nfor mode in (\"look forward once\", \"look forward twice\"):\n    d1 = torch.tensor(0.30, requires_grad=True)  # offset Δb1 predicted by layer 1\n    d2 = torch.tensor(0.20, requires_grad=True)  # offset Δb2 predicted by layer 2\n\n    b1_prime = update(b0, d1)   # b1': keeps the gradient\n    b1 = b1_prime.detach()      # b1: detached, handed to layer 2 as its anchor box\n\n    if mode == \"look forward once\":\n        pred2 = update(b1, d2)\n    else:\n        pred2 = update(b1_prime, d2)\n\n    loss2 = (pred2 - target).abs()  # only layer 2's L1 loss\n    loss2.backward()\n    g1 = 0.0 if d1.grad is None else d1.grad.item()\n    print(f\"{mode}: layer-2 prediction {pred2.item():.4f}, \"\n          f\"gradient for Δb1 {g1:+.4f}, gradient for Δb2 {d2.grad.item():+.4f}\")\n```\n\n```text\nlook forward once: layer-2 prediction 0.5236, gradient for Δb1 +0.0000, gradient for Δb2 -0.2494\nlook forward twice: layer-2 prediction 0.5236, gradient for Δb1 -0.2494, gradient for Δb2 -0.2494\n```\n\nBoth wirings give exactly the same predicted box, 0.5236; the only difference is in the gradients. With look forward once, layer 2's loss never reaches $\\Delta b_1$; with look forward twice, it also flows back to $\\Delta b_1$. Here each layer's offset is simplified to a single learnable number; in the real model, $\\Delta b_i$ is the output of layer $i$, and the gradient continues into that layer's parameters. The gradient also goes back only one layer, because $b_{i-1}$ was already detached before entering layer $i$.\n\nInference computes exactly the same thing as before; this change affects only training.\n\n## Discussion\n\n### 12 Epochs\n\nFirst, the comparison with ResNet-50 on COCO val2017 after 12 epochs of training:\n\n| Model | AP | AP_S | GFLOPs | Params | FPS |\n| --- | ---: | ---: | ---: | ---: | ---: |\n| Faster-RCNN (5-scale) | 37.9 | 22.4 | 207 | 40M | 21 |\n| Deformable DETR (4-scale) | 41.1 | — | 196 | 40M | 24 |\n| DN-Deformable-DETR (4-scale) | 43.4 | 24.8 | 265 | 48M | 23 |\n| DINO-4scale | 49.0 | 32.0 | 279 | 47M | 24 |\n| DINO-5scale | 49.4 | 32.3 | 860 | 47M | 10 |\n\n*Table: Compiled from Table 1 of the paper. All FPS numbers were measured on the same A100, with Faster-RCNN's FPS measured in mmdetection.*\n\nCompared with DN-Deformable-DETR, DINO-4scale is 5.6 AP higher with similar compute and parameter counts, and its small-object AP_S is 7.2 higher. The two also use comparable numbers of queries; DINO uses 900 queries, and DN-Deformable-DETR uses 300 queries with 3 patterns, which the authors say works out to the equivalent of 900. Deformable DETR uses 300.\n\nThe difference between 4-scale and 5-scale is whether the features from backbone stage 1 are used. That level has the highest resolution; it takes GFLOPs from 279 to 860, roughly three times as much, and drops FPS from 24 to 10, in exchange for 0.4 AP. The 49.4 in the abstract is the 5-scale number.\n\n### Training Longer\n\nAt 24 and 36 epochs, 4-scale reaches 50.4 and 50.9, and 5-scale reaches 51.3 and 51.2 (Table 2). The comparison model, DN-Deformable-DETR, was trained for 50 epochs and gets 48.6. When the abstract says DINO is 2.7 AP above DN-DETR at 24 epochs, it is comparing DINO's 24 epochs with DN-DETR's 50. Going from 24 to 36 epochs, the 5-scale score has already stopped climbing.\n\nThe training cost is in Appendix B. On the same 8 A100s, DINO takes about 55 minutes per epoch and 16GB per GPU, so 12 epochs take roughly 11 hours. Deformable DETR (the basic version, without two-stage and box refinement) takes the same time per epoch, and 50 epochs take about 46 hours for 45.4 AP; DETR needs only 16 minutes per epoch but 300 epochs, about 80 hours, for 41.2 AP. DINO's result in this table is 47.9, from the same implementation period as the ablation table below.\n\n### That Spot on the Leaderboard\n\nThe DINO that topped the leaderboard was trained under very different conditions from the above:\n\n- The backbone was switched to SwinL, pre-trained on ImageNet-22K.\n- The detector was first pre-trained on Objects365 (about 1.7 million annotated images) for 26 epochs, using 64 A100s.\n- It was then fine-tuned on COCO for 18 epochs on 16 A100s, with images enlarged 1.5 times (longest side 2000), a step the authors say adds about 0.5 AP.\n- The denoising queries were increased to 1,000.\n\nHere are the results next to several large models:\n\n| Model | Params | Backbone pre-training data | Detection pre-training data | End-to-end | test-dev AP (without / with TTA) |\n| --- | ---: | --- | --- | :---: | ---: |\n| Florence-CoSwin-H | ≥ 637M | FLD-900M | FLD-9M | | — / 62.4 |\n| SwinV2-G | 3.0B | IN-22K-ext-70M | O365 | | — / 63.1 |\n| DINO-SwinL | 218M | IN-22K-14M | O365 | ✓ | 63.2 / 63.3 |\n\n*Table: Compiled from Table 3 of the paper. IN is ImageNet and O365 is Objects365; SwinV2-G was also trained with instance segmentation annotations.*\n\nWithout test-time augmentation (TTA), DINO's 63.2 already beats SwinV2-G's 63.1 with TTA, using 218M parameters against 3.0B. On val2017 it scores 63.1, and 63.2 with TTA. The authors point out that the previous SOTA models in Table 3 all used HTC++ or DyHead detection heads, and DINO was the first end-to-end Transformer detector to take that spot. The official repository's announcement on March 8, 2022 likewise says it took first place on the MS-COCO leaderboard with 63.3 AP.\n\nThat 63.3 uses TTA, meaning multi-scale testing plus horizontal flipping. When traditional detectors combine the results of several inference passes, they pool all the boxes and then remove duplicates with NMS or box voting. The authors say this approach actually lowers DINO's score, because DETR-style models rarely output duplicate boxes in the first place, and NMS tends to delete correct boxes by mistake. They switched to a one-to-one ensembling scheme instead, taking the predictions from one augmentation as the base; for each of its boxes, they find the box in each other augmentation's results with the highest IoU, provided it clears a threshold, and take a weighted average.\n\nThe same paper says in its appendix that DETR-style models rarely output duplicate boxes, and admits in §3.3 that the matching loss and self-attention have only limited power to suppress them. Read together, the two statements come closer to “no duplicates most of the time, with the occasional one slipping through,” and the little boy boxed three times is one that slipped through.\n\n### Ablation: How the 49.4 Stacks Up\n\nFinally, here is how much each of the three new designs contributes:\n\n| Setting | Query selection | CDN | LFT | AP | AP_S |\n| --- | :---: | :---: | :---: | ---: | ---: |\n| 1. DN-DETR | None | | | 43.4 | 24.8 |\n| 2. DN-DETR with a cleaned-up implementation | None | | | 44.9 | 26.9 |\n| 3. Strong baseline (2 plus pure query selection) | Pure | | | 46.5 | 29.6 |\n| 4. 3 with mixed query selection instead | Mixed | | | 47.0 | 31.1 |\n| 5. 4 plus look forward twice | Mixed | | ✓ | 47.4 | 29.9 |\n| 6. DINO (5 plus CDN) | Mixed | ✓ | ✓ | 47.9 | 31.2 |\n\n*Table: Compiled from Table 4 of the paper, ResNet-50, COCO val2017.*\n\nThe numbers in this table are the same as in the paper's first version and leave out the later engineering tweaks that raised Table 1 to 49.0, so the last row is 47.9.\n\nFrom DN-DETR's 43.4 to DINO's 47.9 is a total gain of 4.5 AP. The two biggest steps are the authors' cleanup of the DN-DETR implementation (+1.5) and borrowing Deformable DETR's pure query selection (+1.6). The three new designs add 0.5, 0.4, and 0.5 in turn, 1.4 in total.\n\nThe small-object column moves up and down. Adding look forward twice drops AP_S from 31.1 to 29.9, and it takes CDN to bring it back to 31.2; the paper doesn't explain this drop.\n\nSo the abstract's “49.4AP in 12 epochs” is a result with many things stacked together. The negatives themselves are worth 0.5 AP in this table.\n\nMixed query selection has one more side effect, in Appendix C. With the encoder kept at 6 layers and the decoder cut from 6 layers to 2, DINO loses 3.0 AP (47.4 to 44.4); Dynamic DETR, which the authors cite, loses 13.8 AP when cut to 2 layers. The authors credit this gap to mixed query selection, since anchor boxes taken from the encoder are good from the start, and the model depends less on the decoder refining them layer by layer. This experiment uses 12 epochs and 100 denoising queries, with no negatives.\n\n## Conclusion\n\nDINO adds three things on top of DN-DETR:\n\n1. **Contrastive denoising**: next to the same ground-truth box, place one anchor box that is slightly off and one that is farther off; the first is fixed back to the ground-truth box, and the second answers no object. Under the official settings, negatives have an IoU of at most about 0.25 with the ground-truth box, and what they teach is how far off a starting point can be before the model should let go.\n2. **Mixed query selection**: anchor boxes are chosen from the encoder, while the content stays as learnable vectors.\n3. **Look forward twice**: the next layer's loss passes back one layer further.\n\nWith ResNet-50 and 12 epochs, the 4-scale and 5-scale models reach 49.0 and 49.4 AP; with SwinL and Objects365 pre-training, DINO scores 63.2 on test-dev and 63.3 with TTA, taking first place on the COCO leaderboard at the time. In the ablation table, each of the three new designs is worth about half an AP, and most of the progress comes from the implementation cleanup and query selection. The only direct evidence that negatives suppress duplicate boxes is a single example image; the ATD curves show that small objects get matched to closer anchor boxes.\n\nDETR can skip NMS because training teaches the model which boxes to let go of, and the authors themselves say this ability is limited. When deploying, it's worth doing what the official code does and keeping an NMS switch around, to turn on when needed.\n\nDINO's speed is a separate question. A year later, RT-DETR measured DINO-Deformable-DETR-R50 on a T4 GPU with TensorRT FP16 at an 800×1333 input and got only 5 FPS. RT-DETR's decoder training follows DINO's settings almost exactly, down to carrying denoising queries; what it overhauled was the encoder, because multi-scale features had made the encoder the computational bottleneck. You can pick up the story from there in [RT-DETR](/blog/papers/object-detection/rt-detr).\n","imageSizes":{"/content/papers/assets/6c328e7125d4762c3a4346a1c537cb4e917d47a640869bbf212de53f74ba64fc.png":{"width":942,"height":319},"/content/papers/assets/2016ccac10bf8f058a0d252c75ab4adf790e4c28f042f38eeddd61a2c18c94e5.png":{"width":766,"height":353},"/content/papers/assets/4aed72dc0127639f2eb01d2a2d681c46fbb1112ac82ea7dc1b2126a3559de642.png":{"width":938,"height":313}}}