{"notes":[{"id":"object-detection/deim","category":"object-detection","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"DEIM","subtitle":"Positives by Collage","author":"Z. Yuan","paperDate":"2024-12-05","publishedOn":"2026-06-26","world":"external/deim","paperTitle":"DEIM: DETR with Improved Matching for Fast Convergence","paperUrl":"https://arxiv.org/abs/2412.04234","bibliography":{"title":"DEIM: DETR with Improved Matching for Fast Convergence","authors":["Shihua Huang","Zhichao Lu","Xiaodong Cun","Yongjun Yu","Xiao Zhou","Xi Shen"],"url":"https://arxiv.org/abs/2412.04234","pdfUrl":"https://arxiv.org/pdf/2412.04234v3","codeUrl":"https://github.com/Intellindust-AI-Lab/DEIM","preprint":{"id":"2412.04234","submittedOn":"2024-12-05","url":"https://arxiv.org/abs/2412.04234"},"publication":{"venue":"CVPR","year":2025,"kind":"conference","url":"https://openaccess.thecvf.com/content/CVPR2025/html/Huang_DEIM_DETR_with_Improved_Matching_for_Fast_Convergence_CVPR_2025_paper.html"},"citation":"@misc{objectdetectiondeim2024,\n title = {{DEIM: DETR with Improved Matching for Fast Convergence}},\n author = {Shihua Huang and Zhichao Lu and Xiaodong Cun and Yongjun Yu and Xiao Zhou and Xi Shen},\n year = {2024},\n eprint = {2412.04234},\n archivePrefix = {arXiv},\n url = {https://arxiv.org/abs/2412.04234}\n}\n","citationPath":"/content/papers/citations/object-detection/deim.6126444ef2d4.bib"},"thumbnail":{"src":"/content/papers/assets/728ebf51d648559c31fbbb0d0dafd4107b254df8b6016972cd881c555eb051ae.png","width":1280,"height":1720},"locale":"en","path":"/content/papers/en/object-detection/deim.4685ba8692f3.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"only-as-many-positives-as-objects","text":"Only as Many Positives as Objects","depth":3},{"id":"matched-but-off-target","text":"Matched, but Off Target","depth":3},{"id":"how-earlier-work-added-positives","text":"How Earlier Work Added Positives","depth":3},{"id":"solution","text":"Solution","depth":2},{"id":"dense-o2o-more-objects-in-each-image","text":"Dense O2O: More Objects in Each Image","depth":3},{"id":"mal-more-positives-more-off-target-matches","text":"MAL: More Positives, More Off-Target Matches","depth":3},{"id":"the-training-schedule-warm-up-collage-then-ease-off","text":"The Training Schedule: Warm Up, Collage, Then Ease Off","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"experimental-setup","text":"Experimental Setup","depth":3},{"id":"against-d-fine-and-yolo","text":"Against D-FINE and YOLO","depth":3},{"id":"ablation-what-dense-o2o-and-mal-each-contribute","text":"Ablation: What Dense O2O and MAL Each Contribute","depth":3},{"id":"when-objects-are-packed-tight","text":"When Objects Are Packed Tight","depth":3},{"id":"reproduction-reports","text":"Reproduction Reports","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"object-detection/h-detr","relation":"referenced","topic":""},{"id":"object-detection/dino-detr","relation":"referenced","topic":""},{"id":"object-detection/rt-detr","relation":"referenced","topic":""}]},{"id":"object-detection/h-detr","category":"object-detection","legacyPath":"object-detection/2207-h-detr","legacySlug":"2207-h-detr","legacyPaths":[],"title":"H-DETR","subtitle":"The Rebellion of Silent Queries","author":"Z. Yuan","paperDate":"2022-07-26","publishedOn":"2025-07-06","paperTitle":"DETRs with Hybrid Matching","paperUrl":"https://arxiv.org/abs/2207.13080","bibliography":{"title":"DETRs with Hybrid Matching","authors":["Ding Jia","Yuhui Yuan","Haodi He","Xiaopei Wu","Haojun Yu","Weihong Lin","Lei Sun","Chao Zhang","Han Hu"],"url":"https://arxiv.org/abs/2207.13080","pdfUrl":"https://openaccess.thecvf.com/content/CVPR2023/papers/Jia_DETRs_With_Hybrid_Matching_CVPR_2023_paper.pdf","codeUrl":"https://github.com/HDETR/H-Deformable-DETR","preprint":{"id":"2207.13080","submittedOn":"2022-07-26","url":"https://arxiv.org/abs/2207.13080"},"publication":{"venue":"CVPR","year":2023,"kind":"conference","url":"https://openaccess.thecvf.com/content/CVPR2023/html/Jia_DETRs_With_Hybrid_Matching_CVPR_2023_paper.html"},"citation":"@misc{objectdetectionhdetr2022,\n title = {{DETRs with Hybrid Matching}},\n author = {Ding Jia and Yuhui Yuan and Haodi He and Xiaopei Wu and Haojun Yu and Weihong Lin and Lei Sun and Chao Zhang and Han Hu},\n year = {2022},\n eprint = {2207.13080},\n archivePrefix = {arXiv},\n url = {https://arxiv.org/abs/2207.13080}\n}\n","citationPath":"/content/papers/citations/object-detection/h-detr.822184101757.bib"},"thumbnail":{"src":"/content/papers/assets/63f1bc60b92f58d360457c0e3bbef0a3d4b3bd2a07377983b2dc5f5591ce28c2.jpg","width":1784,"height":350},"locale":"en","path":"/content/papers/en/object-detection/h-detr.005f94461932.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"solution","text":"Solution","depth":2},{"id":"main-branch-one-to-one-supervision","text":"Main Branch: One-to-One Supervision","depth":3},{"id":"auxiliary-branch-one-to-many-supervision","text":"Auxiliary Branch: One-to-Many Supervision","depth":3},{"id":"total-loss-and-implementation-details","text":"Total Loss and Implementation Details","depth":3},{"id":"more-forms-of-hybrid-supervision","text":"More Forms of Hybrid Supervision","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"coco-benchmark","text":"COCO Benchmark","depth":3},{"id":"lvis-benchmark","text":"LVIS Benchmark","depth":3},{"id":"sota-benchmark","text":"SOTA Benchmark","depth":3},{"id":"ablation-study","text":"Ablation Study","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{"auxiliary-branch-more-positive-slots":"auxiliary-branch-one-to-many-supervision","change-when-supervision-happens":"more-forms-of-hybrid-supervision","experimental-results-and-discussion":"discussion","coco-what-happens-when-more-queries-learn":"coco-benchmark","lvis-how-far-does-the-long-tail-evidence-go":"lvis-benchmark","system-comparison-the-conditions-behind-594-ap":"sota-benchmark","ap-training-time-and-inference-cost-across-the-three-schemes":"ablation-study"},"guide":null,"relatedReading":[{"id":"object-detection/deformable-detr","relation":"background","topic":"Set prediction and attention-based detection"}]},{"id":"object-detection/dino-detr","category":"object-detection","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"DINO","subtitle":"Close, but Not Close Enough","author":"Z. Yuan","paperDate":"2022-03-07","publishedOn":"2024-12-19","world":"external/dino-detector","paperTitle":"DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection","paperUrl":"https://arxiv.org/abs/2203.03605","bibliography":{"title":"DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection","authors":["Hao Zhang","Feng Li","Shilong Liu","Lei Zhang","Hang Su","Jun Zhu","Lionel M. Ni","Heung-Yeung Shum"],"url":"https://arxiv.org/abs/2203.03605","pdfUrl":"https://arxiv.org/pdf/2203.03605v4","codeUrl":"https://github.com/IDEA-Research/DINO","preprint":{"id":"2203.03605","submittedOn":"2022-03-07","url":"https://arxiv.org/abs/2203.03605"},"publication":{"venue":"ICLR","year":2023,"kind":"conference","url":"https://iclr.cc/virtual/2023/poster/11884"},"citation":"@misc{objectdetectiondinodetr2022,\n title = {{DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection}},\n author = {Hao Zhang and Feng Li and Shilong Liu and Lei Zhang and Hang Su and Jun Zhu and Lionel M. Ni and Heung-Yeung Shum},\n year = {2022},\n eprint = {2203.03605},\n archivePrefix = {arXiv},\n url = {https://arxiv.org/abs/2203.03605}\n}\n","citationPath":"/content/papers/citations/object-detection/dino-detr.d8c8af775089.bib"},"thumbnail":{"src":"/content/papers/assets/6c328e7125d4762c3a4346a1c537cb4e917d47a640869bbf212de53f74ba64fc.png","width":942,"height":319},"locale":"en","path":"/content/papers/en/object-detection/dino-detr.fcb370e89c60.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"solution","text":"Solution","depth":2},{"id":"the-overall-pipeline","text":"The Overall Pipeline","depth":3},{"id":"contrastive-denoising","text":"Contrastive Denoising","depth":3},{"id":"mixed-query-selection","text":"Mixed Query Selection","depth":3},{"id":"look-forward-twice","text":"Look Forward Twice","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"12-epochs","text":"12 Epochs","depth":3},{"id":"training-longer","text":"Training Longer","depth":3},{"id":"that-spot-on-the-leaderboard","text":"That Spot on the Leaderboard","depth":3},{"id":"ablation-how-the-494-stacks-up","text":"Ablation: How the 49.4 Stacks Up","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"contrastive-learning/dino","relation":"referenced","topic":""},{"id":"object-detection/detr","relation":"referenced","topic":""},{"id":"object-detection/deformable-detr","relation":"referenced","topic":""}]},{"id":"object-detection/rt-detr","category":"object-detection","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"RT-DETR","subtitle":"Putting NMS on the Clock","author":"Z. Yuan","paperDate":"2023-04-17","publishedOn":"2025-01-29","world":"external/rt-detr","paperTitle":"DETRs Beat YOLOs on Real-time Object Detection","paperUrl":"https://arxiv.org/abs/2304.08069","bibliography":{"title":"DETRs Beat YOLOs on Real-time Object Detection","authors":["Yian Zhao","Wenyu Lv","Shangliang Xu","Jinman Wei","Guanzhong Wang","Qingqing Dang","Yi Liu","Jie Chen"],"url":"https://arxiv.org/abs/2304.08069","pdfUrl":"https://arxiv.org/pdf/2304.08069v3","codeUrl":"https://github.com/lyuwenyu/RT-DETR","preprint":{"id":"2304.08069","submittedOn":"2023-04-17","url":"https://arxiv.org/abs/2304.08069"},"publication":{"venue":"CVPR","year":2024,"kind":"conference","url":"https://openaccess.thecvf.com/content/CVPR2024/html/Zhao_DETRs_Beat_YOLOs_on_Real-time_Object_Detection_CVPR_2024_paper.html"},"citation":"@misc{objectdetectionrtdetr2023,\n title = {{DETRs Beat YOLOs on Real-time Object Detection}},\n author = {Yian Zhao and Wenyu Lv and Shangliang Xu and Jinman Wei and Guanzhong Wang and Qingqing Dang and Yi Liu and Jie Chen},\n year = {2023},\n eprint = {2304.08069},\n archivePrefix = {arXiv},\n url = {https://arxiv.org/abs/2304.08069}\n}\n","citationPath":"/content/papers/citations/object-detection/rt-detr.cbe42f7a5b6c.bib"},"thumbnail":{"src":"/content/papers/assets/7e2de3c38daff439f51d86ec04a7bd391c3f708b817ae539f3c4df7c5546f4d6.png","width":1360,"height":332},"locale":"en","path":"/content/papers/en/object-detection/rt-detr.f2b9f4caa500.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"how-much-time-does-nms-take","text":"How Much Time Does NMS Take?","depth":3},{"id":"an-end-to-end-stopwatch","text":"An End-to-End Stopwatch","depth":3},{"id":"where-is-detr-slow","text":"Where Is DETR Slow?","depth":3},{"id":"solution","text":"Solution","depth":2},{"id":"the-big-picture","text":"The Big Picture","depth":3},{"id":"the-encoders-arithmetic","text":"The Encoder's Arithmetic","depth":3},{"id":"aifi-self-attention-only-in-s5","text":"AIFI: Self-Attention Only in S5","depth":3},{"id":"ccff-cross-scale-fusion-by-convolution","text":"CCFF: Cross-Scale Fusion by Convolution","depth":3},{"id":"picking-300-queries","text":"Picking 300 Queries","depth":3},{"id":"the-decoder-dropping-the-last-few-layers","text":"The Decoder: Dropping the Last Few Layers","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"against-yolo-and-dino","text":"Against YOLO and DINO","depth":3},{"id":"how-should-the-thresholds-be-set","text":"How Should the Thresholds Be Set?","depth":3},{"id":"on-another-machine","text":"On Another Machine","depth":3},{"id":"duplicate-boxes-still-turn-up","text":"Duplicate Boxes Still Turn Up","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"object-detection/detr","relation":"referenced","topic":""},{"id":"object-detection/deformable-detr","relation":"referenced","topic":""},{"id":"feature-fusion/panet","relation":"referenced","topic":""}]}],"text":"A photo has 3 annotated objects, and RT-DETR has prepared 300 queries. After one-to-one matching, only 3 queries get to learn where a box should go; the training target for the remaining 297 is \"no object.\"\n\nYOLO doesn't have this problem. It uses one-to-many label assignment, where one object claims several nearby predictions at once, so the supervision is much denser; the price is that at inference the same object produces several boxes, which NMS has to clean up.\n\nOn the DETR side, recent efforts to bring the positives back have mostly bolted something onto the model. [H-DETR](/blog/papers/object-detection/h-detr) opens an extra group of one-to-many queries, Group DETR opens several query groups that are each matched separately, and [DINO](/blog/papers/object-detection/dino-detr) adds a batch of denoising exercises. All this extra supervision is removed at inference, but training has to compute it on top of everything else.\n\nIn December 2024, authors from Intellindust AI Lab, City University of Hong Kong and other institutions proposed DEIM, which works on the data instead. The matching rule stays the same and so does the model; a training image simply gets a few more objects, and one-to-one matching naturally yields a few more positives. It borrows Mosaic and Mixup, two data augmentations the YOLO series has long used. The paper was later published at CVPR 2025.\n\n## Problem Definition\n\n### Only as Many Positives as Objects\n\nFirst, let's write both kinds of assignment as the same formula. An image has $N$ annotated objects, the $i$-th object is matched to $M_i$ predictions, and training sums the loss $f$ over every matched pair:\n\n$$\n\\text{loss} = \\sum_{i=1}^{N} \\sum_{j=1}^{M_i} f(\\hat{y}_{ij}, y_i)\n$$\n\nOne-to-many detectors have $M_i > 1$. DETR's one-to-one matching with the Hungarian algorithm amounts to fixing every $M_i = 1$, so the number of positives in an image is simply $N$, one for each annotated object in the photo. That stays true even with 300 queries.\n\nWhile training RT-DETRv2 (ResNet-50), the authors counted how many positives each image got over one COCO epoch, once with the Hungarian algorithm and once with YOLOX's SimOTA (Figure 3 in the paper). RT-DETRv2 is an improved version of [RT-DETR](/blog/papers/object-detection/rt-detr) that Baidu released in July 2024; SimOTA is a one-to-many assignment that first estimates, from the IoU of the predicted boxes near each object, how many predictions that object should get, a number denoted $k$, and then takes the $k$ predictions with the lowest matching cost as positives.\n\nThe one-to-one distribution piles up into a sharp peak at fewer than 10 positives per image. SimOTA's distribution is spread wide, with some photos getting more than 80 positives each. Divide one by the other and the ratio most often lands around 7 times, and about 10 times in extreme cases.\n\nFigure 3 counts only the positives produced by the matching itself. RT-DETRv2 also trains with a batch of [DN-DETR](/blog/papers/object-detection/dn-detr)-style denoising queries, which DEIM keeps too; that is a separate source of extra supervision.\n\n### Matched, but Off Target\n\nThe authors see a second reason for DETR's slow convergence, which they call low-quality matches.\n\nDetectors like YOLO pack thousands of prediction positions densely across their feature maps (the paper says dense anchors usually number more than 8,000), so every object has a few positions close enough to it. DETR has only 100 or 300 queries, which start out unaligned with where the objects are. DINO and RT-DETR later switched to initializing the queries from the highest-scoring positions of the encoder output; the queries did get closer to the objects, but they tend to crowd around a few prominent ones.\n\nInconspicuous objects have no queries nearby, yet one-to-one matching requires every object to get one. The query that ends up matched to such an object may have a box with a very low IoU against it, and still a high confidence score.\n\n### How Earlier Work Added Positives\n\nBefore DEIM, methods for adding positives all raised the number of positives per object. Group DETR opens K groups of queries during training, each doing one-to-one matching independently, so every object gets K positives; H-DETR's hybrid matching works in a similar way. Co-DETR instead attaches one-to-many auxiliary heads such as Faster R-CNN and FCOS after the encoder.\n\nThe authors raise two objections to these methods. First, they have to run extra decoder groups or auxiliary heads, which raises the training cost; second, with more one-to-many supervision, the model may again learn to output several high-scoring boxes for the same object, back to the old problem of traditional detectors.\n\n\n> [!TIP]\n> \n> For H-DETR's hybrid matching, and how a mask keeps the two query groups apart, see our note:\n> \n> - **[[22.07] H-DETR: The Rebellion of Silent Queries](/blog/papers/object-detection/h-detr)**\n\n\nSo can we get more positives without adding queries or auxiliary heads, and without touching one-to-one matching? And once there are more positives, what should be done about the matches that are off target?\n\n## Solution\n\nDEIM's two components answer these two questions. Dense O2O provides more positives, and the Matchability-Aware Loss (MAL) handles the low-quality matches that come with them. Both change only the training; at inference, the model, the number of queries and the output pipeline are all the same as before.\n\n### Dense O2O: More Objects in Each Image\n\nBack to the formula above. One-to-many raises $M_i$; Dense O2O keeps $M_i$ at 1 and raises $N$ instead.\n\n§3.2 of the paper explains with an example. Shrink a photo containing a single dog, copy it into the four tiles of a 2×2 grid, and stitch them back to the original size; one dog becomes four, so one-to-one matching yields 4 positives, as many as the 4 positives one-to-many would assign to the original dog.\n\nActual training doesn't copy the same photo four times. The authors use two data augmentations common in the YOLO series instead, configured in the official code as follows:\n\n- **Mosaic**: Triggered with probability 0.5 for each training image. Take the current image plus 3 others drawn at random from the dataset, resize each so its short side is 320 pixels, paste them into the four tiles of a 2×2 grid, apply one random affine transform to the whole thing (rotation ±10°, translation 10%, scaling 0.5 to 1.5 times), and finally resize it to 640×640.\n- **Mixup**: Triggered with probability 0.5 for each batch. Each image in the batch is blended with the previous image at a ratio of 0.45 to 0.55, and all the annotated boxes from both images are kept.\n\nWith this, the average number of objects per training image goes from about 10 to about 25 (Table 12 in the appendix). The authors also tried pushing the average to about 50, and AP after 24 epochs fell from 52.5 to 52.2; they attribute this to an imbalance between positives and negatives and to the shift in data distribution caused by too many objects.\n\nA reader once asked on GitHub whether the fourfold relation in the paper's figure is written into the matching code. The first author replied that it is only an example and the actual ratio isn't fixed at 4. The number of negatives is the number of queries minus the number of positives; for example, with 300 queries and 5 objects there are 295 negatives, and when Dense O2O turns that into 20 objects there are 280. [DEIM #126](https://github.com/Intellindust-AI-Lab/DEIM/issues/126)\n\n\n> [!TIP]\n> \n> The Mosaic the paper cites comes from YOLOv4; for what four images stitched into one look like, see our note:\n> \n> - **[[20.04] YOLOv4: Model Design Overview](/blog/papers/object-detection/yolov4)**\n\n\nWhat about the cost? Neither the matching nor the model changes, so the computation on the GPU stays the same. On a single RTX 4090, one epoch of RT-DETRv2-R50 takes 1.181 GPU hours, and 1.183 with DEIM (Table 7). The extra work falls on data loading, since the collaging and blending happen on the CPU; the official repository later added a cache for Mosaic, and the README says this made data loading almost 30% faster.\n\n### MAL: More Positives, More Off-Target Matches\n\nThe authors point out that with more objects, the gap between prominent and inconspicuous objects widens; the total number of matches goes up, and so does the number of low-quality matches. Mosaic also shrinks each photo to roughly half its side length, which adds more small objects too.\n\nThese matches have to be handled by the classification loss. RT-DETR and D-FINE originally use VariFocal Loss (VFL). Let $p$ be the predicted confidence score and $q$ the IoU between the predicted box and its matched object:\n\n$$\n\\text{VFL}(p, q) =\n\\begin{cases}\n-q\\,\\big(q \\log p + (1-q) \\log(1-p)\\big) & q > 0 \\\\\n-\\alpha\\, p^{\\gamma} \\log(1-p) & q = 0\n\\end{cases}\n$$\n\nThe top line is for positives. It sets the classification target score to the IoU, so the confidence score learns to reflect how accurately the box is drawn; the whole term is then multiplied by $q$, so positives with more accurate boxes get larger weights. The bottom line is for negatives; like Focal Loss, it uses $p^\\gamma$ to shrink the weight of the many easy negatives, and $\\alpha$ balances positives against negatives. RT-DETRv2 uses $\\gamma = 2$ and $\\alpha = 0.75$.\n\nThe trouble comes from that multiplied $q$. For a match with an IoU of only 0.05, the whole loss is first multiplied by 0.05, so however high the confidence score, the loss stays small. In DETR, where queries are few and matching is a hard rule, the authors consider such matches more common than in traditional detectors, yet VFL all but ignores them. In addition, VFL separates positives from negatives by whether the IoU is above 0, so a query that was matched but whose box doesn't touch the object at all is treated as a negative.\n\nMAL changes the formula to this:\n\n$$\n\\text{MAL}(p, q, y) =\n\\begin{cases}\n-q^{\\gamma} \\log p - (1-q^{\\gamma}) \\log(1-p) & y = 1 \\\\\n-p^{\\gamma} \\log(1-p) & y = 0\n\\end{cases}\n$$\n\n- Positives and negatives are now separated by $y$, that is, by whether this query was matched to an object.\n- The target score for positives changes from $q$ to $q^\\gamma$, and the term is no longer multiplied by $q$.\n- The negative term drops $\\alpha$.\n- Both lines use the same $\\gamma$, which the paper sets to 1.5.\n\nLet's compute two matches. One is a low-quality match with an IoU of 0.05, the other a good match with an IoU of 0.95; we sweep the confidence score from 0.1 to 0.9 and look only at the positive line:\n\n```python\nimport math\n\ndef vfl_pos(p, q):\n # Positive case of Eq. 3: the target score is q, and the whole term is multiplied by q\n return -q * (q * math.log(p) + (1 - q) * math.log(1 - p))\n\ndef mal_pos(p, q, gamma=1.5):\n # Positive case of Eq. 4: the target score becomes q^γ, and the weight is fixed at 1\n t = q ** gamma\n return -(t * math.log(p) + (1 - t) * math.log(1 - p))\n\nfor q in (0.05, 0.95):\n print(f\"IoU = {q}, MAL target score q^1.5 = {q ** 1.5:.3f}\")\n for p in (0.1, 0.3, 0.5, 0.7, 0.9):\n print(f\" confidence {p}: VFL {vfl_pos(p, q):.3f} MAL {mal_pos(p, q):.3f}\")\n```\n\n```text\nIoU = 0.05, MAL target score q^1.5 = 0.011\n confidence 0.1: VFL 0.011 MAL 0.130\n confidence 0.3: VFL 0.020 MAL 0.366\n confidence 0.5: VFL 0.035 MAL 0.693\n confidence 0.7: VFL 0.058 MAL 1.194\n confidence 0.9: VFL 0.110 MAL 2.278\nIoU = 0.95, MAL target score q^1.5 = 0.926\n confidence 0.1: VFL 2.083 MAL 2.140\n confidence 0.3: VFL 1.104 MAL 1.141\n confidence 0.5: VFL 0.658 MAL 0.693\n confidence 0.7: VFL 0.379 MAL 0.419\n confidence 0.9: VFL 0.204 MAL 0.268\n```\n\nStart with the IoU 0.05 group. MAL's target score is 0.011, which says the model should have almost no confidence in this box. As the confidence rises from 0.1 to 0.9, MAL climbs from 0.130 to 2.278, while VFL climbs only from 0.011 to 0.110. In the IoU 0.95 group, the two are about the same.\n\nWe sweep the confidence score from 0.01 to 0.99 and draw the same two IoUs as curves. Look at the blue line in the top panel and at the distance between the two lines in the bottom panel:\n\n\n\n*Figure: Drawn for this note from the positive terms of Eq. 3 and Eq. 4 in the paper, with γ = 1.5; the paper is licensed under CC BY-NC-SA 4.0, so the original figure is not reproduced here.*\n\nIn the top panel, VFL stays close to 0 however high the confidence; MAL climbs along with the confidence and is 1.59 at p = 0.8, where VFL is only 0.08. In the bottom panel the two lines almost overlap, with the loss growing as confidence falls, so the two losses ask about the same of a good match.\n\nNow the gradients. Differentiating either loss with respect to the logit gives the form \"weight × (confidence − target score)\"; we take one point from each of the two cases above:\n\n```python\n# Continuing from the previous block. Differentiated w.r.t. the logit, both losses take the form \"weight × (confidence − target score)\"\ndef grad_vfl(p, q):\n return q * (p - q)\n\ndef grad_mal(p, q, gamma=1.5):\n return p - q ** gamma\n\nfor q, p in ((0.05, 0.8), (0.95, 0.5)):\n print(f\"IoU {q}, confidence {p}: VFL {grad_vfl(p, q):+.3f}, MAL {grad_mal(p, q):+.3f}\")\n```\n\n```text\nIoU 0.05, confidence 0.8: VFL +0.038, MAL +0.789\nIoU 0.95, confidence 0.5: VFL -0.427, MAL -0.426\n```\n\nFor the box with IoU 0.05 and confidence 0.8, MAL's gradient is about 21 times VFL's, pointing in the direction that pushes the confidence down. For the box with IoU 0.95 and confidence 0.5, both are about −0.43 and push the confidence up alike. MAL pushes harder only on low-quality matches and treats good matches as before.\n\nA query whose box is off target is taught to be unsure of itself, while a query whose box is accurate still has its confidence raised. AP evaluation ranks boxes by confidence score, so the accurate boxes move to the front, and the results at high IoU thresholds look better. This is our interpretation; the authors attribute the gain from adding MAL to better box quality. The numbers in Table 6 later point the same way, with AP75 improving after MAL is added while AP50 barely moves.\n\nAs for $\\gamma$, the authors swept four values on RT-DETRv2-R50 with 24 epochs of training (Table 5), where 1.3, 1.5, 1.8 and 2.0 give 52.2, 52.4, 52.1 and 51.9 AP respectively, and they settled on 1.5.\n\n### The Training Schedule: Warm Up, Collage, Then Ease Off\n\nDense O2O isn't on from start to finish. The authors also designed a data augmentation schedule, paired with the FlatCosine learning rate used in RTMDet, which stays flat and then decays to half with a cosine. Take the official setting for 60 epochs of RT-DETRv2-R50 training as an example (Figure 5 and Table 10 in the paper's appendix):\n\n| Epoch | Data augmentation | Learning rate |\n| --- | --- | --- |\n| 0–4 | Only flipping and resizing (DataAug Warmup) | Warmup over the first 2,000 iterations, then flat at 2e-4 |\n| 4–34 | Dense O2O: Mosaic and Mixup, plus RT-DETR's original augmentations | Flat at 2e-4 |\n| 34–58 | Back to RT-DETR's original augmentations (color jitter, zoom out, IoU crop, flip) | Cosine decay from 2e-4 to 1e-4 |\n| 58–60 | Only flipping and resizing again | 1e-4 |\n\n*Table: Compiled for this note from the official config file `deim_r50vd_60e_coco.yml` and Figure 5 in the appendix.*\n\nThe first 4 epochs deliberately leave the advanced data augmentation off. As the appendix explains, DETR's attention lacks the inductive bias of convolutions and has to learn from scratch, so it learns more easily when given simple images first. Then Dense O2O runs for 30 epochs, exactly half the total length. According to the authors' explanation on GitHub, with $E$ epochs in total, the first 4 are warmup, the next $E/2$ use Dense O2O, and the learning rate also stays flat until epoch $4 + E/2$ before it starts to decay. [DEIM #9](https://github.com/Intellindust-AI-Lab/DEIM/issues/9)\n\nWhy switch it off halfway? The paper only says that turning Dense O2O off after half the epochs gives better results, without a comparison table. Collaged images have more objects, and smaller ones, than the photos seen during validation; returning to normal photos in the second half may let the model readjust to the real distribution. In Table 12, packing in too many objects made things worse, which the authors also attributed to distribution shift. This paragraph is our speculation.\n\nSettings built on D-FINE have one more stage. D-FINE itself runs 8 extra epochs at the end of training with the advanced augmentation off, to search for the EMA decay value, and DEIM keeps this practice. So the 50 epochs in the paper's Table 1 are actually 58 in the official setting, and the 72 epochs listed for D-FINE in the same table are likewise 80 in its own setting. [DEIM #56](https://github.com/Intellindust-AI-Lab/DEIM/issues/56)\n\nThis schedule also makes the validation curves look a little odd. A user compared D-FINE and DEIM-D-FINE on their own single-class dataset and found DEIM's validation AP lagging behind throughout the early phase. The first author made two points in reply. Dense O2O is a stronger data augmentation, so validation scores are naturally lower while it is on, and the fast convergence the paper claims means reaching the same or better results in fewer epochs. Also, the 2,000-iteration learning-rate warmup was set for COCO; on that small dataset, the warmup dragged on until around epoch 100. After the user changed the warmup to 500 iterations, DEIM's best AP was 0.773 and D-FINE's 0.768. [DEIM #5](https://github.com/Intellindust-AI-Lab/DEIM/issues/5)\n\n## Discussion\n\n### Experimental Setup\n\nThe main experiments are all on COCO, training on train2017 and evaluating on val2017, with 640×640 input. The backbones are pretrained on ImageNet-1k, and apart from the Objects365 fine-tuning in Table 8, no extra data is used. Latency reuses D-FINE's end-to-end numbers measured on a T4 GPU with TensorRT FP16; since DEIM doesn't change the architecture, the same model has the same latency with or without DEIM.\n\nDEIM is applied to two base models. One is RT-DETRv2, mentioned above; the other is [D-FINE](https://arxiv.org/abs/2410.13842v1). D-FINE is a real-time DETR proposed by the University of Science and Technology of China in October 2024; built on the RT-DETR architecture, it changes box regression from directly predicting four coordinates to refining, layer by layer, a probability distribution over the position of each edge.\n\n### Against D-FINE and YOLO\n\n| Model | Epochs | AP | APS | Latency (T4) |\n| --- | ---: | ---: | ---: | ---: |\n| D-FINE-L | 72 | 54.0 | 36.5 | 8.07 ms |\n| DEIM-D-FINE-L | 50 | 54.7 | 36.9 | 8.07 ms |\n| D-FINE-X | 72 | 55.8 | 37.3 | 12.89 ms |\n| DEIM-D-FINE-X | 50 | 56.5 | 38.8 | 12.89 ms |\n| YOLO11-X | 500 | 54.1 | 37.0 | 10.52 ms |\n| YOLOv9-E | 500 | 55.6 | 40.2 | 20.53 ms |\n\n*Table: Excerpted from Table 1 of the paper, COCO val2017. According to the table note, YOLO11-X's NMS was tuned with a confidence threshold of 0.01.*\n\nAt the same latency, DEIM gives D-FINE-L and D-FINE-X 0.7 AP more each, and the epoch count in the table drops from 72 to 50, about 30% fewer, which is also what the paper's main text says. Counting the 8 EMA epochs on each side (58 vs. 80), it is about 28% fewer. The abstract's claim of \"reducing training time by 50%\" corresponds to the later RT-DETRv2 comparison of 36 vs. 72 epochs.\n\nAgainst YOLO, DEIM-D-FINE-L at 54.7 AP and 8.07 ms beats YOLO11-X at 54.1 AP and 10.52 ms. Small objects remain a weakness on the DETR side; DEIM-D-FINE-X's APS is 38.8 against 40.2 for YOLOv9-E, a gap the authors also acknowledge in the main text. Epoch counts, on the other hand, are only worth comparing within the same family, since DEIM's backbone has already been pretrained on ImageNet.\n\nThe authors also write that YOLOv10 uses a hybrid of one-to-many and one-to-one assignment, yet DEIM's models still beat it across the board, which they say proves that Dense O2O is effective. That inference is a stretch. The two sides differ in architecture, pretraining and training length, so the gap in the table can't be credited to Dense O2O alone, nor used to judge YOLOv10's dual-head design. To see what Dense O2O itself does, look at the ablations below.\n\n\n> [!TIP]\n> \n> For how YOLOv10 has a one-to-many head train alongside and uses only the one-to-one head at inference, see our note:\n> \n> - **[[24.05] YOLOv10: Goodbye NMS](/blog/papers/object-detection/yolov10)**\n\n\n### Ablation: What Dense O2O and MAL Each Contribute\n\n| Base model | Epochs | Dense O2O | MAL | AP | AP50 | AP75 |\n| --- | ---: | :---: | :---: | ---: | ---: | ---: |\n| RT-DETRv2-R50 | 72 | | | 53.4 | 71.6 | 57.4 |\n| RT-DETRv2-R50 | 36 | ✓ | | 53.6 | 71.9 | 58.2 |\n| RT-DETRv2-R50 | 36 | ✓ | ✓ | 53.9 | 71.7 | 58.6 |\n| D-FINE-L | 72 | | | 54.0 | 71.6 | 58.4 |\n| D-FINE-L | 36 | ✓ | | 54.2 | 72.1 | 58.9 |\n| D-FINE-L | 36 | ✓ | ✓ | 54.6 | 72.2 | 59.5 |\n\n*Table: Table 6 of the paper.*\n\nWith Dense O2O alone, 36 epochs catch up with and slightly exceed the baseline's 72-epoch results; adding MAL gives the two base models another 0.3 and 0.4 AP. In GPU hours (Table 7), the original RT-DETRv2-R50 reaches 53.4 in about 85 hours over 72 epochs, while DEIM reaches 54.3 in about 71 hours over 60 epochs.\n\nThis table compares DEIM at 36 epochs with the original at 72 epochs, and has no control run at the same 36 epochs without Dense O2O. According to the appendix and the official config files, the DEIM experiments also switch to the authors' full training recipe, including the new learning-rate and data augmentation schedules, unfreezing the backbone's BN, and SiLU in the decoder; according to the official config files, RT-DETRv2's base learning rate also goes up from 1e-4 to 2e-4.\n\nIn the appendix, the authors tested those minor changes on their own on D-FINE; with 36 epochs of training, D-FINE-L is unchanged and D-FINE-X goes from 55.4 to 55.5. Incidentally, D-FINE-X already reaches 55.4 after only 36 epochs, not far from the 55.8 at 72 epochs.\n\nThe comparison at equal epochs is in Table 4. RT-DETRv2-R50 is trained for 12 and for 24 epochs, changing only the Mosaic and Mixup probabilities:\n\n| Mosaic | Mixup | AP (12 ep) | AP (24 ep) | APS (24 ep) |\n| ---: | ---: | ---: | ---: | ---: |\n| 0 | 0 | 49.6 | 51.7 | 32.8 |\n| 0.5 | 0 | 50.4 | 51.9 | 34.9 |\n| 0 | 0.5 | 50.1 | 51.5 | 33.2 |\n| 0.5 | 0.5 | 50.4 | 52.5 | 34.9 |\n\n*Table: Excerpted from Table 4 of the paper.*\n\nWith both on, AP rises by 0.8 at both 12 and 24 epochs, and small-object AP rises by 2.1 at 24 epochs. Mixup alone is actually slightly lower at 24 epochs. This is the gain from Dense O2O itself under equal conditions, a cleaner comparison than Table 6, which compares across epoch counts.\n\nTable 13 in the appendix also compares AP on the training set. DEIM is slightly lower on the training set (64.8 vs. 65.1) and higher on the validation set (54.3 vs. 53.4), which the authors read as less overfitting to the training data.\n\n### When Objects Are Packed Tight\n\nCrowdHuman is a dataset of crowded scenes of people. Following D-FINE's setting, the authors trained both models for 120 epochs (Table 3); this time DEIM saved no epochs:\n\n| Method | AP | AP75 | APS |\n| --- | ---: | ---: | ---: |\n| D-FINE-L | 56.0 | 59.4 | 29.0 |\n| DEIM-D-FINE-L | 57.5 | 62.9 | 33.2 |\n\n*Table: Excerpted from Table 3 of the paper.*\n\nAP rises by 1.5, with AP75 up 3.5 and APS up 4.2, so the gains concentrate where boxes must be accurate and objects are small.\n\nA qualitative example in the appendix also concerns duplicate boxes. D-FINE-L outputs four heavily overlapping, high-confidence boxes for the same kite, while the version trained with DEIM keeps only one. This is an example from a single image, with no statistics behind it.\n\nAlso, starting from D-FINE's weights pretrained on Objects365 and fine-tuning on COCO, D-FINE-X reaches 59.3 AP after 32 epochs of fine-tuning, and 59.5 after 24 epochs of fine-tuning with DEIM (Table 8).\n\n### Reproduction Reports\n\nA user on GitHub trained DEIM-D-FINE-X on 8 A100s with the official settings. At epoch 50 it reached 55.8 AP, and after all 58 epochs 56.16, 0.3 below the paper's 56.5; the user also suspected that the batch size and learning rate hadn't been matched. The first author replied that one has to follow the official 50 plus 8 epochs exactly, and that the run-to-run variation under the same settings is about ±0.1 AP. In March 2026, another user in the same thread said they mostly reached only 54.4 with DEIM-D-FINE-L, without attaching logs. [DEIM #112](https://github.com/Intellindust-AI-Lab/DEIM/issues/112)\n\n## Conclusion\n\nDEIM tackles the two causes of DETR's slow convergence separately. For too few positives, it puts more objects in each training image, so one-to-one matching naturally yields more positives while the GPU computation stays the same. For the extra low-quality matches, MAL presses their confidence down and leaves good matches as they are. The training schedule sandwiches Dense O2O in the middle, with a warmup before it and normal photos in the second half.\n\nOn COCO, RT-DETRv2-R50 reaches 53.9 AP in 36 epochs, against 53.4 for the original in 72; D-FINE-L and D-FINE-X each gain 0.7 AP at the same latency. These comparisons also swap in the whole training recipe; for the gain from Dense O2O itself at equal epochs, look at the 0.8 AP in Table 4.\n\nIf you want to train RT-DETR or D-FINE on your own data, DEIM's changes are all on the training side, and the model and inference pipeline stay as they are. The schedule's parameters, though, are set for COCO's length; the 2,000-iteration warmup and the Dense O2O window both need recalculating for the size of your own data. Users on GitHub have also reported that on their own datasets of a few thousand to ten thousand images, DEIM did worse than YOLO ([#61](https://github.com/Intellindust-AI-Lab/DEIM/issues/61), [#106](https://github.com/Intellindust-AI-Lab/DEIM/issues/106)); these discussions haven't reached a conclusion yet.\n\nDEIM's authors later released DEIMv2 in September 2025, with the versions from S upward switching to DINOv3 features. On the YOLO side, [YOLO26](/blog/papers/object-detection/yolo26) does something similar, gradually handing the one-to-many loss weight over to the one-to-one head during training, so the extra supervision is likewise concentrated in the early part of training.\n","imageSizes":{"/content/papers/assets/a2d11ca85f71416c2b08dbb516c98670a9309d2d81e1b77d572cb54b66b0c613.png":{"width":1280,"height":1720}}}