{"notes":[{"id":"object-detection/rf-detr","category":"object-detection","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"RF-DETR","subtitle":"Train First, Pick a Size Later","author":"Z. Yuan","paperDate":"2025-11-12","publishedOn":"2026-05-12","paperTitle":"RF-DETR: Neural Architecture Search for Real-Time Detection Transformers","paperUrl":"https://arxiv.org/abs/2511.09554","bibliography":{"title":"RF-DETR: Neural Architecture Search for Real-Time Detection Transformers","authors":["Isaac Robinson","Peter Robicheaux","Matvei Popov","Deva Ramanan","Neehar Peri"],"url":"https://arxiv.org/abs/2511.09554","pdfUrl":"https://arxiv.org/pdf/2511.09554v2","codeUrl":"https://github.com/roboflow/rf-detr","preprint":{"id":"2511.09554","submittedOn":"2025-11-12","url":"https://arxiv.org/abs/2511.09554"},"publication":{"venue":"ICLR","year":2026,"kind":"conference","url":"https://iclr.cc/virtual/2026/poster/10007257"},"citation":"@misc{objectdetectionrfdetr2025,\n  title = {{RF-DETR: Neural Architecture Search for Real-Time Detection Transformers}},\n  author = {Isaac Robinson and Peter Robicheaux and Matvei Popov and Deva Ramanan and Neehar Peri},\n  year = {2025},\n  eprint = {2511.09554},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2511.09554}\n}\n","citationPath":"/content/papers/citations/object-detection/rf-detr.f78ec4489185.bib"},"thumbnail":{"src":"/content/papers/assets/10f93bab733752bf97f8b3ed5c86569a2c4d85840b99b338a6443431fb506555.png","width":1080,"height":775},"locale":"en","path":"/content/papers/en/object-detection/rf-detr.0bb46b8eb0c1.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"tuned-too-closely-to-coco","text":"Tuned Too Closely to COCO","depth":3},{"id":"each-size-fits-one-piece-of-hardware","text":"Each Size Fits One Piece of Hardware","depth":3},{"id":"solution","text":"Solution","depth":2},{"id":"built-on-lw-detr","text":"Built on LW-DETR","depth":3},{"id":"five-knobs","text":"Five Knobs","depth":3},{"id":"how-many-tokens-each-size-sees","text":"How Many Tokens Each Size Sees","depth":3},{"id":"picking-after-training","text":"Picking After Training","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"first-calibrate-the-ruler-for-latency","text":"First, Calibrate the Ruler for Latency","depth":3},{"id":"coco","text":"COCO","depth":3},{"id":"rf100-vl","text":"RF100-VL","depth":3},{"id":"ablation-where-the-extra-2-ap-comes-from","text":"Ablation: Where the Extra 2 AP Comes From","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"object-detection/detr","relation":"referenced","topic":""},{"id":"object-detection/deformable-detr","relation":"referenced","topic":""},{"id":"contrastive-learning/dinov2","relation":"referenced","topic":""}]},{"id":"object-detection/detr","category":"object-detection","legacyPath":"object-detection/2005-detr","legacySlug":"2005-detr","legacyPaths":[],"title":"DETR","subtitle":"A Cross-Domain Pioneer","author":"Z. Yuan","paperDate":"2020-05-26","publishedOn":"2024-06-04","paperTitle":"End-to-End Object Detection with Transformers","paperUrl":"https://www.ecva.net/papers/eccv_2020/papers_ECCV/html/832_ECCV_2020_paper.php","bibliography":{"title":"End-to-End Object Detection with Transformers","authors":["Nicolas Carion","Francisco Massa","Gabriel Synnaeve","Nicolas Usunier","Alexander Kirillov","Sergey Zagoruyko"],"url":"https://www.ecva.net/papers/eccv_2020/papers_ECCV/html/832_ECCV_2020_paper.php","pdfUrl":"https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123460205.pdf","codeUrl":"https://github.com/facebookresearch/detr","preprint":{"id":"2005.12872","submittedOn":"2020-05-26","url":"https://arxiv.org/abs/2005.12872"},"publication":{"venue":"ECCV","year":2020,"kind":"conference","url":"https://www.ecva.net/papers/eccv_2020/papers_ECCV/html/832_ECCV_2020_paper.php"},"citation":"@misc{objectdetectiondetr2020,\n  title = {{End-to-End Object Detection with Transformers}},\n  author = {Nicolas Carion and Francisco Massa and Gabriel Synnaeve and Nicolas Usunier and Alexander Kirillov and Sergey Zagoruyko},\n  year = {2020},\n  eprint = {2005.12872},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2005.12872}\n}\n","citationPath":"/content/papers/citations/object-detection/detr.5258ffe0b9a0.bib"},"thumbnail":{"src":"/content/papers/assets/dd7e8441e6460ff562cf558374789924518fafaed7ecde0e3e35ee778c316330.jpg","width":1024,"height":266},"locale":"en","path":"/content/papers/en/object-detection/detr.493e749cd884.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"solution","text":"Solution","depth":2},{"id":"model-architecture","text":"Model Architecture","depth":3},{"id":"loss-design","text":"Loss Design","depth":3},{"id":"dataset","text":"Dataset","depth":3},{"id":"technical-details","text":"Technical Details","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"how-does-it-perform-this-time","text":"How does it perform this time?","depth":3},{"id":"component-analysis-of-the-model","text":"Component Analysis of the Model","depth":3},{"id":"model-behavior-analysis","text":"Model Behavior Analysis","depth":3},{"id":"generalization-to-unseen-object-counts","text":"Generalization to Unseen Object Counts","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{"defining-the-problem":"problem-definition","solving-the-problem":"solution","from-an-image-to-a-set-of-objects":"model-architecture","match-first-then-compute-the-loss":"loss-design","experimental-results-and-discussion":"discussion","how-did-it-do-start-with-the-conditions":"how-does-it-perform-this-time","which-components-matter":"component-analysis-of-the-model","what-did-the-queries-learn":"model-behavior-analysis"},"guide":null,"relatedReading":[{"id":"object-detection/deformable-detr","relation":"next","topic":"Set prediction and attention-based detection"},{"id":"object-detection/smca-detr","relation":"comparison","topic":"Faster convergence for Transformer detectors"}]},{"id":"object-detection/deformable-detr","category":"object-detection","legacyPath":"object-detection/2010-deformable-detr","legacySlug":"2010-deformable-detr","legacyPaths":[],"title":"Deformable DETR","subtitle":"Deformable Attention Mechanism","author":"Z. Yuan","paperDate":"2020-10-08","publishedOn":"2024-09-20","paperTitle":"Deformable DETR: Deformable Transformers for End-to-End Object Detection","paperUrl":"https://arxiv.org/abs/2010.04159","bibliography":{"title":"Deformable DETR: Deformable Transformers for End-to-End Object Detection","authors":["Xizhou Zhu","Weijie Su","Lewei Lu","Bin Li","Xiaogang Wang","Jifeng Dai"],"url":"https://arxiv.org/abs/2010.04159","pdfUrl":"https://arxiv.org/pdf/2010.04159","codeUrl":null,"preprint":{"id":"2010.04159","submittedOn":"2020-10-08","url":"https://arxiv.org/abs/2010.04159"},"publication":{"venue":"ICLR","year":2021,"kind":"conference","url":"https://arxiv.org/abs/2010.04159"},"citation":"@misc{objectdetectiondeformabledetr2020,\n  title = {{Deformable DETR: Deformable Transformers for End-to-End Object Detection}},\n  author = {Xizhou Zhu and Weijie Su and Lewei Lu and Bin Li and Xiaogang Wang and Jifeng Dai},\n  year = {2020},\n  eprint = {2010.04159},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2010.04159}\n}\n","citationPath":"/content/papers/citations/object-detection/deformable-detr.df721a30472e.bib"},"thumbnail":{"src":"/content/papers/assets/36edfc5595a85c73fcdc9a4bbb9e1ca512e1f100676eb7c37203d71c84905c05.jpg","width":1224,"height":752},"locale":"en","path":"/content/papers/en/object-detection/deformable-detr.68972544ff9b.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"problem-solution","text":"Problem Solution","depth":2},{"id":"deformable-attention","text":"Deformable Attention","depth":3},{"id":"multi-scale-computation","text":"Multi-Scale Computation","depth":3},{"id":"model-architecture","text":"Model Architecture","depth":3},{"id":"code-breakdown","text":"Code Breakdown","depth":3},{"id":"training-strategy","text":"Training Strategy","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"comparison-with-detr","text":"Comparison with DETR","depth":3},{"id":"ablation-studies","text":"Ablation Studies","depth":3},{"id":"sota-comparison","text":"SOTA Comparison","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"object-detection/detr","relation":"background","topic":"Set prediction and attention-based detection"},{"id":"object-detection/h-detr","relation":"next","topic":"Set prediction and attention-based detection"},{"id":"object-detection/smca-detr","relation":"comparison","topic":"Faster convergence for Transformer detectors"}]},{"id":"contrastive-learning/dinov2","category":"contrastive-learning","legacyPath":"contrastive-learning/2304-dinov2","legacySlug":"2304-dinov2","legacyPaths":[],"title":"DINOv2","subtitle":"Universal Feature Forging Method","author":"Z. Yuan","paperDate":"2023-04-14","publishedOn":"2025-07-01","paperTitle":"DINOv2: Learning Robust Visual Features without Supervision","paperUrl":"https://arxiv.org/abs/2304.07193","bibliography":{"title":"DINOv2: Learning Robust Visual Features without Supervision","authors":["Maxime Oquab","Timothée Darcet","Théo Moutakanni","Huy V. Vo","Marc Szafraniec","Vasil Khalidov","Pierre Fernandez","Daniel Haziza","Francisco Massa","Alaaeldin El-Nouby","Mahmoud Assran","Nicolas Ballas","Wojciech Galuba","Russell Howes","Po-Yao Huang","Shang-Wen Li","Ishan Misra","Michael Rabbat","Vasu Sharma","Gabriel Synnaeve","Hu Xu","Hervé Jégou","Julien Mairal","Patrick Labatut","Armand Joulin","Piotr Bojanowski"],"url":"https://arxiv.org/abs/2304.07193","pdfUrl":"https://arxiv.org/pdf/2304.07193","codeUrl":"https://github.com/facebookresearch/dinov2","preprint":{"id":"2304.07193","submittedOn":"2023-04-14","url":"https://arxiv.org/abs/2304.07193"},"publication":{"venue":"Transactions on Machine Learning Research","shortVenue":"TMLR","year":2024,"kind":"journal","url":"https://openreview.net/forum?id=a68SUt6zFt"},"citation":"@misc{contrastivelearningdinov22023,\n  title = {{DINOv2: Learning Robust Visual Features without Supervision}},\n  author = {Maxime Oquab and Timothée Darcet and Théo Moutakanni and Huy V. Vo and Marc Szafraniec and Vasil Khalidov and Pierre Fernandez and Daniel Haziza and Francisco Massa and Alaaeldin El-Nouby and Mahmoud Assran and Nicolas Ballas and Wojciech Galuba and Russell Howes and Po-Yao Huang and Shang-Wen Li and Ishan Misra and Michael Rabbat and Vasu Sharma and Gabriel Synnaeve and Hu Xu and Hervé Jégou and Julien Mairal and Patrick Labatut and Armand Joulin and Piotr Bojanowski},\n  year = {2023},\n  eprint = {2304.07193},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2304.07193}\n}\n","citationPath":"/content/papers/citations/contrastive-learning/dinov2.e904e3854892.bib"},"thumbnail":{"src":"/content/papers/assets/9c68ef105804d4462aee8950a1bfd9f0929511f494b1a29690d857bf931da892.jpg","width":1224,"height":368},"locale":"en","path":"/content/papers/en/contrastive-learning/dinov2.2d46c57f4666.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"data-collection","text":"Data Collection","depth":2},{"id":"1-preparing-the-data-pool","text":"1. Preparing the Data Pool","depth":3},{"id":"2-removing-duplicates","text":"2. Removing Duplicates","depth":3},{"id":"3-self-supervised-retrieval","text":"3. Self-Supervised Retrieval","depth":3},{"id":"engineering-scale-and-efficiency","text":"Engineering Scale and Efficiency","depth":3},{"id":"solution","text":"Solution","depth":2},{"id":"image-level-feature-alignment","text":"Image-Level Feature Alignment","depth":3},{"id":"patch-level-prediction-task","text":"Patch-Level Prediction Task","depth":3},{"id":"head-design","text":"Head Design","depth":3},{"id":"sinkhorn-knopp-centering","text":"Sinkhorn-Knopp Centering","depth":3},{"id":"koleo-regularization","text":"KoLeo Regularization","depth":3},{"id":"high-resolution-training-phase","text":"High-Resolution Training Phase","depth":3},{"id":"engineering-implementation","text":"Engineering Implementation","depth":2},{"id":"flashattention","text":"FlashAttention","depth":3},{"id":"sequence-packing","text":"Sequence Packing","depth":3},{"id":"stochastic-depth","text":"Stochastic Depth","depth":3},{"id":"distributed-training","text":"Distributed Training","depth":3},{"id":"model-distillation","text":"Model Distillation","depth":3},{"id":"ablation-experiments","text":"Ablation Experiments","depth":2},{"id":"model-training-process","text":"Model Training Process","depth":3},{"id":"pretraining-data","text":"Pretraining Data","depth":3},{"id":"importance-of-loss-components","text":"Importance of Loss Components","depth":3},{"id":"knowledge-distillation-effect","text":"Knowledge Distillation Effect","depth":3},{"id":"high-resolution-training","text":"High-Resolution Training","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"imagenet-classification","text":"ImageNet Classification","depth":3},{"id":"video-classification","text":"Video Classification","depth":3},{"id":"instance-recognition","text":"Instance Recognition","depth":3},{"id":"semantic-segmentation","text":"Semantic Segmentation","depth":3},{"id":"qualitative-results","text":"Qualitative Results","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"contrastive-learning/dino","relation":"background","topic":"Self-distillation for visual representations"}]}],"text":"When choosing a real-time object detector, many people first look at the COCO leaderboard and pick a model family with good-looking scores; then, among sizes such as N, S, M and L, they find one that runs fast enough on their GPU; finally, they fine-tune it on their own data.\n\nThis paper takes issue with every step. A ranking on COCO may not hold up on your own dataset; the few sizes cut in advance may not fit your hardware exactly; and as for \"fast enough,\" papers don't agree on how to measure latency.\n\nIf you haven't heard of Roboflow, here is some background. It is an American company that runs a platform for computer vision annotation, training and deployment, and the datasets its users share publicly are collected in its Roboflow Universe. In March 2025, the company announced RF-DETR, a real-time detector from the DETR family, on its blog. It takes an image and directly outputs each object's box, class and confidence score, with no NMS in between, and the code is open source under the Apache 2.0 license. [Roboflow, 2025-03-20](https://blog.roboflow.com/rf-detr/)\n\nThe paper only went up on arXiv in November 2025; its authors are from Roboflow and Carnegie Mellon University, and it was later published at ICLR 2026. It explains where that lineup of model sizes came from, namely the neural architecture search (NAS) in the title. Roboflow itself runs a model training platform and is a direct competitor of Ultralytics, which publishes the YOLO series.\n\n## Problem Definition\n\n### Tuned Too Closely to COCO\n\nThe authors argue that recent specialist detectors, which only recognize a fixed set of classes, have been tuned to COCO in everything from the architecture and learning-rate schedule to the data augmentation. On a dataset whose objects per image, number of classes and amount of data all differ from COCO's, these choices may no longer fit.\n\nYOLOv8 in the paper is one example. Under the authors' measurements, scaling YOLOv8 from N to M raises its COCO AP from 35.2 to 47.3; on RF100-VL the same sizes rise only from 55.0 to 56.5, and scaling up to XL still gives 56.5.\n\nRF100-VL is a benchmark these authors compiled in 2025 from 100 datasets picked from Roboflow Universe; a model is fine-tuned and evaluated on each dataset separately, and the scores are averaged. The authors treat it as a proxy for \"transferability to any target domain.\"\n\nAnother route is open-vocabulary vision-language models such as GroundingDINO. Fine-tuned on the target data, these models do very well, reaching 62.3 AP on RF100-VL, but they take 309.9 ms per image (GroundingDINO doesn't support TensorRT, so this is PyTorch latency). The authors also note that fine-tuning weakens its original open-vocabulary ability.\n\nThe training recipe carries assumptions of its own. A cosine learning-rate schedule needs to know the total number of training steps in advance, and data augmentation presumes what the data looks like. The authors' example is a pedestrian detector for self-driving cars; if vertical flips are used in training, the model may also take reflections in puddles for pedestrians.\n\n### Each Size Fits One Piece of Hardware\n\nThe other problem is size. In existing detector families, such as D-FINE's N to X, each size is designed by hand and trained separately. When the target hardware changes, you can only pick the closest of the ready-made sizes, or redesign and retrain.\n\nFinding different accuracy–latency trade-offs automatically is exactly what NAS is for. Early hardware-aware NAS had to search and retrain for every new kind of hardware; OFA (Once-for-All), from 2019, decoupled training from search, training thousands of weight-sharing subnetworks within one network at once and picking among them as needed after training. NAS for object detection mostly just swaps the backbone for a searched network and leaves the rest of the detector as it was.\n\nSo this paper sets out to answer two questions. Can a single training run on the target dataset yield a whole accuracy–latency curve, from which you pick a point to suit your hardware? And how should the latency of that point be measured so that it can sit next to other people's numbers?\n\n## Solution\n\n### Built on LW-DETR\n\nRF-DETR is built on LW-DETR, which Baidu proposed in 2024. LW-DETR stacks a ViT encoder, a projector and a very shallow DETR decoder, and its ViT interleaves window attention with global attention to save computation.\n\n[![RF-DETR architecture diagram. On the left, the image is cut into patches, positional encodings are added, and the result goes into the ViT backbone, which has four blocks, each made of two window-attention layers plus one global-attention layer; the outputs of the four blocks enter the projector, then pass through Query Selection, are added to the query embeddings and go into the decoder; each decoder layer consists of self-attention, deformable cross-attention and a feed-forward layer in turn, with Decoder Groups 1 to 13 stacked behind it; at the upper right are the detection head and the segmentation head](/content/papers/assets/10f93bab733752bf97f8b3ed5c86569a2c4d85840b99b338a6443431fb506555.png)](/content/papers/assets/10f93bab733752bf97f8b3ed5c86569a2c4d85840b99b338a6443431fb506555.png)\n\n*Figure 2 from the paper, excerpted from [**page 4 of the v2 PDF**](https://arxiv.org/pdf/2511.09554v2#page=4), CC BY 4.0. The ViT backbone is on the left, the decoder in the middle, the segmentation head on the right, and the two kinds of output at the top.*\n\nStart at the lower left. The image is cut into patches, positional encodings are added, and the result goes into the ViT backbone. The backbone in the figure has four blocks, each doing two layers of window attention followed by one layer of global attention. The outputs of all four blocks go into the projector together, which turns them into the feature map the decoder uses.\n\nNext comes Query Selection. Every position of the projector output first predicts its own set of class scores and a candidate box, and the 300 highest-scoring positions become the starting points of the decoder queries. This is the two-stage approach introduced by Deformable DETR.\n\nIn each decoder layer, the queries first do self-attention among themselves; then deformable cross-attention samples a few points on the feature map near each candidate box; finally comes a feed-forward network. The output of every layer is connected to a detection head and gets its own loss, and the knob for cutting decoder layers later relies on exactly this.\n\nThe 13 decoder groups stacked on the right of the figure are Group DETR, carried over from LW-DETR. Training prepares 13 groups of queries that share the same decoder weights, and each group is matched one to one with the annotations on its own, so every object gets 13 positives per iteration; at inference only one group is kept, so NMS is still unnecessary. The official code also defaults to 13 groups.\n\nAdding a lightweight segmentation head to the same architecture gives RF-DETR-Seg; this note covers detection only.\n\nRF-DETR makes three main changes to LW-DETR:\n\n- **The backbone becomes DINOv2**. LW-DETR originally used CAEv2. DINOv2 is a ViT that Meta pretrained with self-supervision, and the authors found that initializing from its weights clearly improves detection accuracy on small datasets. Speed is the price, since DINOv2 has 12 layers against CAEv2's 10, and the later NAS is left to win this latency back. On COCO, N through L use ViT-S, and XL and 2XL use ViT-B.\n- **The projector switches to LayerNorm**. This lets consumer GPUs reach large batch sizes through gradient accumulation too. BatchNorm's statistics only see the small batch actually fed in each time, so they become inaccurate under gradient accumulation.\n- **The training recipe gets simpler**. The learning rate drops from LW-DETR's 4e-4 to 1e-4, with no learning-rate schedule and no warm-up. Data augmentation keeps only horizontal flips and random crops; multi-scale resizing now resizes the whole batch together, which reduces the padding pixels.\n\n\n> [!TIP]\n> \n> RF-DETR's decoder carries over DETR's one-to-one matching and Deformable DETR's deformable attention, and its backbone is DINOv2; for background, see:\n> \n> - **[[20.05] DETR: A Cross-Domain Pioneer](/blog/papers/object-detection/detr)**\n> - **[[20.10] Deformable DETR: Deformable Attention Mechanism](/blog/papers/object-detection/deformable-detr)**\n> - **[[23.04] DINOv2: Universal Feature Forging Method](/blog/papers/contrastive-learning/dinov2)**\n\n\n### Five Knobs\n\nWith the base settled, the authors added five knobs that can be turned. Each panel of the figure has an arrow labeled faster at one end and more accurate at the other:\n\n[![Five diagrams of the NAS search space. a is patch size, where small 8×8 patches produce more tokens and are more accurate, and large 32×32 patches are faster; b is the number of decoder layers, with fewer layers faster and more layers more accurate; c is dropping some of the queries; d is the input resolution; e is the number of windows, with the grid on the right split into four windows of different colors](/content/papers/assets/f0e8bf4f9ecad4cb2f090117a18034f54eab8bc3e901414f94864de103d3425a.png)](/content/papers/assets/f0e8bf4f9ecad4cb2f090117a18034f54eab8bc3e901414f94864de103d3425a.png)\n\n*Figure 3 from the paper, excerpted from [**page 5 of the v2 PDF**](https://arxiv.org/pdf/2511.09554v2#page=5), CC BY 4.0. The five knobs are patch size, number of decoder layers, number of queries, resolution and number of windows.*\n\n- **Patch size** (a). The smaller the patch, the more tokens it produces, and the more accurate and the slower the model. Training uses FlexiViT's method to resize the same set of patch embedding weights to different sizes.\n- **Decoder layers** (b). Every layer has its own loss, so any number of layers can be cut from the end at inference. Cutting all of them works too; then only the 300 candidate boxes picked by Query Selection remain, and the model becomes a single-stage detector that needs no NMS.\n- **Number of queries** (c). The queries are sorted by the scores Query Selection gives them, and the lowest-scoring ones are simply dropped at inference. The number of queries is also the maximum number of objects that can be output per image.\n- **Resolution** (d). The higher the resolution, the easier small objects are to recognize, and the slower the model. The positional encoding is allocated up front as one full table sized by \"maximum resolution ÷ minimum patch,\" and other combinations are interpolated from that table.\n- **Number of windows** (e). Window attention lets tokens see only other tokens in the same window. The more windows, the less computation, and the harder it is for information to flow between windows.\n\nDuring training, each step samples one configuration uniformly from the grid and uses it for one forward pass and gradient update. The training grid in Appendix A has resolutions from 320 to 960 in steps of 64, patches of 8, 10, 12, 16, 20, 24 and 32, and window counts of 1, 2 and 4. The decoder depth and the number of queries are fixed at 6 layers and 300 queries during training and only cut down at inference.\n\nThe authors liken this training to the ensemble view of dropout, where each step updates only one subnetwork and thousands of subnetworks learn together. They also found that randomly switching architectures has a regularizing effect in itself, so they call it \"architecture augmentation.\" The ablation later shows the evidence.\n\n\n> [!TIP]\n> \n> For how one set of weights adapts to different patch sizes, see our FlexiViT note:\n> \n> - **[[22.12] FlexiViT: Flexible Patch Sizes](/blog/papers/vision-transformers/flexivit)**\n\n\n### How Many Tokens Each Size Sees\n\nTable 7 in the appendix lists the configurations finally picked on COCO. Let's count how many tokens each configuration actually feeds into the ViT.\n\nTable 7 gives the number of windows as 2. The official code treats this number as the windows per side, so there are actually 2×2 = 4 windows, matching the four drawn in Figure 3(e).\n\n```python\n# Table 7: Pareto configurations for COCO detection (resolution, patch, windows per side)\nconfigs = {\n    \"N\": (384, 16, 2), \"S\": (512, 16, 2), \"M\": (576, 16, 2),\n    \"L\": (704, 16, 2), \"XL\": (700, 20, 1), \"2XL\": (880, 20, 2),\n}\nfor name, (res, patch, win) in configs.items():\n    side = res // patch\n    tokens = side * side\n    print(f\"{name:>3}: {res}/{patch} = {side:>2}, {tokens:>4} tokens in total, \"\n          f\"windows: {win * win}, {tokens // (win * win):>4} per window\")\n\n# Appendix L: fix the resolution at 640 and change only the patch to get a similar number of cells per side\nfor name, patch in [(\"N\", 27), (\"S\", 21), (\"M\", 18)]:\n    print(f\"640/{patch} = {640 / patch:.1f} ({name} had {configs[name][0] // configs[name][1]})\")\n\n# The positional encoding is preallocated by \"maximum resolution ÷ minimum patch\" (Appendix A training grid)\nprint(\"Positional encoding grid:\", 960 // 8, \"×\", 960 // 8)\n# Appendix A: the number of combinations enumerated at inference\nprint(\"Combinations:\", 11 * 7 * 7 * 3 * 4)\n```\n\n```text\n  N: 384/16 = 24,  576 tokens in total, windows: 4,  144 per window\n  S: 512/16 = 32, 1024 tokens in total, windows: 4,  256 per window\n  M: 576/16 = 36, 1296 tokens in total, windows: 4,  324 per window\n  L: 704/16 = 44, 1936 tokens in total, windows: 4,  484 per window\n XL: 700/20 = 35, 1225 tokens in total, windows: 1, 1225 per window\n2XL: 880/20 = 44, 1936 tokens in total, windows: 4,  484 per window\n640/27 = 23.7 (N had 24)\n640/21 = 30.5 (S had 32)\n640/18 = 35.6 (M had 36)\nPositional encoding grid: 120 × 120\nCombinations: 6468\n```\n\nStart with the first six lines. In the paper, N to L are four points on the curve of the same DINOv2-S training run; they all use patch 16 and differ mainly in resolution, growing from 24 to 44 cells per side, while the decoder also grows from 2 to 4 layers. So these sizes have very similar parameter counts, 30.5M for N and 33.7M for M, while latency goes from 2.3 ms to 4.4 ms (Table 2). RF-DETR follows LW-DETR's convention of dividing sizes by latency rather than by parameter count.\n\nL and 2XL have the same number of tokens, 1,936 each; 2XL switches to the DINOv2-B backbone, with patch 20 at a resolution of 880. XL uses only 1 window, which makes every layer global attention.\n\nMore windows aren't always better, either. DINOv2 relies on the CLS token in pretraining, and RF-DETR gives each window its own copy of it; with more windows come more copies, which ends up hurting speed. Appendix L notes that the selected configurations mostly settle at 2 windows per side, while LW-DETR uses 4.\n\nThe middle three lines are an experiment from Appendix L. The authors found that accuracy tracks the \"cells per side,\" and that neither the resolution nor the patch alone predicts it. So they fixed the resolution at 640 and used patches of 27, 21 and 18 to get cell counts close to those of N, S and M; the results were almost the same as with the original configurations, and patches 27 and 18 never once appeared during training.\n\nThe positional encoding line is derived from Appendix A's training grid, with a maximum resolution of 960 and a minimum patch of 8, so 120×120 positions have to be prepared in advance. The last line is the number of combinations enumerated at inference, 11 resolutions, 7 patch sizes, 7 decoder depths (0 to 6 layers), 3 window counts and 4 query counts (50, 100, 200, 300), for a total of 6,468.\n\n### Picking After Training\n\nAfter training on the target dataset, the authors run all 6,468 configurations on the validation set, measure accuracy and latency, and keep the Pareto front, the points that are most accurate at a given latency. For deployment, you set a latency limit and pick the most accurate point on the front. For RF100-VL, target latencies for N through 2XL are fixed first, and on each dataset they look for configurations whose latency falls within 10% of the target.\n\nNAS training takes about 2 to 4 times as long as ordinary single-size training, depending on the dataset; for the search itself, the authors estimate around 10,000 GPU hours (200 T4s running for 48 hours). In exchange, every size comes out of the same training run, whereas non-NAS approaches need another training run for every extra size.\n\nAfter picking a configuration, you can choose to fine-tune it once more. On COCO this barely helps, at most +0.4 AP for any size (Table 15); on RF100-VL it helps a little more, for example taking N from 57.8 to 58.6. The authors' explanation is that 100 epochs of training on a small dataset aren't enough for architecture augmentation to converge.\n\n## Discussion\n\n### First, Calibrate the Ruler for Latency\n\nComparing latency is tricky to begin with. Every new paper re-measures earlier models on its own machines, and the numbers often don't match up. The authors list several reasons in §4:\n\n1. **YOLO latency often leaves out NMS**. Comparing it with end-to-end detectors that need no NMS isn't fair. RT-DETR pointed this out back in 2023.\n2. **YOLO's segmentation models are timed only up to the prototypes**. What gets measured is the time to an intermediate product, before it has been assembled into a mask each object can use directly.\n3. **GPUs throttle**. During continuous inference the GPU overheats, its clock speed is lowered, and the measured latency drifts with it. The authors pause 200 ms between forward passes.\n4. **Latency in FP16, accuracy in FP32**. Many papers measure speed with an FP16 model but accuracy with an FP32 model. Naive quantization sometimes loses a great deal; D-FINE converted directly to FP16 drops to just 0.5 AP, and recovers only when exported with ONNX opset 17. The authors argue that accuracy and latency should be measured on the same model file.\n5. **NMS may differ between validation and inference**. The authors couldn't reproduce the published mAP of YOLOv8 and YOLO11 on TensorRT, and suspect the reason is that these models use multi-class NMS during validation, where one box can keep several classes at once, while actual inference uses only single-class NMS.\n\nUnder this protocol, YOLO always has NMS attached, with a confidence threshold of 0.01; the hardware is a T4 with TensorRT 10.4 and CUDA 12.4. Table 1 sets the numbers that several M-size models published in their original papers side by side with this protocol's measurements:\n\n| Model | Published AP | Published latency | FP32 AP | FP32 latency | FP16 AP | FP16 latency |\n| --- | ---: | ---: | ---: | ---: | ---: | ---: |\n| YOLOv8-M | 50.2 | 5.86 | 49.3 | 14.8 | 47.3 | 5.4 |\n| YOLO11-M | 51.5 | 4.7 | 49.7 | 18.7 | 48.3 | 5.2 |\n| RT-DETR-R18 | 49.0 | 4.61 | 49.0 | 12.2 | 49.0 | 4.4 |\n| LW-DETR-M | 52.5 | 5.6 | 52.6 | 26.8 | 52.6 | 4.4 |\n| D-FINE-M | 55.1 | 5.62 | 55.1 | 13.9 | 55.0 | 5.4 |\n| RF-DETR-M | – | – | 54.8 | 20.5 | 54.7 | 4.4 |\n\n*Table: Table 1 of the paper, latency in ms. The last four columns all use the protocol with a 200 ms pause between forward passes; D-FINE's FP16 numbers are from after the export fix.*\n\nGoing from FP32 to FP16, the two YOLOs lose 2.0 and 1.4 AP, while the four DETR-style models lose at most 0.1. The published numbers also differ from the authors' FP32 measurements, by 0.9 for YOLOv8-M and 1.8 for YOLO11-M. The FP32 latencies range from 12.2 to 26.8 ms, 2.6 to 6.1 times the FP16 ones.\n\nIn September 2025, before the paper came out, a user opened an issue on RF-DETR's GitHub saying that the YOLO11 numbers listed there were wrong. The first author replied that these numbers were actually measured with TensorRT-fused NMS, FP16 and pycocotools; Ultralytics validates with multi-label NMS, which in their experience inflates COCO mAP by about 1, and the speed it reports is FP16 without NMS. [rf-detr #350](https://github.com/roboflow/rf-detr/issues/350) The paper's §4 turns the same argument into a measurement protocol.\n\nHow much does the 200 ms pause matter? Table 20 compares this on the same set of M models; latency with and without a 200 ms pause differs by 0 to 0.3 ms, and pausing longer changes it by no more than 0.1 ms. The authors also acknowledge that this protocol measures single-image latency on a GPU that hasn't overheated and isn't meant for sustained throughput; if your service runs inference continuously, the numbers in the table are closer to what the GPU does while it is still cool. Each TensorRT compilation can also differ by up to 0.1 ms, so the paper reports latency to only one decimal place.\n\nThe choice of protocol also affects the rankings. The authors turn on CUDA graphs in TensorRT for every model, so that the GPU's kernels are queued up in advance instead of being launched one by one by the CPU. This helps RT-DETR, LW-DETR and RF-DETR but not D-FINE; the authors themselves write that with it on, LW-DETR and D-FINE fall on the same accuracy–latency curve.\n\nNMS thresholds change the numbers too. Under this protocol, YOLOv8-N's total latency including NMS is 2.1 ms, at 35.2 AP (Table 2). The same YOLOv8-N has a total latency of 6.16 ms under the YOLOv10 paper's protocol, of which the model itself takes only 1.77 ms; YOLOv10 uses each model's NMS settings from validation, which leave far more candidate boxes. When you see a figure like \"NMS takes a few milliseconds,\" first find out where the thresholds were set.\n\n\n> [!TIP]\n> \n> For the end-to-end protocol that counts NMS in the latency, and the numbers for the same YOLOv8-N under different protocols, see:\n> \n> - **[[23.04] RT-DETR: Putting NMS on the Clock](/blog/papers/object-detection/rt-detr)**\n> - **[[24.05] YOLOv10: Goodbye NMS](/blog/papers/object-detection/yolov10)**\n\n\n### COCO\n\nStart with the upper-left panel of Figure 1:\n\n[![Accuracy–latency curves for COCO object detection, with latency on a T4 from about 1.5 to 20 ms on the horizontal axis and AP50:95 from 35 to 60 on the vertical axis; the purple RF-DETR (DINOv2-S) and the pink RF-DETR (DINOv2-B) are continuous curves, while YOLOv8, YOLO11, D-FINE, RT-DETR and LW-DETR are polylines connecting points marked N to XL](/content/papers/assets/5d7dc32c6480e109cfe21fc73d7a8f9fbc1a7218232165cc360266eb0d5c9849.png)](/content/papers/assets/5d7dc32c6480e109cfe21fc73d7a8f9fbc1a7218232165cc360266eb0d5c9849.png)\n\n*Upper-left panel of Figure 1 from the paper, excerpted from [**page 3 of the v2 PDF**](https://arxiv.org/pdf/2511.09554v2#page=3), CC BY 4.0. The horizontal axis is latency (ms), and the vertical axis is COCO AP50:95.*\n\nThe other families are polylines connecting a few sizes, while RF-DETR's two lines are continuous curves. The purple one is a single training run with a DINOv2-S backbone and the pink one a single run with DINOv2-B; every point on a curve is the same set of weights with a different configuration.\n\nAt the low-latency end, the purple line sits above every other family. Further right, around L, the brown D-FINE catches up with the purple line and pulls a notch above it by XL; the pink curve only crosses 60 at around 17 ms.\n\nA few rows of numbers for comparison:\n\n| Model | Params | Latency (ms) | AP |\n| --- | ---: | ---: | ---: |\n| YOLOv8-N | 3.2M | 2.1 | 35.2 |\n| LW-DETR-T | 12.1M | 1.9 | 42.9 |\n| D-FINE-N | 3.8M | 2.1 | 42.7 |\n| RF-DETR-N | 30.5M | 2.3 | 48.0 |\n| YOLOv8-M | 25.9M | 5.4 | 47.3 |\n| LW-DETR-M | 28.2M | 4.4 | 52.6 |\n| D-FINE-M | 19.2M | 5.4 | 55.0 |\n| RF-DETR-M | 33.7M | 4.4 | 54.7 |\n| D-FINE-XL | 62M | 11.5 | 59.3 |\n| RF-DETR-XL | 126.4M | 11.5 | 58.6 |\n| RF-DETR-2XL | 126.9M | 17.2 | 60.1 |\n\n*Table: Compiled from Tables 2 and 12 of the paper, all measured by the authors with the same protocol.*\n\n1. **The gap is widest at N**. RF-DETR-N is 5.3 AP above D-FINE-N, with 0.2 ms more latency; its 48.0 is also higher than YOLOv8-M's 47.3, at less than half the latency.\n2. **From M up, the wins and losses are mixed**. D-FINE-M is 0.3 AP above RF-DETR-M but 1.0 ms slower; at XL, both run at 11.5 ms, and D-FINE is the one ahead, by 0.7.\n3. **2XL passes 60**. At 60.1 AP and 17.2 ms, it is 0.8 above D-FINE-XL, with 5.7 ms more latency. The authors say that, to the best of their knowledge, it is the first real-time detector to exceed 60 AP on COCO.\n\nBesides DINOv2, RF-DETR was also pretrained on Objects365; so were D-FINE's S through XL, but not D-FINE-N, as the authors themselves note in the appendix. RF-DETR also has far more parameters, 30.5M for N against 3.8M for D-FINE-N, an eightfold difference, which devices short on memory or storage need to take into account. By \"real-time\" the paper means within 40 ms on a T4; the appendix also has a Max version at 61.8 AP, but it needs 98 ms, which puts it outside that range.\n\nRoboflow's March 2025 launch post already called RF-DETR the first real-time model to exceed 60 mAP, referring at the time to the 60.5 of the old RF-DETR-large at a resolution of 728; in the paper, the claim rests on 2XL's 60.1 instead.\n\n### RF100-VL\n\nOn RF100-VL, again a few rows:\n\n| Model | Latency (ms) | AP | Fine-tuned after NAS |\n| --- | ---: | ---: | ---: |\n| YOLOv8-N | 2.6 | 55.0 | – |\n| YOLO11-M | 5.1 | 57.0 | – |\n| D-FINE-N | 2.0 | 58.2 | – |\n| RF-DETR-N | 2.5 | 57.8 | 58.6 |\n| D-FINE-M | 5.6 | 60.6 | – |\n| RF-DETR-M | 4.6 | 61.7 | 62.0 |\n| GroundingDINO-T (fine-tuned) | 309.9 | 62.3 | – |\n| RF-DETR-2XL | 15.6 | 63.3 | 63.5 |\n\n*Table: Compiled from Table 4 of the paper, averaged over 100 datasets. GroundingDINO's latency is measured in PyTorch, the others in TensorRT.*\n\nYOLO stays between 55 and 57 whatever the size. At the smallest size, D-FINE-N's 58.2 is actually higher than RF-DETR-N without the extra fine-tuning; at M, RF-DETR leads by 1.1 AP with 1.0 ms less latency. 2XL is 1.0 AP above the fine-tuned GroundingDINO (1.2 after the extra fine-tuning) at about a twentieth of its latency, though the two use different inference engines.\n\nMost of the gap between YOLO and the DETR-style models lies in small objects. YOLO's AP_S is only 4.7 to 7.3 across sizes, while the DETR-style models are all above 30; the paper doesn't analyze why. The authors also caution that YOLO keeps the 0.01 confidence threshold here, which may not suit every dataset, so the latency numbers may not show YOLO at its best.\n\nAppendix I has another experiment, which carries the configurations picked on COCO straight over to RF100-VL without searching again on the new datasets. N, S and M get 57.7, 60.2 and 61.2, against 57.8, 60.9 and 61.7 for configurations re-searched on RF100-VL, a difference of 0.1 to 0.7 AP; L's fixed configuration is actually 0.2 higher, though with 0.6 ms more latency. The fixed configurations already beat LW-DETR of the same size across the board.\n\n### Ablation: Where the Extra 2 AP Comes From\n\nTable 5 starts from LW-DETR-M and turns it into RF-DETR-M one step at a time:\n\n| Step | Latency (ms) | AP |\n| --- | ---: | ---: |\n| LW-DETR-M | 4.4 | 52.6 |\n| + Gentler hyperparameters | 4.4 | 51.6 |\n| + DINOv2 backbone | 4.7 | 53.6 |\n| + More Objects365 pretraining | 4.7 | 54.3 |\n| + Weight-sharing NAS (same configuration) | 4.7 | 54.6 |\n| 　Patch to 16, resolution to 640 | 4.7 | 54.4 |\n| 　Resolution down to 576 | 4.0 | 53.6 |\n| 　Windows per side 4 → 2 | 4.3 | 54.3 |\n| 　Decoder 3 → 4 layers | 4.4 | 54.6 |\n\n*Table: Compiled from Table 5 of the paper, COCO.*\n\nSwitching to gentler hyperparameters (a smaller learning rate, LayerNorm and so on) first costs 1.0 AP; the authors say LayerNorm hurts accuracy a little, but without it training on consumer hardware isn't possible. Switching in DINOv2 brings AP back up by 2.0, with 0.3 ms more latency.\n\nThe configuration at the fifth step is patch 14 at resolution 560, and patch 14 isn't even in the training grid. Simply training the model along with NAS gives the same configuration 0.3 more AP, which is the evidence for the architecture augmentation mentioned earlier.\n\nThe last four steps turn the knobs to bring the latency back to 4.4 ms. The final 54.6 AP is 2.0 higher than LW-DETR-M at the same latency.\n\nThe backbone comparison is in Table 6. Under the same settings, DINOv2 ViT-S gets 54.3, CAEv2 52.3 and SigLIPv2 50.4; SAM2's Hiera-S reaches 53.6, but its latency is 11.2 ms, more than twice DINOv2's 4.7 ms. (SigLIPv2 and SAM2 score very poorly after conversion to FP16, so these two rows show FP32 accuracy, which amounts to their upper bound.)\n\n## Conclusion\n\nRF-DETR swaps LW-DETR's backbone for DINOv2, trains once on the target dataset with weight-sharing NAS, then enumerates over six thousand configurations on the validation set and picks points by latency. These are the paper's takeaways:\n\n1. **Sizes can be picked after the fact**. One training run gives a whole accuracy–latency curve, and different hardware picks different points. The price is 2 to 4 times the training time, plus some ten thousand GPU hours for the search.\n2. **Look at a second benchmark besides COCO**. YOLO improves steadily with size on COCO but barely moves on RF100-VL; which of D-FINE and RF-DETR wins also flips with the dataset. RF100-VL, though, is a benchmark the authors compiled themselves.\n3. **Latency measurements need their conditions spelled out**. Whether NMS is counted, what the thresholds are, FP16 or FP32, whether the GPU gets to cool down; if any one of these differs, the numbers in the tables can't be compared directly.\n\nThe NAS code isn't open source. In a December 2025 issue, the first author said the code on GitHub is for fine-tuning the fixed configurations that NAS found, and running NAS on your own data has to go through Roboflow's platform. [rf-detr #483](https://github.com/roboflow/rf-detr/issues/483) By the numbers in Appendix I, COCO's fixed configurations used on RF100-VL come within 0.7 AP of re-searched configurations when neither is fine-tuned again. The licensing is split as well, with the N to L detection models under Apache 2.0 and the XL and 2XL detection models under Roboflow's own PML 1.0, which requires a separate payment for commercial self-hosting. [rf-detr README](https://github.com/roboflow/rf-detr)\n\nThe number of queries can only be cut, with the 300 used in training as the ceiling. For scenes with hundreds of objects per image, the first author suggests cutting the large image into tiles before feeding it in. [rf-detr #350](https://github.com/roboflow/rf-detr/issues/350) Also, the official post-processing flattens the \"query × class\" scores and takes the top 300, so the same box may show up with two classes. A user has reported overlapping boxes; the first author said that with enough data the model should learn to avoid them on its own, while a Roboflow engineer suggested simply attaching NMS after the output. [rf-detr #187](https://github.com/roboflow/rf-detr/issues/187)\n\nNext time a detector claims to be faster than YOLO, turn first to the section where it measures latency, and check what the NMS thresholds are and whether speed and accuracy were measured on the same model file. To see how the YOLO side handles NMS, read Ultralytics' [YOLO26](/blog/papers/object-detection/yolo26) next.\n","imageSizes":{"/content/papers/assets/10f93bab733752bf97f8b3ed5c86569a2c4d85840b99b338a6443431fb506555.png":{"width":1080,"height":775},"/content/papers/assets/f0e8bf4f9ecad4cb2f090117a18034f54eab8bc3e901414f94864de103d3425a.png":{"width":1005,"height":653},"/content/papers/assets/5d7dc32c6480e109cfe21fc73d7a8f9fbc1a7218232165cc360266eb0d5c9849.png":{"width":788,"height":555}}}