{"notes":[{"id":"object-detection/yolo-world","category":"object-detection","legacyPath":"object-detection/2401-yolo-world","legacySlug":"2401-yolo-world","legacyPaths":[],"title":"YOLO-World","subtitle":"The Pursuer of Nouns","author":"Z. Yuan","paperDate":"2024-01-30","publishedOn":"2025-06-09","paperTitle":"YOLO-World: Real-Time Open-Vocabulary Object Detection","paperUrl":"https://openaccess.thecvf.com/content/CVPR2024/html/Cheng_YOLO-World_Real-Time_Open-Vocabulary_Object_Detection_CVPR_2024_paper.html","bibliography":{"title":"YOLO-World: Real-Time Open-Vocabulary Object Detection","authors":["Tianheng Cheng","Lin Song","Yixiao Ge","Wenyu Liu","Xinggang Wang","Ying Shan"],"url":"https://openaccess.thecvf.com/content/CVPR2024/html/Cheng_YOLO-World_Real-Time_Open-Vocabulary_Object_Detection_CVPR_2024_paper.html","pdfUrl":"https://openaccess.thecvf.com/content/CVPR2024/papers/Cheng_YOLO-World_Real-Time_Open-Vocabulary_Object_Detection_CVPR_2024_paper.pdf","codeUrl":"https://github.com/AILab-CVC/YOLO-World","preprint":{"id":"2401.17270","submittedOn":"2024-01-30","url":"https://arxiv.org/abs/2401.17270"},"publication":{"venue":"CVPR","year":2024,"kind":"conference","url":"https://openaccess.thecvf.com/content/CVPR2024/html/Cheng_YOLO-World_Real-Time_Open-Vocabulary_Object_Detection_CVPR_2024_paper.html"},"citation":"@misc{objectdetectionyoloworld2024,\n  title = {{YOLO-World: Real-Time Open-Vocabulary Object Detection}},\n  author = {Tianheng Cheng and Lin Song and Yixiao Ge and Wenyu Liu and Xinggang Wang and Ying Shan},\n  year = {2024},\n  eprint = {2401.17270},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2401.17270}\n}\n","citationPath":"/content/papers/citations/object-detection/yolo-world.06f133b6c9ac.bib"},"thumbnail":{"src":"/content/papers/assets/7a09b97227f7f10a9b85e7c3cc305b05f5db9be415ef710cf9a6940180a3c663.jpg","width":1552,"height":650},"locale":"en","path":"/content/papers/en/object-detection/yolo-world.1ea4f52472a6.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"solution","text":"Solution","depth":2},{"id":"repvl-pan","text":"RepVL-PAN","depth":3},{"id":"pretraining-strategy","text":"Pretraining Strategy","depth":3},{"id":"contrastive-learning","text":"Contrastive Learning","depth":3},{"id":"task-separation","text":"Task Separation","depth":3},{"id":"automatic-annotation","text":"Automatic Annotation","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"generalization-ability-of-small-models","text":"Generalization Ability of Small Models","depth":3},{"id":"module-ablation-study","text":"Module Ablation Study","depth":3},{"id":"encoder-fine-tuning-strategy","text":"Encoder Fine-tuning Strategy","depth":3},{"id":"downstream-task-extensibility","text":"Downstream Task Extensibility","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{"experimental-results-and-discussion":"discussion"},"guide":null,"relatedReading":[{"id":"multimodality/mdetr","relation":"comparison","topic":"Detecting objects described in language"},{"id":"multimodality/clip","relation":"comparison","topic":"Detecting objects described in language"}]},{"id":"multimodality/mdetr","category":"multimodality","legacyPath":"multimodality/2104-mdetr","legacySlug":"2104-mdetr","legacyPaths":[],"title":"MDETR","subtitle":"The Art of Continuity","author":"Z. Yuan","paperDate":"2021-04-26","publishedOn":"2024-06-04","paperTitle":"MDETR - Modulated Detection for End-to-End Multi-Modal Understanding","paperUrl":"https://arxiv.org/abs/2104.12763","bibliography":{"title":"MDETR - Modulated Detection for End-to-End Multi-Modal Understanding","authors":["Aishwarya Kamath","Mannat Singh","Yann LeCun","Gabriel Synnaeve","Ishan Misra","Nicolas Carion"],"url":"https://arxiv.org/abs/2104.12763","pdfUrl":"https://openaccess.thecvf.com/content/ICCV2021/papers/Kamath_MDETR_-_Modulated_Detection_for_End-to-End_Multi-Modal_Understanding_ICCV_2021_paper.pdf","codeUrl":"https://github.com/ashkamath/mdetr","preprint":{"id":"2104.12763","submittedOn":"2021-04-26","url":"https://arxiv.org/abs/2104.12763"},"publication":{"venue":"ICCV","year":2021,"kind":"conference","url":"https://openaccess.thecvf.com/content/ICCV2021/html/Kamath_MDETR_-_Modulated_Detection_for_End-to-End_Multi-Modal_Understanding_ICCV_2021_paper.html"},"citation":"@misc{multimodalitymdetr2021,\n  title = {{MDETR - Modulated Detection for End-to-End Multi-Modal Understanding}},\n  author = {Aishwarya Kamath and Mannat Singh and Yann LeCun and Gabriel Synnaeve and Ishan Misra and Nicolas Carion},\n  year = {2021},\n  eprint = {2104.12763},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2104.12763}\n}\n","citationPath":"/content/papers/citations/multimodality/mdetr.46559f486386.bib"},"thumbnail":{"src":"/content/papers/assets/b270f17cbb7a758589f1e40ab2239d5b8ba070f3e637505b70834cace8d3c8d9.jpg","width":1024,"height":338},"locale":"en","path":"/content/papers/en/multimodality/mdetr.1eaa3d74f06a.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"solution","text":"Solution","depth":2},{"id":"mdetr-model-design","text":"MDETR Model Design","depth":3},{"id":"training-method","text":"Training Method","depth":3},{"id":"contrastive-alignment","text":"Contrastive Alignment","depth":3},{"id":"all-losses","text":"All Losses","depth":3},{"id":"dataset","text":"Dataset","depth":3},{"id":"technical-details","text":"Technical Details","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"downstream-task-performance-analysis","text":"Downstream Task Performance Analysis","depth":3},{"id":"few-shot-capabilities","text":"Few-Shot Capabilities?","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{"defining-the-problem":"problem-definition","the-solution":"solution","bring-text-into-detection-before-predicting-boxes":"mdetr-model-design","a-box-needs-a-place-in-the-sentence":"training-method","count-images-and-aligned-pairs-separately":"dataset","experimental-results-and-discussion":"discussion","flickr30k-one-phrase-may-point-to-several-boxes":"downstream-task-performance-analysis","referring-expressions-and-gqa-need-their-own-conditions":"downstream-task-performance-analysis","lvis-few-shot-detection-rare-class-gains-have-an-inference-cost":"few-shot-capabilities"},"guide":null,"relatedReading":[{"id":"object-detection/yolo-world","relation":"comparison","topic":"Detecting objects described in language"},{"id":"multimodality/clip","relation":"comparison","topic":"Detecting objects described in language"}]},{"id":"multimodality/clip","category":"multimodality","legacyPath":"multimodality/2103-clip","legacySlug":"2103-clip","legacyPaths":["multi-model/2021/clip"],"title":"CLIP","subtitle":"Breaking the Dimensional Barrier","author":"Z. Yuan","paperDate":"2021-02-26","publishedOn":"2024-04-04","paperTitle":"Learning Transferable Visual Models From Natural Language Supervision","paperUrl":"https://arxiv.org/abs/2103.00020","bibliography":{"title":"Learning Transferable Visual Models From Natural Language Supervision","authors":["Alec Radford","Jong Wook Kim","Chris Hallacy","Aditya Ramesh","Gabriel Goh","Sandhini Agarwal","Girish Sastry","Amanda Askell","Pamela Mishkin","Jack Clark","Gretchen Krueger","Ilya Sutskever"],"url":"https://arxiv.org/abs/2103.00020","pdfUrl":"https://cdn.openai.com/papers/Learning_Transferable_Visual_Models_From_Natural_Language.pdf","codeUrl":"https://github.com/openai/CLIP","preprint":{"id":"2103.00020","submittedOn":"2021-02-26","url":"https://arxiv.org/abs/2103.00020"},"publication":{"venue":"ICML","year":2021,"kind":"conference","url":"https://proceedings.mlr.press/v139/radford21a.html"},"citation":"@misc{multimodalityclip2021,\n  title = {{Learning Transferable Visual Models From Natural Language Supervision}},\n  author = {Alec Radford and Jong Wook Kim and Chris Hallacy and Aditya Ramesh and Gabriel Goh and Sandhini Agarwal and Girish Sastry and Amanda Askell and Pamela Mishkin and Jack Clark and Gretchen Krueger and Ilya Sutskever},\n  year = {2021},\n  eprint = {2103.00020},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2103.00020}\n}\n","citationPath":"/content/papers/citations/multimodality/clip.a63b118c04c7.bib"},"thumbnail":{"src":"/content/papers/assets/03a0527e167fb49905db2879fecd0e0b7b140cb837e239778ffa7f83f02e25ef.jpg","width":1726,"height":622},"locale":"en","path":"/content/papers/en/multimodality/clip.f9c727d9ae23.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"the-solution","text":"The Solution","depth":2},{"id":"model-architecture","text":"Model Architecture","depth":3},{"id":"large-scale-dataset","text":"Large-Scale Dataset","depth":3},{"id":"training-details","text":"Training Details","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"zero-shot-learning","text":"Zero-Shot Learning","depth":3},{"id":"are-the-representations-any-good","text":"Are the Representations Any Good?","depth":3},{"id":"is-clip-truly-generalizable","text":"Is CLIP Truly Generalizable?","depth":3},{"id":"human-vs-machine","text":"Human vs. Machine","depth":3},{"id":"overlap-in-data","text":"Overlap in Data","depth":3},{"id":"limitations","text":"Limitations","depth":2},{"id":"ethical-concerns","text":"Ethical Concerns","depth":3},{"id":"conclusion","text":"Conclusion","depth":2},{"id":"key-contributions","text":"Key Contributions:","depth":3},{"id":"future-directions","text":"Future Directions:","depth":3}],"anchorAliases":{"match-the-text-before-trying-to-write-it":"model-architecture","four-hundred-million-pairs-and-a-sizable-training-bill":"large-scale-dataset","experimental-results-and-discussion":"discussion","strong-zero-shot-results-depend-on-the-task-and-the-baseline":"zero-shot-learning","distribution-shift-figure-13s-strength-and-figure-14s-warning":"is-clip-truly-generalizable","humans-overlapping-data-and-use-boundaries":"human-vs-machine"},"guide":null,"relatedReading":[{"id":"model-tuning/coop","relation":"next","topic":"Adapting image-text models with prompts"},{"id":"multimodality/flip","relation":"comparison","topic":"Scaling and accelerating image-text pretraining"},{"id":"multimodality/florence","relation":"comparison","topic":"Scaling and accelerating image-text pretraining"}]}],"text":"In this era of object detection represented by YOLO, speed and efficiency have long ceased to be new topics.\n\nWhat is truly challenging is the closed nature of categories.\n\n## Problem Definition\n\nOver the past decade, the progress in object detection has been revolutionary.\n\nWe witnessed the R-CNN series gradually constructing the rule-based world of two-stage detection, saw SSD and RetinaNet simplify the process by querying each anchor on a pixel basis; we also saw DETR introduce query design and Transformers, transforming \"detection\" from mere localization to semantic interaction.\n\nAmong all these architectures, the YOLO series remains a distinctive existence.\n\nIt continuously compresses computational costs, pushing the limits of real-time inference with a minimalist architecture: **fast! faster! fastest!**\n\nHowever, despite these technological iterations, one thing has rarely been shaken:\n\n> **Almost all mainstream detection models still live in a world of \"fixed vocabularies.\"**\n\nTraining data defines the language the model can speak; if \"kite\" is absent from a fixed detector’s class list, it cannot report a detected region as a kite at inference. COCO has 80 categories, Objects365 has 365, but the real world contains many more objects without names in these datasets.\n\nHow can we enable models to have the ability to \"open the dictionary\" instead of only responding within a few known options?\n\n**Open-Vocabulary Object Detection** (abbreviated OVD) was born to solve this bottleneck. It aims for object detection to transcend fixed-class boundaries and learn to understand \"semantics\" rather than merely memorizing the mapping between images and labels.\n\nBut every attempt at OVD comes at a high cost.\n\n## Solution\n\n\n\n![YOLO-World architecture: text and multi-scale image features meet in RepVL-PAN before box prediction and region-text matching (paper Figure 3)](/content/papers/assets/7a09b97227f7f10a9b85e7c3cc305b05f5db9be415ef710cf9a6940180a3c663.jpg)\n\n\n\nThe architecture of YOLO-World is shown above.\n\nIt retains the overall design familiar in the YOLO series: after the image is encoded by a backbone, it passes through a multi-scale feature fusion module, ultimately outputting predicted bounding boxes and object representations. Unlike past fixed-category setups, YOLO-World introduces a language encoder and cross-modal fusion module, enabling the whole architecture to handle open-vocabulary detection tasks.\n\nThe overall process is as follows:\n\n1. The user input prompt (which could be a description, a set of nouns, or a class list) is first converted into semantic vectors by CLIP’s text encoder.\n\n2. The image is processed by YOLOv8’s image encoder to extract multi-scale image features.\n\n3. Then, through the newly proposed RepVL-PAN, language and image features are aligned and fused.\n\n4. Finally, a decoupled head simultaneously outputs bounding boxes and object semantic representations, which are matched against the input vocabulary.\n\nThis design allows YOLO-World to no longer rely on fixed-class classifiers but to find the regions in the image that best match the input prompt via semantic matching. The vocabulary can be changed for inference after the new words are encoded and the offline vocabulary weights are updated; zero-shot transfer still depends on the vision-language alignment learned during pretraining.\n\nUnder such a structure, object detection shifts from \"predicting which class\" to \"whether this region corresponds to the concept described by a certain word.\"\n\nNext, let's take a closer look at the design of RepVL-PAN.\n\n### RepVL-PAN\n\n\n\n![RepVL-PAN: T-CSPLayer injects text into visual features, while I-Pooling Attention updates text from the image (paper Figure 4)](/content/papers/assets/4e32d04c0d7a7d42bc38361034c182f7631c8f6368ca38ca84d32c41b09819ed.jpg)\n\n\n\nAs shown above, YOLO-World proposes RepVL-PAN (Re-parameterizable Vision-Language Path Aggregation Network), a module that enhances vision-language interaction and operates during YOLO’s multi-scale feature fusion stage.\n\nTraditional PAN only handles top-down and bottom-up image feature passing and fusion. However, in open-vocabulary detection tasks, relying solely on visual signals is insufficient for semantic matching. Therefore, RepVL-PAN injects language information into the image feature pipeline, allowing semantics to permeate the feature map generation process.\n\nIt consists of two key components:\n\n1. **Text-guided CSPLayer**\n\n   In the YOLO architecture, CSPLayer is an improved residual stack module responsible for maintaining feature stability and channel expressiveness. In YOLO-World, the authors design a text-guided version of CSPLayer, enabling each feature layer to be adjusted based on vocabulary semantics.\n\n   Specifically, for an image feature at a certain scale $X_l \\in \\mathbb{R}^{H \\times W \\times D}$, and a set of word embedding vectors from the text encoder $W = \\{w_j\\}_{j=1}^C$, we compute the similarity between each position and the semantics, and apply a sigmoid function to form attention weights:\n\n   $$\n   X_l' = X_l \\cdot \\delta\\left( \\max_{j \\in \\{1..C\\}} (X_l W_j^\\top) \\right)^\\top\n   $$\n\n   Here $\\delta(\\cdot)$ is sigmoid: it turns the maximum similarity to the supplied words at each location into a gate. This design weights each position’s image feature according to semantic cues, generating a feature map $X_l'$ that better corresponds to the vocabulary.\n\n   These enhanced image features are concatenated with the cross-stage branch features, then passed to the next PAN layer, gradually building a semantically sensitive pyramid representation.\n\n2. **Image-Pooling Attention**\n\n   Besides letting language influence images, YOLO-World also designs a feedback path so image information can refine language representations. This is implemented through Image-Pooling Attention.\n\n   Specifically, image features from various scales are pooled down to several regional vectors (a $3 \\times 3$ grid per scale, or 27 tokens across three scales), denoted as $\\tilde{X} \\in \\mathbb{R}^{27 \\times D}$, which serve as Key and Value to update the original word embeddings via multi-head attention:\n\n   $$\n   W' = W + \\text{MultiHeadAttention}(W, \\tilde{X}, \\tilde{X})\n   $$\n\n   As a result, the word embeddings are no longer static semantic vectors but enriched with current image context, strengthening the correspondence between words and image content.\n\n---\n\nThese two designs form a symmetrical semantic loop: one allows language to infiltrate the image, the other lets the image refine the language.\n\nUltimately, these updated language and image features jointly participate in object detection, enabling the model to perform stable semantic matching without relying on fixed-class classifiers.\n\nA key aspect of this entire mechanism is that it supports **re-parameterization**: the chosen vocabulary can be encoded once and folded into convolutional or linear weights. For a fixed offline vocabulary, the CLIP text encoder need not run for every image, reducing computational cost. Changing the vocabulary requires encoding the new words and updating those weights.\n\n### Pretraining Strategy\n\nAlthough language and images are different modalities, if we want the model to understand what a \"word refers to,\" it ultimately comes down to one thing:\n\n> **The correspondence between words and image regions.**\n\nYOLO-World adopts a region-text contrastive learning strategy, aiming to teach the model to distinguish: **\"This is a dog,\" not \"This looks like a dog.\"**\n\nTo train this ability, the authors unify the annotation format for detection, grounding, and image-text data, representing each annotated sample as:\n\n$$\n\\Omega = \\{(B_i, t_i)\\}_{i=1}^N\n$$\n\nwhere $B_i$ is the bounding box of the $i$-th region in the image, and $t_i$ is the corresponding word.\n\nThis word can be a category noun or a noun phrase extracted from the description.\n\n### Contrastive Learning\n\nDuring the forward phase, YOLO-World predicts $K$ object regions per training image, each with a representation vector $e_k$. Meanwhile, the text encoder generates corresponding word vectors $\\{w_j\\}$ for the input vocabulary set $T$.\n\nTo make the similarity between these vectors meaningful, the model calculates similarity as follows:\n\n$$\ns_{k,j} = \\alpha \\cdot \\text{L2Norm}(e_k) \\cdot \\text{L2Norm}(w_j)^\\top + \\beta\n$$\n\nwhere $\\alpha, \\beta$ are trainable scale and bias parameters to stabilize training; L2 normalization ensures similarity mainly reflects direction (semantics) rather than magnitude.\n\nWith this similarity matrix, the model transforms the word classification task into a semantic matching task. It first matches predictions to annotations through task-aligned assignment, then computes the **region-text contrastive loss** for positive predictions:\n\n$$\n\\mathcal{L}_{\\text{con}} = \\text{CrossEntropy}(s_{k,j}, y_{k})\n$$\n\n### Task Separation\n\nIn the overall training process, YOLO-World uses data not only from annotated datasets such as Objects365 but also samples from grounding tasks or image-text paired datasets.\n\nConsidering data quality differences, the authors use an indicator variable $\\lambda_I$ to control whether to apply box regression losses:\n\n- When image $I$ comes from detection or grounding data, IoU loss and distribution focal loss are calculated.\n- When image $I$ comes from image-text data (e.g., CC3M), only the semantic contrastive loss is calculated.\n\nThe total loss is:\n\n$$\n\\mathcal{L}(I) = \\mathcal{L}_{\\text{con}} + \\lambda_I \\cdot (\\mathcal{L}_{\\text{iou}} + \\mathcal{L}_{\\text{dfl}})\n$$\n\nThis design enables YOLO-World to flexibly handle data of varying quality and sources, maintaining generalization ability while preserving bounding box regression accuracy.\n\n### Automatic Annotation\n\nThe real key enabling YOLO-World to conduct large-scale pretraining is the **pseudo-labeling** strategy.\n\nAfter all, large image-text datasets like CC3M have rich descriptions but lack bounding box annotations; relying on manual labeling is impractical.\n\nTherefore, YOLO-World adopts a three-stage automatic annotation pipeline:\n\n1. **Word Extraction:** Extract noun phrases from descriptions (e.g., \"a dog running in the park\" → \"dog,\" \"park\").\n2. **Bounding Box Generation:** Use a pretrained GLIP model to generate predicted boxes in the image based on these words.\n3. **Filtering:** Use CLIP to evaluate both image-caption and region-word relevance, remove unreliable pseudo pairs and images, and apply NMS to eliminate redundant boxes.\n\nIn the end, the authors collected 246k images from CC3M and generated 821k pseudo region-text annotations, becoming a crucial data source for training YOLO-World.\n\n## Discussion\n\nThe experiments in the paper cover three aspects: zero-shot, fine-tuning, and open-vocabulary segmentation, verifying YOLO-World’s generalization ability, practical value, and extensibility respectively.\n\nWe highlight several key results here; please refer to the original paper for full details.\n\n### Generalization Ability of Small Models\n\n\n\n![Zero-shot detection on LVIS minival; parentheses show parameters and FPS before re-parameterization (paper Table 2)](/content/papers/assets/715c3e5fba7865d5fae13b73e712bc3cec2bc2ec43c82e2961cdec1b45fa58c1.jpg)\n\n\n\nThe authors conducted zero-shot evaluation on LVIS, shown above. Parentheses show both parameter counts and FPS before re-parameterization; the values outside them describe the deployed version with offline vocabulary and no per-image text encoding.\n\nWithout any LVIS fine-tuning, YOLO-World-L pretrained on O365, GoldG, and pseudo-labeled CC3M achieves **35.4 Fixed AP / 52.0 FPS** on LVIS minival, exceeding the GLIP-T and Grounding DINO-T rows shown in this table for both AP and speed. Their pretraining data and parameter counts differ, so the gap cannot be assigned to architecture alone.\n\nMore importantly, this result was measured on a pure V100 setup without TensorRT or FP16 acceleration, demonstrating the model’s highly efficient inference pipeline.\n\nSeveral observations arise from this result:\n\n- Offline vocabulary avoids re-encoding text for every image. This LVIS evaluation uses 1,203 category names and retains at most 1,000 predictions per image; box filtering remains part of detection, so this is not a post-processing-free result.\n- YOLO-World-S, the smallest version with **13M parameters after re-parameterization** (77M before), still reaches **26.2 AP**, close to other medium-sized architectures, showing that the pretraining scheme also works on the S variant.\n\nPreviously, open-vocabulary detection models were often impractical for deployment due to their large size and language model dependencies.\n\nThis makes real-time open-vocabulary detection plausible beyond very large detectors. The 52 FPS figure comes from one NVIDIA V100, however; edge-device latency remains to be measured.\n\n### Module Ablation Study\n\n\n\n![RepVL-PAN ablation of T→I, I→T, and GQA (paper Table 4)](/content/papers/assets/86d83bc45036f49aeb0ebf03dabfbc53fe4f54fc63b559cea7ad99d77bb54ea7.jpg)\n\n\n\nTo validate the contribution of the RepVL-PAN module in YOLO-World, the authors designed ablation studies on two core modules: **Text-guided CSPLayer (T→I)** and **Image-Pooling Attention (I→T)**, corresponding to language influencing image and image feeding back to language respectively. The table shows zero-shot detection performance on LVIS under five settings.\n\nHere:\n\n- “T→I” means enabling Text-guided CSPLayer;\n- “I→T” means enabling Image-Pooling Attention;\n- “GQA” indicates whether the language-rich GQA dataset was used in pretraining.\n\nThe experiments divide into two parts:\n\nFirst, without GQA pretraining:\n\n- The **baseline with no language module** has AP only **22.4**, and rare category ($AP_r$) at **14.5**.\n- Enabling **T→I** module raises AP to **23.2**, with slight increase of $AP_r$ to **15.2**.\n- Enabling both **T→I and I→T** (bidirectional language fusion) further improves AP to **23.5**, and notably raises $AP_r$ to **16.2**, with rare-class AP rising another 1.0 point after I→T is added in this setting.\n\nThis shows that even without the richer GQA text annotations, adding the language modules improves YOLO's zero-shot AP in this O365 setting.\n\nNext, with GQA pretraining:\n\n- Adding GQA alone without language modules improves the model significantly to **29.7 AP / 21.0 $AP_r$**, indicating the semantic information in GQA benefits recognition.\n- Enabling both language modules with GQA yields the highest scores: **31.9 AP / 22.5 $AP_r$**, with common and frequent categories ($AP_c$, $AP_f$) also reaching **29.9 / 35.4**.\n\nOverall:\n\n- **T→I provides early fusion, language-guided capability**, gating PAN image features with text;\n- **I→T supplements contextual feedback**, making word embeddings more image-context aware;\n- When pretrained on semantically dense data (like GQA), the joint gain rises from 1.1 to 2.2 AP; the table does not isolate each module under GQA.\n\nThese results show YOLO-World does not rely on stacking backbones but establishes bidirectional language fusion paths within the PAN structure, enabling stable and fine-grained semantic alignment for open vocabularies.\n\n### Encoder Fine-tuning Strategy\n\n\n\n![BERT-base and CLIP-base, frozen or fine-tuned, on zero-shot LVIS (paper Table 5)](/content/papers/assets/ac9c574c34aed9e6c31823b09311dc1e1d1cbf7a31d2d2a31a2419f3f13d1df9.jpg)\n\n\n\nFor generating word embeddings, YOLO-World adopts two text encoders with different traits: CLIP-base and BERT-base, comparing their performance when frozen and fine-tuned.\n\nObservations:\n\n- When frozen, **CLIP-base greatly outperforms BERT-base**, with a rare category gap on LVIS of **+11.1 AP** (14.5 vs. 3.4), and overall AP increasing from **14.6** to **22.4**.\n- Fine-tuning BERT-base improves its performance to **18.3 AP** and rare category to **6.6 AP**, but still below frozen CLIP.\n- Conversely, **fine-tuning CLIP results in performance degradation**: AP drops from **22.4** to **19.3**, $AP_r$ falls drastically from **14.5** to **8.6**.\n\nThis indicates several insights:\n\n- CLIP was pretrained on image-text pairs; in this LVIS zero-shot experiment, frozen CLIP outperforms frozen BERT.\n- Fine-tuning BERT raises AP from 14.6 to 18.3 here. The table alone cannot show how its internal semantic mapping changes.\n- Fine-tuning CLIP lowers AP in this setting. The authors suggest that O365's narrower 365-category text distribution may weaken its prior generalization, an explanation for this experiment rather than a general rule.\n\nThis reminds us that, in this O365-to-LVIS experiment, **whether text-encoder fine-tuning helps depends on its pretraining and the target vocabulary**.\n\n### Downstream Task Extensibility\n\n\n\n![Open-vocabulary instance segmentation: AP is mask AP and AP^b is box AP (paper Table 8)](/content/papers/assets/00bb2450a552bb62b732727aaa0741b3cde07bb518ad4698c6a8d0bcac1b6a7a.jpg)\n\n\n\nYOLO-World extends beyond detection to **Open-Vocabulary Instance Segmentation (OVIS)**, requiring the model not only to locate \"what it is\" but also to outline \"its contour.\"\n\nThe authors designed two experiment settings:\n\n1. **COCO → LVIS**: fine-tune the pretrained YOLO-World on COCO masks for 80 classes, then evaluate transfer to 1,203 LVIS classes.\n2. **LVIS-base → LVIS**: fine-tune on LVIS-base masks for 866 classes, test generalization on full set including 337 rare classes.\n\nTwo training strategies:\n\n- **Fine-tune only the segmentation head**, preserving the detection backbone and pretrained semantic generalization.\n- **Fine-tune all modules (All)**, allowing full learning of segmentation context but possibly sacrificing zero-shot flexibility.\n\nWith only the segmentation head fine-tuned:\n\n- Training on COCO, YOLO-World-L achieves **16.2 AP / 12.4 $AP_r$**;\n- Using the semantically denser, more finely annotated LVIS-base improves to **19.1 AP / 14.2 $AP_r$**.\n\nThis shows that, with the detector frozen, training only the segmentation head still yields nonzero mask AP on LVIS rare classes. The two training sources also differ in annotation density, so the gap cannot be assigned to semantic pretraining alone.\n\nAllowing full-model fine-tuning (including PAN, language module, detection head):\n\n- Under LVIS-base training, YOLO-World-L reaches **28.7 AP**, a significant improvement over segmentation-head-only tuning.\n- However, $AP_r$ rises only from 14.2 to 15.0, and box $AP_r$ falls from 18.0 to 17.4, a 0.6-point drop.\n\nThis highlights a classic open-vocabulary dilemma:\n\n> **Overall AP can hide changes in rare-class and box detection.**\n\nIf the priority is overall mask AP, full fine-tuning scores higher in this experiment; when unfamiliar classes matter, both mask $AP_r$ and box $AP_r^b$ should guide which modules to freeze.\n\nIn multimodal systems, **accuracy and generalization form a dynamic balance, not an infinitely coexisting goal**. Adjusting this balance per scenario may be a more important design choice going forward.\n\n## Conclusion\n\nThe bidirectional fusion mechanism of RepVL-PAN enables words to be not mere labels attached to predictions but active participants in feature construction. This design philosophy is not language-model-assisted nor vision-model-dominant, but an equal cooperative structure.\n\nFor developers, YOLO-World offers several insights:\n\n- **Language-vision fusion does not need to wait for large multimodal models to start.**\n\n  With careful design and streamlined architecture, even a speed-focused YOLO framework can achieve generalizable semantic alignment.\n\n- **Open-vocabulary tasks can move toward real-time inference.**\n\n  With modules like RepVL-PAN, we can evaluate deployment of detectors that recognize word-defined concepts in resource-constrained settings, beyond relying on fixed-class classifiers for dynamic scenes.\n\n- **Re-parameterizable language modules provide a practical trade-off design.**\n\n  They move the cost of encoding a chosen vocabulary to setup time. Changing that vocabulary requires fresh embeddings and weights; V100 speed does not settle edge-device performance.\n\nThe contribution of this paper is not just plugging CLIP into YOLO, but rethinking language-vision intersection at the feature pathway level, and enabling semantic alignment without deployment burden via re-parameterization.\n\nThe fusion of words and vision is no longer the privilege of high-cost architectures.\n","imageSizes":{"/content/papers/assets/7a09b97227f7f10a9b85e7c3cc305b05f5db9be415ef710cf9a6940180a3c663.jpg":{"width":1552,"height":650},"/content/papers/assets/4e32d04c0d7a7d42bc38361034c182f7631c8f6368ca38ca84d32c41b09819ed.jpg":{"width":1224,"height":1004},"/content/papers/assets/715c3e5fba7865d5fae13b73e712bc3cec2bc2ec43c82e2961cdec1b45fa58c1.jpg":{"width":1664,"height":648},"/content/papers/assets/86d83bc45036f49aeb0ebf03dabfbc53fe4f54fc63b559cea7ad99d77bb54ea7.jpg":{"width":872,"height":384},"/content/papers/assets/ac9c574c34aed9e6c31823b09311dc1e1d1cbf7a31d2d2a31a2419f3f13d1df9.jpg":{"width":968,"height":348},"/content/papers/assets/00bb2450a552bb62b732727aaa0741b3cde07bb518ad4698c6a8d0bcac1b6a7a.jpg":{"width":1522,"height":418}}}
