{"notes":[{"id":"multimodality/vinvl","category":"multimodality","legacyPath":"multimodality/2101-vinvl","legacySlug":"2101-vinvl","legacyPaths":[],"title":"VinVL","subtitle":"Revisiting Oscar","author":"Z. Yuan","paperDate":"2021-01-02","publishedOn":"2024-06-04","paperTitle":"VinVL: Revisiting Visual Representations in Vision-Language Models","paperUrl":"https://arxiv.org/abs/2101.00529v2","bibliography":{"title":"VinVL: Revisiting Visual Representations in Vision-Language Models","authors":["Pengchuan Zhang","Xiujun Li","Xiaowei Hu","Jianwei Yang","Lei Zhang","Lijuan Wang","Yejin Choi","Jianfeng Gao"],"url":"https://arxiv.org/abs/2101.00529v2","pdfUrl":"https://arxiv.org/pdf/2101.00529v2","codeUrl":"https://github.com/pzzhang/VinVL","preprint":{"id":"2101.00529","submittedOn":"2021-01-02","url":"https://arxiv.org/abs/2101.00529v2"},"publication":{"venue":"CVPR","year":2021,"kind":"conference","url":"https://openaccess.thecvf.com/content/CVPR2021/html/Zhang_VinVL_Revisiting_Visual_Representations_in_Vision-Language_Models_CVPR_2021_paper.html"},"citation":"@misc{multimodalityvinvl2021,\n  title = {{VinVL: Revisiting Visual Representations in Vision-Language Models}},\n  author = {Pengchuan Zhang and Xiujun Li and Xiaowei Hu and Jianwei Yang and Lei Zhang and Lijuan Wang and Yejin Choi and Jianfeng Gao},\n  year = {2021},\n  eprint = {2101.00529},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2101.00529v2}\n}\n","citationPath":"/content/papers/citations/multimodality/vinvl.1172342c1d76.bib"},"thumbnail":{"src":"/content/papers/assets/3d2a267cacc822414884a9ae3970774c767892d38e532ec87146a4fffc049f41.jpg","width":1224,"height":416},"locale":"en","path":"/content/papers/en/multimodality/vinvl.9b2df23d0100.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"object-detection-pretraining","text":"Object Detection Pretraining","depth":3},{"id":"attribute-injection","text":"Attribute Injection","depth":3},{"id":"improving-efficiency","text":"Improving Efficiency","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"performance-on-downstream-tasks","text":"Performance on Downstream Tasks","depth":3},{"id":"how-important-are-visual-features","text":"How Important Are Visual Features?","depth":3},{"id":"impact-of-visual-concept-diversity","text":"Impact of Visual Concept Diversity","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{"define-the-problem":"defining-the-problem","solve-the-problem":"solving-the-problem","what-does-oscar-receive":"solving-the-problem","train-a-detector-with-more-concepts-and-attributes":"object-detection-pretraining","keep-region-extraction-usable":"improving-efficiency","experimental-results-and-discussion":"discussion","seven-tasks-and-their-comparison-conditions":"performance-on-downstream-tasks","how-much-comes-from-vision-and-language-pretraining":"how-important-are-visual-features","vocabulary-and-region-quality":"impact-of-visual-concept-diversity"},"guide":null,"relatedReading":[{"id":"multimodality/oscar","relation":"background","topic":"Object information in image-text pretraining"}]},{"id":"multimodality/oscar","category":"multimodality","legacyPath":"multimodality/2004-oscar","legacySlug":"2004-oscar","legacyPaths":[],"title":"Oscar","subtitle":"The Anchors of Oscar","author":"Z. Yuan","paperDate":"2020-04-13","publishedOn":"2024-06-04","paperTitle":"Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks","paperUrl":"https://arxiv.org/abs/2004.06165","bibliography":{"title":"Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks","authors":["Xiujun Li","Xi Yin","Chunyuan Li","Pengchuan Zhang","Xiaowei Hu","Lei Zhang","Lijuan Wang","Houdong Hu","Li Dong","Furu Wei","Yejin Choi","Jianfeng Gao"],"url":"https://arxiv.org/abs/2004.06165","pdfUrl":"https://arxiv.org/pdf/2004.06165","codeUrl":null,"preprint":{"id":"2004.06165","submittedOn":"2020-04-13","url":"https://arxiv.org/abs/2004.06165"},"publication":{"venue":"ECCV","year":2020,"kind":"conference","url":"https://arxiv.org/abs/2004.06165"},"citation":"@misc{multimodalityoscar2020,\n  title = {{Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks}},\n  author = {Xiujun Li and Xi Yin and Chunyuan Li and Pengchuan Zhang and Xiaowei Hu and Lei Zhang and Lijuan Wang and Houdong Hu and Li Dong and Furu Wei and Yejin Choi and Jianfeng Gao},\n  year = {2020},\n  eprint = {2004.06165},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2004.06165}\n}\n","citationPath":"/content/papers/citations/multimodality/oscar.f753816326f7.bib"},"thumbnail":{"src":"/content/papers/assets/a3a3bdfd3ba2e6387f71c701021f8048603307f1ed8674c75f2fae390cd34238.jpg","width":1224,"height":484},"locale":"en","path":"/content/papers/en/multimodality/oscar.1e7424bf62aa.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"solution","text":"Solution","depth":2},{"id":"oscar-model-design","text":"Oscar Model Design","depth":3},{"id":"pre-training-objectives","text":"Pre-training Objectives","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"parameter-efficiency-comparison","text":"Parameter Efficiency Comparison","depth":3},{"id":"model-performance-comparison","text":"Model Performance Comparison","depth":3},{"id":"qualitative-study","text":"Qualitative Study","depth":3},{"id":"ablation-study","text":"Ablation Study","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"multimodality/vinvl","relation":"next","topic":"Object information in image-text pretraining"},{"id":"multimodality/ernie-vil","relation":"comparison","topic":"Objects and structured semantics in vision-language pretraining"}]}],"text":"After reading the previous Oscar note, the follow-up research arrived right away.\n\n- [**Oscar**](/blog/papers/multimodality/oscar)\n\nThe authors of this paper think Oscar is good, but it still has a shortcoming:\n\n- It does not explore visual representations deeply enough.\n\nWhat, then, would count as going deeper? Let's look closely at the paper.\n\n## Defining the Problem\n\n![VinVL demo](/content/papers/assets/3d2a267cacc822414884a9ae3970774c767892d38e532ec87146a4fffc049f41.jpg)\n\nThe authors focus on the field of Visual-Language Pretraining (VLP), specifically on enhancing the visual representations generated by Object Detection (OD) models. The effectiveness and richness of visual features are crucial for the performance of models in various Vision-Language (VL) tasks. Here are the main points of concern:\n\n1. **Need for Rich Visual Representations**\n\n   Current VLP methods heavily rely on visual representations provided by object detection models. The authors argue that while these models offer valuable visual information for VL tasks, there is room for improvement, particularly in handling complex, diverse, and semantically rich image scenes.\n\n2. **Effectiveness of Cross-Modal Fusion Models**\n\n   VLP typically consists of two components: a pretrained object detection model and a cross-modal fusion model, which combines visual and language features. While much research focuses on improving the cross-modal model, the authors emphasize that stronger, semantically richer visual representations are equally critical.\n\n3. **Limitations of Existing Object Detection Models**\n\n   The authors note that the widely used OD models are often treated as a \"black box\" in literature and may have limitations in their training datasets and objectives, impacting their performance in various VL tasks.\n\n4. **Diversity of Visual Objects and Concepts**\n\n   In many practical applications, models must recognize and understand a wide range of visual objects and concepts. Current models, especially those trained on datasets like Open Images, may not fully capture and express this rich diversity of visual content.\n\n## Solving the Problem\n\n![Oscar's word, object-tag, and region-feature inputs](/content/papers/assets/a3a3bdfd3ba2e6387f71c701021f8048603307f1ed8674c75f2fae390cd34238.jpg)\n\nThis imported diagram is from the earlier Oscar paper; it shows the word, object-tag, and region-feature inputs that VinVL aims to improve, rather than a new VinVL architecture.\n\nThe main change in this paper is training a stronger object-and-attribute detector to provide VinVL visual features. The authors also use those features to train the improved cross-modal model Oscar+ and compare what each part contributes.\n\nVision-language models usually have two main parts: a Vision module and a cross-modal VL module. The Vision module converts images into semantic representations $q$ (such as detected object names) and high-dimensional latent representations $v$ (region feature vectors, as in a pretrained Faster R-CNN). These features are then combined with language input for VL tasks. This paper looks more deeply at how to improve the vision module through object-detection pretraining and attribute information:\n\n### Object Detection Pretraining\n\n- **Data Processing**\n\n  The authors use four major public datasets (COCO, OpenImagesV5, Objects365V1, and Visual Genome) for pretraining the object detection model. They employ strategies like class-aware sampling and balancing for imbalanced datasets to create a unified, large corpus.\n\n- **Model Architecture**\n\n  Earlier object-detection studies often favored FPN over C4, but this paper and its cited study found C4 region features more effective in the VL tasks they tested, so the authors chose C4.\n\n\n> [!TIP]\n> \n> **What is the C4 model?**\n> \n> This confused me when I first saw it, so I followed the paper to a [related architecture study](https://arxiv.org/abs/2001.03615). C4 is a detector configuration built around the ResNet $C_4$ stage feature map, usually at about 1/16 the input image scale. A $C_5$ block then processes each region. That study compares complete C4 and FPN region-feature extractors on VQA and other tasks. VinVL extends the comparison to pretraining weights and region-feature heads.\n\n\n- **Pretraining Strategy**\n\n  The authors combine classical and experiential methods to enhance the model's performance.\n\n  - Freezing the first convolutional layer, the first residual block, and all batch-normalization layers as part of the authors' detector training setup.\n  - Applying data augmentation techniques like horizontal flipping and multi-scale training to expand the dataset and improve model generalization.\n\n  The object detection model's basic structure is the \"X152-C4 architecture,\" with initial weights based on the ImageNet-5K dataset, trained over 1.8 million iterations with a batch size of 16 images.\n\n### Attribute Injection\n\n- **Pretrained Model and Attribute Branch**\n\n  - Pretrained OD model: first train an object detector on a larger dataset so it learns general visual features and basic detection ability.\n  - Attribute branch: add a branch to the pretrained OD model to predict attributes alongside objects.\n\n- **Fine-Tuning on Visual Genome (VG) Dataset**\n  - Attributes: The VG dataset provides 524 attribute categories (e.g., color, shape, size).\n  - Fine-Tuning Strategy: After adding the branch, fine-tune the model on VG so it learns attribute-related visual features as well as detecting objects.\n  - Attribute Loss Weight Adjustment: Increasing the attribute loss weight from 0.5 to 1.25 during fine-tuning to emphasize the importance of attribute learning.\n\nThis approach allows the model to not only detect objects but also predict multiple attributes on the VG dataset, demonstrating significant advantages over previous models.\n\n### Improving Efficiency\n\nObject detection models face computational challenges due to the diversity of visual features and attributes, especially during feature extraction.\n\n- **Non-Maximum Suppression (NMS) Challenge**\n\n  The diversity of visual objects and attributes requires extensive computational resources for class-aware NMS during post-processing, slowing down feature extraction.\n\n- **Efficiency Strategies**\n\n  - Using class-agnostic NMS: The final object-box postprocessing runs NMS once instead of once per class. Counting the RPN stage, the complete feature extractor still performs NMS twice.\n  - Adjusting Convolutional Layer Settings: Replacing dilated convolutions with non-dilated ones to further optimize computation.\n\nThe paper reports that these two substitutions considerably speed up region extraction without reducing accuracy on the tested downstream VL tasks; each substitution’s contribution requires a separate ablation.\n\n\n> [!TIP]\n> \n> **What is Dilated Convolution?**\n> \n> Dilated convolution, also called atrous convolution, spaces out where a kernel samples the input and thus expands its receptive field. With the same kernel size and channel counts, dilation alone does not necessarily increase multiply-add operations. The speed result here is for this detector implementation.\n\n\n## Discussion\n\n### Performance on Downstream Tasks\n\n![VinVL results](/content/papers/assets/495de9d9024d7569416cdb70cce7910f83d8beaa1fe196d79de8a959d3fecfc7.jpg)\n\n1. To evaluate model parameter efficiency, state-of-the-art models (SOTA) are categorized into three groups:\n\n   - SOTAS: Small models that achieved the best performance before Transformer-based VLP models.\n   - SOTAB: The best-performing VLP models comparable in size to BERT Base.\n   - SOTAL: The best-performing VLP models comparable in size to BERT Large.\n\n2. Models using VinVL features exceed the listed earlier best results on most metrics across seven tasks. The image-captioning B@4 score is an exception: VinVL-B is 40.9 and VinVL-L is 41.0, below SoTA-L's 41.7.\n3. VQA: Oscar+-B with VinVL features scores 75.95 on test-dev and 76.12 on test-std. The latter exceeds the listed InterBERT Large ensemble's 76.10 by only 0.02.\n4. GQA: Oscar+-B with VinVL features scores 64.65 on test-std, above the listed NSM score of 63.17; the authors call it the first VLP model at the time to surpass NSM.\n5. Image Captioning: VinVL-B reaches 140.6 CIDEr versus the listed previous best of 140.0. Its B@4 is 40.9 versus 41.7, so the result depends on which metric you read.\n6. NoCaps: The authors report that their COCO-captioning model with VinVL features, without VLP initialization, exceeds the then-listed human CIDEr reference. Combining VinVL features with VIVO visual-vocabulary pretraining raises CIDEr from 86.58 to 92.46. These are separate experimental settings.\n\n### How Important Are Visual Features?\n\n![VinVL ablation](/content/papers/assets/247c5ecc15e0d905ee91e4a6604e6ed21ed775d0a6634ab191203c133085088b.jpg)\n\nThe authors compare visual features from R101-C4 and the four-dataset-pretrained X152-C4 (VinVL), paired with no VLP, Oscar, or Oscar+, on VQA. The table has no \"VinVL features + original Oscar\" cell, so we can only interpret combinations it actually tests.\n\n1. **Model Comparison and Performance Improvement**\n\n   - Oscar versus Oscar+ and visual features: R101-C4 with Oscar scores 72.38; the same features with Oscar+ score 72.46. Oscar+ with VinVL features reaches 74.90, a total gain of 2.52 percentage points.\n   - Contribution Analysis: By the table's numbers, the Oscar+ change contributes 0.08/2.52, about 3.2% of the gain along this comparison path. The visual-feature change contributes 2.44/2.52, about 96.8%. The paper's prose summarizes these as 5%/95%, which are not the exact ratios of the displayed values.\n\n2. **Importance of Visual Representations**\n   - These VQA results show that changing the vision model or improving VLP can raise the score, but the size of each improvement depends on its own baseline.\n   - Without VLP, changing to VinVL features raises 68.52 to 71.34 (+2.82). With R101-C4 features, moving from no VLP to Oscar+ raises 68.52 to 72.46 (+3.94). Adding those gains predicts 75.28; the observed combination is 74.90. They are close but not identical.\n\n### Impact of Visual Concept Diversity\n\nThe paper explores how the diversity of visual concepts, particularly object and attribute vocabularies, affects VL models, especially in the VQA task.\n\n- **Richness of Visual Concept Vocabulary**\n\n  ![VinVL concepts](/content/papers/assets/3311d65b80c765952221db5f27d62672b15d6feeca6fe4a0a7e185485fb08e3b.jpg)\n\n  - In this R50-C4 + BERT-B experiment, VG w/o attr (1,594 classes) scores 66.51, above VG-obj (317 classes) at 64.25 and ImageNet (1,000 classes) at 66.13. Data sources and feature types differ too, so this table alone cannot attribute the full gap to vocabulary size.\n  - The authors point out that common object classes omit concepts such as \"sky\" and \"water.\" The examples support the possibility that a richer vocabulary helps VQA.\n  - In the VG settings, adding 524 attributes raises VQA from 66.51 (VG w/o attr) to 67.86 (VG). The 4Sets→VG setting reaches 68.39 but also has larger-scale pretraining, so that gap should not be credited to attributes alone.\n  - The R50-C4 comparisons also show a benefit from richer visual pretraining in these tested settings.\n\n- **Impact of Rich Visual Semantics on VL Tasks**\n\n  ![VinVL concepts](/content/papers/assets/5afe46f7ad959f7afb8e1e019cc936ab8b214d14fe2bfe3e0147de74026a87ef.jpg)\n\n  - When the table replaces proposed regions with COCO ground-truth boxes (GT-Obj / GT-Obj&Stuff), both models score below their Anderson- or VinVL-region settings on VQA. The table compares VQA performance across region sources.\n  - Holding the model fixed while changing the region source, or holding regions fixed while changing model weights, changes VQA scores. This supports the authors' focus on rich visual semantics, but the effect cannot all be assigned to boxes or weights alone.\n\n## Conclusion\n\nAlthough VinVL attempts to incorporate more visual features, it still adheres to the concept of categories. The selected objects and attributes might not be sufficiently rich or may carry inherent biases, limiting the model's ability to capture all visual concepts fully and affecting its broader application potential.\n\nAdditionally, the large scale and complexity of the model pose significant challenges. The authors seem aware of this and have included a section on improving efficiency to guide users on how to enhance inference speed.\n\nVinVL's large architecture and complex computation demand substantial resources, which limits its applicability and widespread adoption in resource-constrained environments.\n\nFinally, the integration of attributes and visual concepts presents another significant challenge. Training models across multiple data sources to cover a wide range of visual concepts and attributes requires ensuring data consistency and leveraging their advantages fully, which remains a major hurdle in developing a highly consistent and broadly applicable model.\n\nDespite these challenges, VinVL demonstrates strong capabilities and potential in various VL tasks, particularly in object recognition and understanding visual attributes. Its exceptional performance across multiple tasks highlights its value as a visual-language pretraining model.\n","imageSizes":{"/content/papers/assets/3d2a267cacc822414884a9ae3970774c767892d38e532ec87146a4fffc049f41.jpg":{"width":1224,"height":416},"/content/papers/assets/a3a3bdfd3ba2e6387f71c701021f8048603307f1ed8674c75f2fae390cd34238.jpg":{"width":1224,"height":484},"/content/papers/assets/495de9d9024d7569416cdb70cce7910f83d8beaa1fe196d79de8a959d3fecfc7.jpg":{"width":2554,"height":450},"/content/papers/assets/247c5ecc15e0d905ee91e4a6604e6ed21ed775d0a6634ab191203c133085088b.jpg":{"width":1024,"height":248},"/content/papers/assets/3311d65b80c765952221db5f27d62672b15d6feeca6fe4a0a7e185485fb08e3b.jpg":{"width":1024,"height":117},"/content/papers/assets/5afe46f7ad959f7afb8e1e019cc936ab8b214d14fe2bfe3e0147de74026a87ef.jpg":{"width":1024,"height":228}}}
