{"notes":[{"id":"multimodality/pixelbert","category":"multimodality","legacyPath":"multimodality/2004-pixelbert","legacySlug":"2004-pixelbert","legacyPaths":[],"title":"Pixel-BERT","subtitle":"The Language of Pixels","author":"Z. Yuan","paperDate":"2020-04-02","publishedOn":"2024-06-04","paperTitle":"Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers","paperUrl":"https://arxiv.org/abs/2004.00849","bibliography":{"title":"Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers","authors":["Zhicheng Huang","Zhaoyang Zeng","Bei Liu","Dongmei Fu","Jianlong Fu"],"url":"https://arxiv.org/abs/2004.00849","pdfUrl":"https://arxiv.org/pdf/2004.00849","codeUrl":null,"preprint":{"id":"2004.00849","submittedOn":"2020-04-02","url":"https://arxiv.org/abs/2004.00849"},"publication":null,"citation":"@misc{multimodalitypixelbert2020,\n  title = {{Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers}},\n  author = {Zhicheng Huang and Zhaoyang Zeng and Bei Liu and Dongmei Fu and Jianlong Fu},\n  year = {2020},\n  eprint = {2004.00849},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2004.00849}\n}\n","citationPath":"/content/papers/citations/multimodality/pixelbert.c08c8ff05b89.bib"},"thumbnail":{"src":"/content/papers/assets/6bef1a64980cdb97551b5daaf2d3ff21f4bdb4b5bc852b778a2b24b84959a78e.jpg","width":1224,"height":628},"locale":"en","path":"/content/papers/en/multimodality/pixelbert.d7f7b2a07b45.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"solution","text":"Solution","depth":2},{"id":"pixel-bert-model-design","text":"Pixel-BERT Model Design","depth":3},{"id":"pixel-feature-encoding","text":"Pixel Feature Encoding","depth":3},{"id":"sentence-feature-encoding","text":"Sentence Feature Encoding","depth":3},{"id":"cross-modal-learning","text":"Cross-Modal Learning","depth":3},{"id":"pre-training-methods","text":"Pre-Training Methods","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"vqa-performance","text":"VQA Performance","depth":3},{"id":"nlvr2-performance","text":"NLVR2 Performance","depth":3},{"id":"image-text-retrieval","text":"Image-Text Retrieval","depth":3},{"id":"key-design-elements","text":"Key Design Elements","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"multimodality/vilt","relation":"comparison","topic":"Learning image-text features from pixels"},{"id":"multimodality/simvlm","relation":"comparison","topic":"Learning image-text features from pixels"}]},{"id":"multimodality/vilt","category":"multimodality","legacyPath":"multimodality/2102-vilt","legacySlug":"2102-vilt","legacyPaths":[],"title":"ViLT","subtitle":"Enter the Scene","author":"Z. Yuan","paperDate":"2021-02-05","publishedOn":"2024-04-05","paperTitle":"ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision","paperUrl":"https://arxiv.org/abs/2102.03334","bibliography":{"title":"ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision","authors":["Wonjae Kim","Bokyung Son","Ildoo Kim"],"url":"https://arxiv.org/abs/2102.03334","pdfUrl":"https://arxiv.org/pdf/2102.03334","codeUrl":"https://github.com/dandelin/vilt","preprint":{"id":"2102.03334","submittedOn":"2021-02-05","url":"https://arxiv.org/abs/2102.03334"},"publication":{"venue":"ICML","year":2021,"kind":"conference","url":"https://proceedings.mlr.press/v139/kim21k.html"},"citation":"@misc{multimodalityvilt2021,\n  title = {{ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision}},\n  author = {Wonjae Kim and Bokyung Son and Ildoo Kim},\n  year = {2021},\n  eprint = {2102.03334},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2102.03334}\n}\n","citationPath":"/content/papers/citations/multimodality/vilt.1555ccf87693.bib"},"thumbnail":{"src":"/content/papers/assets/9365d24c758fb8098d2dee6d761d9398a0800b712acb97926f5aac408eea2cce.jpg","width":1224,"height":456},"locale":"en","path":"/content/papers/en/multimodality/vilt.55290e45b2d7.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solution","text":"Solution","depth":2},{"id":"vilt-design-philosophy","text":"ViLT Design Philosophy","depth":3},{"id":"vilt-design-details","text":"ViLT Design Details","depth":3},{"id":"pretraining-strategies","text":"Pretraining Strategies","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"performance-analysis","text":"Performance Analysis","depth":3},{"id":"key-contributions","text":"Key Contributions","depth":3},{"id":"complexity-analysis","text":"Complexity Analysis","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{"replace-the-visual-front-end-with-patch-projection":"vilt-design-details","three-training-signals-with-no-word-to-box-labels":"pretraining-strategies","experimental-results-and-discussion":"discussion","read-the-tasks-and-inputs-before-the-scores":"performance-analysis","ablations-duration-masking-and-a-failed-extra-objective":"key-contributions","treat-15-ms-as-a-benchmark-condition":"complexity-analysis"},"guide":null,"relatedReading":[{"id":"multimodality/visualbert","relation":"comparison","topic":"Joint image-text Transformer encoders"},{"id":"multimodality/vlbert","relation":"comparison","topic":"Joint image-text Transformer encoders"},{"id":"multimodality/pixelbert","relation":"comparison","topic":"Learning image-text features from pixels"}]},{"id":"multimodality/simvlm","category":"multimodality","legacyPath":"multimodality/2108-simvlm","legacySlug":"2108-simvlm","legacyPaths":[],"title":"SimVLM","subtitle":"Simplifying Things","author":"Z. Yuan","paperDate":"2021-08-24","publishedOn":"2024-06-04","paperTitle":"SimVLM: Simple Visual Language Model Pretraining with Weak Supervision","paperUrl":"https://arxiv.org/abs/2108.10904","bibliography":{"title":"SimVLM: Simple Visual Language Model Pretraining with Weak Supervision","authors":["Zirui Wang","Jiahui Yu","Adams Wei Yu","Zihang Dai","Yulia Tsvetkov","Yuan Cao"],"url":"https://arxiv.org/abs/2108.10904","pdfUrl":"https://arxiv.org/pdf/2108.10904","codeUrl":null,"preprint":{"id":"2108.10904","submittedOn":"2021-08-24","url":"https://arxiv.org/abs/2108.10904"},"publication":{"venue":"ICLR","year":2022,"kind":"conference","url":"https://openreview.net/pdf?id=GUrhfTuf_3"},"citation":"@misc{multimodalitysimvlm2021,\n  title = {{SimVLM: Simple Visual Language Model Pretraining with Weak Supervision}},\n  author = {Zirui Wang and Jiahui Yu and Adams Wei Yu and Zihang Dai and Yulia Tsvetkov and Yuan Cao},\n  year = {2021},\n  eprint = {2108.10904},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2108.10904}\n}\n","citationPath":"/content/papers/citations/multimodality/simvlm.3f1debdaa343.bib"},"thumbnail":{"src":"/content/papers/assets/3b0abd9539f567530912cf12033a3707473305523048b725fdac2b67934909fd.jpg","width":1224,"height":704},"locale":"en","path":"/content/papers/en/multimodality/simvlm.d1fc3a0c4d0c.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"solution","text":"Solution","depth":2},{"id":"simvlm-model-design","text":"SimVLM Model Design","depth":3},{"id":"pretraining-strategy","text":"Pretraining Strategy","depth":3},{"id":"datasets","text":"Datasets","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"how-does-the-model-perform","text":"How Does the Model Perform?","depth":3},{"id":"how-is-the-zero-shot-performance","text":"How Is the Zero-shot Performance?","depth":3},{"id":"what-contributes-to-the-models-success","text":"What Contributes to the Model’s Success?","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{"defining-the-problem":"problem-definition","one-image-half-a-sentence":"simvlm-model-design","doesnt-this-look-like-vl-t5":"simvlm-model-design","image-text-and-data":"datasets","experimental-results-and-discussion":"discussion","main-benchmarks-check-the-model-and-metric":"how-does-the-model-perform","what-happens-without-target-task-fine-tuning":"how-is-the-zero-shot-performance","which-part-earns-its-keep":"what-contributes-to-the-models-success"},"guide":null,"relatedReading":[{"id":"multimodality/pixelbert","relation":"comparison","topic":"Learning image-text features from pixels"},{"id":"multimodality/vilt","relation":"comparison","topic":"Learning image-text features from pixels"}]}],"text":"In the past, when discussing the integration of vision and language models, most researchers followed a conventional standard procedure:\n\n1. Use object detection models like Faster R-CNN to extract region features from images.\n2. Combine these features with language models for cross-modal learning.\n3. Explore various designs for cross-modal learning.\n\nBut who said a pre-trained object detection model must be the first step in the image encoding process?\n\nMaybe no one explicitly stated it, but it became a common practice over time.\n\nFor researchers, it is often easier to follow previous works, make improvements, and achieve a slight performance increase of 1% to 3% to produce a commendable paper.\n\nThis trend continues until a breakthrough method is proposed or the evaluation dataset's performance saturates, leading to the next wave of changes.\n\nFuture researchers might question: Why did people back then rely so much on Transformer models?\n\nThe authors challenge the convention of extracting fixed detector-region features before image–text fusion, not the validity of object detection as a task.\n\nObject detection models are typically designed for specific visual tasks, potentially introducing biases that are not aligned with language understanding, especially when the visual representation capability of these models is limited by specific task categories.\n\nPixel-BERT offers a fresh perspective. It no longer relies on pre-trained object detection models but learns visual encoding directly from image pixels. This approach captures \"pixel-level\" visual information, including shapes, textures, and spatial relationships, which might be lost in object detection models. Combined with language encoding, this model effectively captures the subtle interactions between vision and language, learning richer and more detailed cross-modal representations.\n\n## Problem Definition\n\nThe authors identify and define the following issues in this paper:\n\n1. **Cross-Modal Semantic Gap**\n\n   Current cross-modal learning (especially between vision and language) faces challenges due to the semantic gap between visual and language representations.\n\n2. **Limitations of Visual Features**\n\n   Previous methods tend to use region-based visual features extracted from object detection models. However, these region-based feature extractors are designed for specific visual tasks, leading to information gaps when combined with language understanding. Important visual information like object shapes, spatial relationships between overlapping objects, and object states might be lost.\n\n3. **Constraints of Existing Visual Features**\n\n   Region-based visual features currently used have limitations in representing broader visual semantics such as scenes and emotions, which object detection models might not capture.\n\n## Solution\n\n### Pixel-BERT Model Design\n\n![Pixel-BERT Model Architecture](/content/papers/assets/6bef1a64980cdb97551b5daaf2d3ff21f4bdb4b5bc852b778a2b24b84959a78e.jpg)\n\nEarlier models commonly use learned CNN/ROI feature vectors from Faster R-CNN proposals. They do not simply feed the raw RGB values inside each box to the image–text model.\n\nBut this method has its limitations:\n\n- **Information Loss**: Bounding boxes might include some irrelevant background or miss parts of important objects, leading to information loss.\n- **Task-Specific Constraints**: These object detection models are designed for specific visual tasks and may not be suitable for other cross-modal tasks.\n\n### Pixel Feature Encoding\n\nPixel-BERT uses jointly trainable CNN grid features. “Pixel” here refers to a feature-map position, not one Transformer token per original image pixel. The CNN still starts from ImageNet-pretrained weights.\n\n1. **Pixel Learning**: Using CNNs like ResNet, visual features are learned directly from the entire image, considering every pixel, not just those within bounding boxes.\n2. **Feature Extraction**:\n   - **Image Input**: The input image first enters the model.\n   - **CNN Backbone**: A pre-trained neural network is used to extract visual features from the input image.\n   - **Convolution (Conv)**: Further processing of features through convolutional layers.\n   - **Pooling**: Add 2×2 max pooling, giving an overall spatial downsampling factor of about 64 along each axis.\n   - **Random Sample**: During pre-training, randomly selects 100 feature-map positions for the Transformer to limit its input length.\n3. **Semantic Encoding**: Each pixel feature is augmented with a semantic encoding vector (sv). This special vector differentiates visual encoding from language encoding. All pixel features share the same sv, acting as a bias term added to the CNN backbone.\n\n### Sentence Feature Encoding\n\nThe model follows BERT’s method to encode the language information of sentences.\n\n1. **Sentence Tokenization**\n\n   A given sentence is tokenized into a sequence of words, a standard step in natural language processing, breaking the sentence into its constituent words.\n\n2. **Word Tokenization**\n\n   Using the WordPiece model, each word is tokenized. This method, commonly used in BERT, breaks a word into smaller recognizable segments or subwords, useful for handling rare or out-of-vocabulary words.\n\n3. **Encoding Matrix Transformation**\n\n   Each token is encoded into a vector through an encoding matrix, mapping each unique token or subword into a fixed-dimensional vector space.\n\n4. **Positional Encoding**\n\n   Positional information is crucial for models like Transformers. Thus, similar to BERT, Pixel-BERT adds positional encoding to capture the position of each token in the sequence.\n\n5. **Combining Encoded Features**\n\n   Each input token is represented as $\\hat w_i=\\operatorname{LayerNorm}(w_i+p_i+s_w)$. Since $s_w$ is shared across text positions, it can be absorbed into learned positional embeddings and is omitted as a separate term. These are pre-Transformer inputs.\n\n6. **Final Language Representation**\n\n   The resulting token sequence is passed to the cross-modal Transformer; it is not yet a final contextualized sentence vector.\n\n### Cross-Modal Learning\n\nThe model uses Transformers to learn cross-modal attention, particularly the relationships between CNN feature-map positions and language tokens. All encoded features are combined into a long input sequence, with special tokens like [CLS] and [SEP] to distinguish its structure. This design lets gradients from the cross-modal task reach the CNN backbone.\n\n### Pre-Training Methods\n\nThroughout this section, “pixel features” means CNN feature-map tokens. Robustness and reduced overfitting are motivations supported by the experiments, not guarantees under arbitrary occlusion.\n\nTo learn general visual and sentence representations for vision and language-related tasks, the authors adopted a self-supervised approach for pre-training on large aggregated datasets.\n\n- **Masked Language Model (MLM)**\n\n  About 15% of **language tokens** become masked-prediction targets. The model uses remaining text and CNN visual tokens to predict the original words. It does not reconstruct image pixels; random visual-feature sampling is a separate mechanism.\n\n  Traditional BERT is unimodal, meaning it performs masked prediction only within language tokens. However, due to Pixel-BERT’s cross-modal nature, it leverages visual tokens for prediction. This capability makes Pixel-BERT more adept at handling contexts that might be ambiguous or challenging to interpret purely from language. For instance, when masked tokens in the language correlate directly with objects or scenes in the image, visual tokens provide crucial context, aiding the model in making accurate predictions.\n\n- **Image-Text Matching (ITM)**\n\n  Image-Text Matching (ITM) is a complex strategy designed to evaluate the model's ability to understand the deep semantic relationships between images and their corresponding text descriptions. It’s not just about finding direct matches but about recognizing nuanced connections.\n\n  ITM classifies the fused `[CLS]` representation, training with equal numbers of original pairs and randomly shuffled negatives. A blossom caption paired with a blossom image is a positive example, while pairing it with a desert image is negative. The objective encourages a higher positive score.\n\n  However, to enhance the model's discrimination ability, \"negative pairs\" (where the image and text do not match) are introduced during training. For instance, pairing the text \"under the cherry blossoms in spring\" with an image showing a desert landscape. This design helps the model learn when the image and text are mismatched. Through this combination of positive and negative training pairs, the model can more accurately assess image-text match quality in real-world applications.\n\n- **Pixel Random Sampling**\n\n  Pixel Random Sampling is a unique strategy in Pixel-BERT aimed at enhancing the model's robustness and reducing overfitting risk.\n\n  Pre-training samples 100 CNN feature-map positions as visual tokens for the Transformer. The whole image still passes through the CNN; this reduces Transformer input length rather than discarding raw image pixels before encoding.\n\n  This approach has several advantages:\n\n  - **Enhanced Robustness**: By seeing a different subset of feature-map positions each time, the model is encouraged to learn from incomplete visual-token input.\n  - **Increased Efficiency**: Fewer visual tokens reduce the Transformer’s computation during pre-training; the whole image still passes through the CNN.\n  - **Reduced Overfitting Risk**: Changing the visual-token subset may discourage reliance on fixed feature positions; the ablation measures downstream scores, not robustness under arbitrary occlusion.\n\n  Sampling is used only in pre-training; downstream fine-tuning and testing use the complete CNN feature grid. The authors choose this to avoid losing information during short fine-tuning runs and to keep train/test inputs consistent, not as a universal requirement for every task.\n\n## Discussion\n\n### VQA Performance\n\n![Pixel-BERT VQA Performance](/content/papers/assets/2b80a937b1a2d9944b860a5d49955516c4f1a96582892e842714a6fd85ea2199.jpg)\n\n- The ResNet-50 version scores 71.35/71.42 on VQA test-dev/test-std. Its 71.35 test-dev score exceeds the listed ViLBERT 70.55 and VisualBERT 70.80, but trails VL-BERT 71.79 and LXMERT 72.42.\n\n- The ResNeXt-152 version scores 74.45/74.55 on test-dev/test-std. This is the historical comparison in the cited paper; backbone, input resolution and training configuration also influence results.\n- The table lists UNITER at 72.46 on test-std; the paper's prose separately cites UNITER-large at 73.40. They are different comparison rows, and the latter is not Pixel-BERT’s score.\n\nThese results support learning visual and language attention from CNN feature-map positions. Differences in visual backbones and pre-training settings still matter in comparisons across models.\n\n### NLVR2 Performance\n\n![Pixel-BERT NLVR2 Performance](/content/papers/assets/ec2244d439329ee14c38f2dc890c92ed11cee21a3117698f517b21dac9044c22.jpg)\n\nIn the natural language visual reasoning (NLVR2) task, the goal is to predict whether a language description corresponds to the given image pair. Pixel-BERT takes two image-language pairs as input, obtaining two encoded vectors from the [CLS] token. These vectors are concatenated and then used to train a classifier to predict \"true\" or \"false\" using cross-entropy loss.\n\nThe ResNeXt-152 version scores 76.5/77.2 on NLVR2 dev/test-P, above the listed LXMERT and UNITER Pair but below UNITER Pair-biattn at 77.1/77.9. ResNet-50 scores 71.7/72.4; the backbone setting must be specified.\n\nPixel-BERT performs well not only on VQA but also on this task of evaluating a language description against two images. These results indicate its flexibility in adapting to different input formats.\n\n### Image-Text Retrieval\n\nRetrieval fine-tuning updates only Transformer parameters.\n\n![Pixel-BERT Image-Text Retrieval Performance](/content/papers/assets/8efdc1f31aca77c4fd28ae57764814ff0f11f8ebfce9a1adf6ed64f6447e7c88.jpg)\n\n- The figure is Table 4's Flickr30K 1K test set, not MS-COCO. With ResNeXt-152, Pixel-BERT reaches text retrieval (TR) R@1 of 87.0, above Unicoder-VL's 86.2 and UNITER's 84.7. Its image retrieval (IR) R@1 of 71.5 ties those two models.\n- The paper separately reports MS-COCO in Table 5. There, ResNeXt-152 scores TR R@1 of 84.9/63.6 and IR R@1 of 71.6/50.1 on the 1K/5K tests. The prose's gains of 0.6/0.3 points for TR and 1.9/1.7 for IR refer to particular baselines on these COCO splits, not to the Flickr30K figure.\n\nPixel-BERT shows strong image-text retrieval results, especially on MS-COCO text-to-image R@1. These are comparisons across models with different visual backbones and training settings.\n\n### Key Design Elements\n\n![Pixel-BERT Ablation Study](/content/papers/assets/4b50b06f369db54386de6572c32e2bc0b7db24ac2423c6a3cb70b608f3fe3bfd.jpg)\n\nThe authors attribute part of Pixel-BERT's effectiveness and excellent performance to several key design elements:\n\n1. **Choice of Pre-Training Tasks**\n\n   Against row 1 without pre-training, MLM alone in row 3 raises VQA from 63.50 to 71.13 and NLVR2 from 52.0 to 70.9. ITM alone in row 2 improves retrieval, but NLVR2 slips from 52.0 to 51.9.\n\n2. **Combining Different Pre-Training Tasks**\n\n   With the same ResNet-50 backbone and random sampling, combining MLM and ITM in row 5 improves the reported scores over either objective alone.\n\n3. **Random Pixel Sampling**\n\n   This design aims to enhance robustness and reduce overfitting. Holding the ResNet-50 backbone and MLM+ITM fixed, random sampling in row 5 versus all feature positions in row 4 raises VQA 70.84→71.35, TR 72.0→75.7, IR 57.7→59.8, and NLVR2 71.3→71.7.\n\n4. **Choice of Visual Backbone**\n\n   The model's performance also depends on the selected visual backbone. Using a more powerful ResNeXt-152 significantly improves performance, showcasing Pixel-BERT's ability to combine with strong visual backbones.\n\n## Conclusion\n\n![Pixel-BERT Visualization](/content/papers/assets/f7e4ca7951a75b8cd0540d0419bd1c219c1deb8bcf8fbc7e7167483d5e1b7f8b.jpg)\n\nThe visualization relates textual attention to image positions and offers qualitative clues about model behavior.\n\nIn recent research developments, pre-training models in the vision and language field have achieved remarkable results. To address the limitations of region-based visual representations, the authors introduce a CNN-based visual encoder, cleverly combined with multi-modal Transformers, creating the Pixel-BERT model.\n\nPixel-BERT explores the connections between image features and text, with its attention maps offering a view of the visual-language representations it learns. Its random sampling strategy improves the measured downstream results. Pre-trained on Visual Genome and MS-COCO, the model achieves strong results on the evaluated tasks. The authors also plan to pre-train on more datasets and explore self-supervised visual-content tasks to improve it further.\n\n\n> [!TIP]\n> \n> Setting aside the authors' own conclusions, the biggest contribution is replacing the \"object detection model\" while still matching or even surpassing the performance of state-of-the-art works like UNITER!\n> \n> It means we no longer need to plug in a bulky, possibly pre-trained Faster R-CNN (or other architectures). This is definitely a notable and particularly delightful achievement for engineers!\n","imageSizes":{"/content/papers/assets/6bef1a64980cdb97551b5daaf2d3ff21f4bdb4b5bc852b778a2b24b84959a78e.jpg":{"width":1224,"height":628},"/content/papers/assets/2b80a937b1a2d9944b860a5d49955516c4f1a96582892e842714a6fd85ea2199.jpg":{"width":768,"height":596},"/content/papers/assets/ec2244d439329ee14c38f2dc890c92ed11cee21a3117698f517b21dac9044c22.jpg":{"width":812,"height":608},"/content/papers/assets/8efdc1f31aca77c4fd28ae57764814ff0f11f8ebfce9a1adf6ed64f6447e7c88.jpg":{"width":1024,"height":611},"/content/papers/assets/4b50b06f369db54386de6572c32e2bc0b7db24ac2423c6a3cb70b608f3fe3bfd.jpg":{"width":1024,"height":358},"/content/papers/assets/f7e4ca7951a75b8cd0540d0419bd1c219c1deb8bcf8fbc7e7167483d5e1b7f8b.jpg":{"width":1024,"height":766}}}
