Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training

By Hongwei Xue, Yupan Huang, Bei Liu, Houwen Peng, Jianlong Fu, Houqiang Li, Jiebo Luo. 2021

100 citations recorded by the source corpus.

Vision-Language Pre-training (VLP) aims to learn multi-modal representations from image-text pairs and serves for downstream vision-language tasks in a fine-tuning fashion. The dominant VLP models adopt a CNN-Transformer architecture, which embeds images with a CNN, and then aligns images and text with a Transformer. Visual relationship between visual contents plays an important role in image understanding and is the basic for inter-modal alignment learning. However, CNNs have limitations in visual relation learning due to local receptive field's weakness in modeling long-range dependencies. Thus the two objectives of learning visual relation and inter-modal alignment are encapsulated in the same Transformer network. Such design might restrict the inter-modal alignment learning in the Transformer by ignoring the specialized characteristic of each objective. To tackle this, we propose a fully Transformer visual embedding for VLP to better learn visual relation and further promote inter-modal alignment. Specifically, we propose a metric named Inter-Modality Flow (IMF) to measure the interaction between vision and language modalities (i.e., inter-modality). We also design a novel masking optimization mechanism named Masked Feature Regression (MFR) in Transformer to further promote the inter-modality learning. To the best of our knowledge, this is the first study to explore the benefit of Transformer for visual feature learning in VLP. We verify our method on a wide range of vision-language tasks, including Image-Text Retrieval, Visual Question Answering (VQA), Visual Entailment and Visual Reasoning. Our approach not only outperforms the state-of-the-art VLP performance, but also shows benefits on the IMF metric.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhe Gan
2. Jianfeng Gao
3. Lijuan Wang
4. Zicheng Liu
5. Linjie Li
6. Jianfeng Wang
7. Jianwei Yang
8. Jingjing Liu

In Computer Vision and Pattern Recognition

1. Jianfeng Gao
2. Zhe Gan
3. Luc Van Gool
4. Dacheng Tao
5. Cordelia Schmid
6. Lijuan Wang
7. Zicheng Liu
8. Jianwei Yang

In Multimodal Learning

1. Zhe Gan
2. Lijuan Wang
3. Linjie Li
4. Jianfeng Gao
5. Zicheng Liu
6. Jianfeng Wang
7. Xiaowei Hu
8. Zhengyuan Yang

Citations in the corpus

• Scaling Up Vision-Language Pretraining for Image Captioning (2021, background): “More recently, instead of using object detectors for image feature extraction, end-to-end VLP based on convolution networks and transformers are becoming popular [17,18,22,27,49].”
• Florence: A New Foundation Model for Computer Vision (2021, background): “Recently, there is an increasing trend (Huang et al., 2021; Xue et al., 2021; Wang et al., 2021; Kim et al., 2021; Dou et al., 2021) of end-to-end approaches to reduce dependency on the object bounding box, which instead consider grid-based feature representations as the fine-grained features for…”
• GIT: A Generative Image-to-text Transformer for Vision and Language (2022, methodology): “For example, MLM and ITM are widely adopted pre-training tasks (Li et al., 2020; Kim et al., 2021; Zhang et al., 2021a; Wang et al., 2020; Xue et al., 2021; Lu et al., 2019; Tan & Bansal, 2019).”
• Playing Lottery Tickets with Vision and Language (2021, methodology): “While most of these methods rely on an object detection module to extract visual features offline, recently, end-to-end VLP methods (Huang et al. 2020, 2021; Kim, Son, and Kim 2021; Xue et al. 2021; Li et al. 2021; Dou et al. 2021) are becoming increasingly popular.”
• LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking (2022, methodology): “Still, most rely on separate self-attention networks to learn visual features; thus, their computational cost is not reduced [11, 27, 52].”
• Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, methodology): “30 PIM [154] NeurIPS-2021 image-text Trans MLM, MML, MOR Measure and reveal the V+L fusion using the proposed inter-modality flow metric 48M -”
• Multimodal Learning With Transformers: A Survey (2022, methodology): “In general, existing multimodal Transformer models [9], [199], [263] are superior only for specific MML tasks, as they are designed specifically for only a subset of specific tasks [137], [142], [212], [249], [260], [261], [265], [266].”
• An Empirical Study of Training End-to-End Vision-and-Language Transformers (2021, methodology): “Recent works [25,29,60] that tried to adopt vision trans-formers have not shown satisfactory performance and typically underperform state-of-the-art region-feature-based VLP models ( e.g. , VinVL [65]).”
• CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Alignment (2022, background): “Vision-Language Pre-Training End-to-end models (Lei et al., 2021; Xue et al., 2022; Zellers et al., 2021; Fu et al., 2021; Huang et al., 2020; 2021b; Xue et al., 2021; Li et al., 2021; Kim et al., 2021; Huang et al., 2021a; Sun et al., 2022) for vision-language pre-training are replacing the…”
• Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone (2022, methodology): “In these approaches, convolution networks or vision transformers [15] are used as the image backbone, with additional transformer layers for modeling multimodal fusion [26, 25, 31, 74, 35, 70].”