VinVL: Making Visual Representations Matter in Vision-Language Models

By Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, Jianfeng Gao. 2021

175 citations recorded by the source corpus.

This paper presents a detailed study of improving visual representations for vision language (VL) tasks and develops an improved object detection model to provide object-centric representations of images. Compared to the most widely used \emph{bottom-up and top-down} model \cite{anderson2018bottom}, the new model is bigger, better-designed for VL tasks, and pre-trained on much larger training corpora that combine multiple public annotated object detection datasets. Therefore, it can generate representations of a richer collection of visual objects and concepts. While previous VL research focuses mainly on improving the vision-language fusion model and leaves the object detection model improvement untouched, we show that visual features matter significantly in VL models. In our experiments we feed the visual features generated by the new object detection model into a Transformer-based VL fusion model \oscar \cite{li2020oscar}, and utilize an improved approach \short\ to pre-train the VL model and fine-tune it on a wide range of downstream VL tasks. Our results show that the new visual features significantly improve the performance across all VL tasks, creating new state-of-the-art results on seven public benchmarks. We will release the new object detection model to public.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

UFO: A UniFied TransfOrmer for Vision-Language Representation Learning (2021, methodology): “The image can be encoded as object features [5,6,13,18,27,29,30,30,32,39,42,47,53, 54] through a Faster RCNN [36], as grid features [20, 22] from a convolutional neural network, or as patch features [25] from a linear projection on raw pixels.”
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation (2021, background): “Furthermore, ALBEF does not require object detectors, a major computation bottleneck for many existing methods [1, 2, 3, 8, 17].”
Unifying Vision-and-Language Tasks via Text Generation (2021, neutral): “…models (Lu et al., 2019; Tan & Bansal, 2019; Chen et al., 2020; Huang et al., 2020; Li et al., 2020b; Cho et al., 2020; Radford et al., 2021; Zhang et al., 2021) and video+text pretraining models (Sun et al., 2019b;a; Li et al., 2020a; Zhu & Yang, 2020; Miech et al., 2020) have also shown…”
Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models (2023, methodology): “To better extract visual features, frozen pre-trained image encoders are adopted in the early works [9, 25, 54], and recent LiT [52] apply the CLIP pre-training [30] with frozen ViT model [51].”
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models (2023, neutral): “…the image encoder, including the early work which adopts a frozen object detector to extract visual features (Chen et al., 2020; Li et al., 2020; Zhang et al., 2021), and the recent LiT (Zhai et al., 2022) which uses a frozen pre-trained image encoder for CLIP (Radford et al., 2021)…”
GIT: A Generative Image-to-text Transformer for Vision and Language (2022, methodology): “During pre-training, Masked Language Modeling (MLM) and Image-Text Matching (ITM) tasks have been widely used [106, 26, 66, 127, 13, 23, 105, 54].”
ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision (2021, methodology): “, 2020b) and VinVL (Zhang et al., 2021), we use the pair method.”
Compressing Visual-linguistic Model via Knowledge Distillation (2021, background): “[81] demonstrate the significant role of visual features in VL pre-training and looks for more effective visual representations from a larger object detector.”
EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought (2023, methodology): “For example, models like Uniter [18], Oscar [19], VinVL [20], and LiT [21] freeze the image encoder, while Frozen [22] and VGPT [23] freeze the language model.”
GLaMM: Pixel Grounding Large Multimodal Model (2023, unknown): “CIDEr SPICE CIDEr SPICE VinVLM [55] 95.5 13.5 - - LEMON [12] 106.8 14.1 - - SimVLM [46] 110.3 14.5 - - CoCa [54] 120.6 15.5 - - BLIP [23] 113.2 14.7 - - BLIP-2 [24] 121 region-specific captions given an image, a user-specified region via a bounding box and related text.”
E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning (2021, background): “These two tasks can help learn high-quality visual representations (Zhang et al., 2021; Desai and Johnson, 2020).”
Image Scene Graph Generation (SGG) Benchmark (2021, methodology): “The features exacted from our large models, when used as the input of downstream vision-language models, achieve SOTA performance on seven major vision-language tasks [23].”
Visual Commonsense in Pretrained Unimodal and Multimodal Models (2022, background): “Most of them learn joint 080 image and text representations from cross-modal081 training of transformers with self-attention, includ-082 ing LXMERT (Tan and Bansal, 2019), ViLBERT083 (Lu et al., 2019), UNITER (Chen et al., 2020), etc.084 Oscar (Li et al., 2020) additionally uses object tags085 in im”
Language Grounding with 3D Objects (2021, background): “There has been a recent proliferation of off-the-shelf, increasingly large-scale pretrained vision and language alignment models applicable for language grounding in robotics [1, 6, 7, 8, 47].”
Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarization (2021, background): “These models usually receive a pair of text and image as input, where the image is processed into objects (Zhang et al., 2021), patches (Kim et al.”
Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Loss (2021, background): “…be concluded as: • We propose a visual-semantic data decomposing and multi-task constructing scheme for VTR task, and it is capable of being extended to other cross-modal tasks, such as image text generation, image text retrieval, image caption (Ding et al. 2021; Li et al. 2020; Zhang et al. 2021).”
Transform-Retrieve-Generate: Natural Language-Centric Outside-Knowledge Visual Question Answering (2022, background): “This task has been well studied in the research communities, and numerous cross-modal methods have achieved state-of-the-art performance [6,14,21,30,31, 34, 50, 63, 67, 69].”