VinVL: Revisiting Visual Representations in Vision-Language Models

By Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, Jianfeng Gao. 2021

1,112 citations recorded by the source corpus.

This paper presents a detailed study of improving visual representations for vision language (VL) tasks and develops an improved object detection model to provide object-centric representations of images. Compared to the most widely used bottom-up and top-down model [2], the new model is bigger, better-designed for VL tasks, and pre-trained on much larger training corpora that combine multiple public annotated object detection datasets. Therefore, it can generate representations of a richer collection of visual objects and concepts. While previous VL research focuses mainly on improving the vision-language fusion model and leaves the object detection model improvement untouched, we show that visual features matter significantly in VL models. In our experiments we feed the visual features generated by the new object detection model into a Transformer-based VL fusion model OSCAR [20], and utilize an improved approach OSCAR+ to pre-train the VL model and fine-tune it on a wide range of downstream VL tasks. Our results show that the new visual features significantly improve the performance across all VL tasks, creating new state-of-the-art results on seven public benchmarks. Code, models and pre-extracted features are released at https://github.com/pzzhang/VinVL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Towards Language-Free Training for Text-to-Image Generation (2021, methodology): “Note that MS-COCO image-text pairs were used to train the author-provided VinVL image captioning model, so the MS-COCO comparison is unfairly biased in favor of the baseline due to this information leakage.”
CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIP (2022, background): “, image-text matching [9, 27, 34], image captioning [44], Visual Question Answering [9,20,33,34]).”
How Much Can CLIP Benefit Vision-and-Language Tasks? (2021, background): “We hypothesize that such decrease is due to the lack of visual localization inside the ViT feature map since different pooling strategies may affect the localization ability (Zhou et al., 2016).”
Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation (2022, methodology): “Unlike previous works (An-339 derson et al., 2018; Chen et al., 2020; Li et al.,340 2020; Yu et al., 2021; Zhang et al., 2021; Kim341 et al., 2021b; Li et al., 2021a) that treat the VQA342 task as a discriminative problem, we let the model343 generate answers freely, which is more aligned344 with…”
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, background): “[106, 20, 129]), that are made more expensive because some of their computational primitives (like non-maximum suppression) are difficult to make efficient on-device.”
MAGMA - Multimodal Augmentation of Generative Models through Adapter-based Finetuning (2021, methodology): “After ﬁnetuning, MAGMA achieves competitive scores across all benchmarks, setting a new state of the art accuracy on OKVQA, as well as attaining strong scores on the NoCaps benchmark - to our knowledge, being surpassed only by SimVLM and VinVL [60], see Table 2.”
Scaling Up Vision-Language Pretraining for Image Captioning (2021, methodology): “We use the pre-trained Faster R-CNN detector from [48] to extract image region features, which are concatenated with scaled bounding boxes as position encoding.”
Florence: A New Foundation Model for Computer Vision (2021, methodology): “For the image encoder, we chose hierarchical Vision Transformers (e.g. , Swin (Liu et al., 2021a), CvT (Wu et al., 2021), Vision Longformer (Zhang et al., 2021a), Focal Transformer (Yang et al., 2021), and CSwin (Dong et al., 2021)).”
ClipCap: CLIP Prefix for Image Captioning (2021, methodology): “[46] also utilize BERT, but require the additional supervision of object tags.”
FILIP: Fine-grained Interactive Language-Image Pre-Training (2021, methodology): “We compare our FILIP model against methods with complex attention layers including Unicoder-VL (Li et al., 2020a), ImageBERT (Qi et al., 2020), UNITER (Chen et al., 2020), VILLA (Gan et al., 2020), ERNIE-ViL (Yu et al., 2021), Oscar (Li et al., 2020b), VinVL (Zhang et al., 2021), ALBEF (Li et al., 2”
VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts (2021, methodology): “Following OSCAR [26] and VinVL [49], we convert the triplet input to two image-text pairs, each containing the text description and one image.”
An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA (2021, methodology): “We represent images either as captions with VinVL (Zhang et al. 2021), or enhance captions with tags predicted by the public Microsoft Azure tagging API2.”
Winner Team Mia at TextVQA Challenge 2021: Vision-and-Language Representation Learning with Pre-trained Sequence-to-Sequence Model (2021, positive): “Object labels and features are extracted from VinVL(Revisiting Visual Representations in VisionLanguage Models) [3], which can generate representations of a richer collection of visual objects and concepts.”
MERLOT: Multimodal Neural Script Knowledge Models (2021, background): “[113], who use an even larger object detector trained on more labeled data, and Kim et al.”
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, methodology): “Unlike neural networks pretrained on multimodal data (e.g., [39,48,57,66,67,82,83]), our method only requires a small amount of multimodal training data and focuses on adapting linguistic knowledge learned from the textual modality.”
Generalized Decoding for Pixel, Image, and Language (2022, methodology): “The field has evolved from a transformer fusion model [10, 46, 96] with pre-extracted object features [1] to end-to-end transformers [21, 36, 43], that directly learn from raw image pixels.”