Unified Vision-Language Pre-Training for Image Captioning and VQA

By Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason J. Corso, Jianfeng Gao. 2019

1,067 citations recorded by the source corpus.

This paper presents a unified Vision-Language Pre-training (VLP) model. The model is unified in that (1) it can be fine-tuned for either vision-language generation (e.g., image captioning) or understanding (e.g., visual question answering) tasks, and (2) it uses a shared multi-layer transformer network for both encoding and decoding, which differs from many existing methods where the encoder and decoder are implemented using separate models. The unified VLP model is pre-trained on a large amount of image-text pairs using the unsupervised learning objectives of two tasks: bidirectional and sequence-to-sequence (seq2seq) masked vision-language prediction. The two tasks differ solely in what context the prediction conditions on. This is controlled by utilizing specific self-attention masks for the shared transformer network. To the best of our knowledge, VLP is the first reported model that achieves state-of-the-art results on both vision-language generation and understanding tasks, as disparate as image captioning and visual question answering, across three challenging benchmark datasets: COCO Captions, Flickr30k Captions, and VQA 2.0. The code and the pre-trained models are available at https://github.com/LuoweiZhou/VLP.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Experience Grounds Language (2020, background): “These combined text and vision benchmarks are rich enough to train large-scale, multimodal transformers (Li et al., 2019a; Lu et al., 2019; Zhou et al., 2019) without language pretraining (e.”
TAP: Text-Aware Pre-training for Text-VQA and Text-Caption (2020, background): “For the former, we reﬁne the pre-training tasks in VLP [37, 34] to support the extra scene text input.”
Learning Visual Representations with Caption Annotations (2020, background): “Building on the recent advances in natural language processing [13,54], several works have fine-tuned BERT [13] to fuse visual and textual information [37,50,51,52,64] for VL tasks.”
ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph (2020, methodology): “Most of them ( [5] [6] [7] [8] [9] [20]) use a uniform cross-modal Transformer modelling both image and text representations, while the others like ViLBERT [4] and LXMERT [21] are based on two-stream cross-modal Transformers, which bring more specific representations for image and text.”
Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, background): “…pre-training has brought leaping advances in vision+language understanding tasks such as VQA and VCR, with great potential in extending to visual captioning [84, 70], visual dialog [47, 68], vision-language navigation [18, 43], as well as video-and-language representation learning [62, 61, 41, 31].”
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks (2020, methodology): “[41] uses the object prediction probability as a soft label and concatenate it with its corresponding region features.”
Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, neutral): “As the proactive counterpart, Active Scene Description encompasses sources of feedback that are provided directly in response to active queries by the LLM planner, which are answered either by a person, or by another pretrained model, such as a Visual Question Answering (VQA) model [93, 94, 95, 96].”
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation (2022, methodology): “, 2021) can excel at both types of tasks, whereas a single unified encoder-decoder (Zhou et al., 2020) also limits the model’s capability.”
Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks (2021, methodology): “For visual-linguistic recognition, recent works [14, 36, 37, 46, 53, 62, 65, 78] also adopt Transformers as the backbone, while they usually take regional features as inputs, which are typically extracted by off-the-shelf object detectors ( e.g ., Faster R-CNN [57] pre-trained on Visual Genome…”
Scaling Up Vision-Language Pretraining for Image Captioning (2021, methodology): “To enable text generation with the encoder layers, we use the sequence-tosequence attention mask [49] in each self-attention layer for the captioning module.”
UFO: A UniFied TransfOrmer for Vision-Language Representation Learning (2021, methodology): “The image can be encoded as object features [5,6,13,18,27,29,30,30,32,39,42,47,53, 54] through a Faster RCNN [36], as grid features [20, 22] from a convolutional neural network, or as patch features [25] from a linear projection on raw pixels.”
ClipCap: CLIP Prefix for Image Captioning (2021, methodology): “Most close to ours, are works that employ vision-andlanguage pre-training to create a shared latent space of both vision and text [19,25,35,46,47].”
VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts (2021, background): “The second category models the interaction of images and text using a deep fusion encoder with cross-modal attention [43, 30, 41, 24, 51, 3, 26, 25, 14, 49, 16, 17, 20, 23, 46].”