Unifying Vision-and-Language Tasks via Text Generation

By Jaemin Cho, Jie Lei, Hao Hao Tan, Mohit Bansal. 2021

635 citations recorded by the source corpus.

Existing methods for vision-and-language learning typically require designing task-specific architectures and objectives for each task. For example, a multi-label answer classifier for visual question answering, a region scorer for referring expression comprehension, and a language decoder for image captioning, etc. To alleviate these hassles, in this work, we propose a unified framework that learns different tasks in a single architecture with the same language modeling objective, i.e., multimodal conditional text generation, where our models learn to generate labels in text based on the visual and textual inputs. On 7 popular vision-and-language benchmarks, including visual question answering, referring expression comprehension, visual commonsense reasoning, most of which have been previously modeled as discriminative tasks, our generative approach (with a single unified architecture) reaches comparable performance to recent task-specific state-of-the-art vision-and-language models. Moreover, our generative approach shows better generalization ability on questions that have rare answers. Also, we show that our framework allows multi-task learning in a single architecture with a single set of parameters, achieving similar performance to separately optimized single-task models. Our code is publicly available at: https://github.com/j-min/VL-T5

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Flamingo: a Visual Language Model for Few-Shot Learning (2022, methodology): “We show in row (ii) the importance of our gradient accumulation strategy compared to using round-robin updates [17].”
Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation (2022, methodology): “We speculate that the weakened NLU ability 485 of these models is caused by the catastrophic for- 486 getting of the pre-trained BERT weights during the 487 multimodal pre-training.”
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation (2022, methodology): “, 2021) nor encoder-decoder models (Cho et al., 2021; Wang et al., 2021) can excel at both types of tasks, whereas a single unified encoder-decoder (Zhou et al.”
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, background): “2Recent papers propose exceptions, like generating masked-out spans [21] or text [77, 116], but it is unclear whether they can outperform the ‘VisualBert’s on vision-language tasks like VCR [124].”
VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks (2021, methodology): “We extract the 7 × 7 grid features produced by the last convolutional layer, and then apply adaptive maximum-pooling over the features for down-sampling then to 6 × 6 for a fair comparison to [7].”
Scaling Up Vision-Language Pretraining for Image Captioning (2021, methodology): “Since the birth of ViLBERT [30] and LXMERT [40], we have witnessed a boom of methods for vision-language pre-training [6,7,13,22,26, 37, 45, 46].”
UFO: A UniFied TransfOrmer for Vision-Language Representation Learning (2021, methodology): “In [7, 23], a transformer encoder and decoder are applied on the visual features and text prefix.”
SimVLM: Simple Visual Language Model Pretraining with Weak Supervision (2021, methodology): “…detection for image region feature regression or tagging as part of the pre-training objectives (Tan & Bansal, 2019; Su et al., 2020; Li et al., 2019; Chen et al., 2020b; Gan et al., 2020; Li et al., 2020; Yu et al., 2021; Li et al., 2021; Zhang et al., 2021; Hu et al., 2021; Cho et al., 2021).”
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models (2023, neutral): “…(Radford et al., 2021; Jia et al., 2021), the fusion-encoder architecture (Tan & Bansal, 2019; Li et al., 2021), the encoder-decoder architecture (Cho et al., 2021; Wang et al., 2021b; Chen et al., 2022b), and more recently, the unified transformer architecture (Li et al., 2022; Wang et al.,…”
GIT: A Generative Image-to-text Transformer for Vision and Language (2022, methodology): “Since most VL tasks can be formulated as the text generation task (Cho et al., 2021), a single generation model can be pre-trained to support various downstream tasks.”
Generative Negative Text Replay for Continual Vision-Language Pretraining (2022, background): “However, they typically use an external object detector to generate visual region descriptions as the input to the multi-modal encoder [11,10], which can be computationally expensive.”
CoCa: Contrastive Captioners are Image-Text Foundation Models (2022, methodology): “, 2021b), VL-T5 (Cho et al., 2021)) in this direction relies on pretrained object detection modules such as Fast(er) R-CNN (Ren et al.”
OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework (2022, methodology): “Following this idea, [56] and [57] demonstrate text-generation-based multimodal pretrained models.”
PaLI: A Jointly-Scaled Multilingual Language-Image Model (2022, methodology): “In Cho et al. (2021), the vision-language model is trained to recover masked text.”