12-in-1: Multi-Task Vision and Language Representation Learning

By Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, Stefan Lee. 2019

509 citations recorded by the source corpus.

Much of vision-and-language research focuses on a small but diverse set of independent tasks and supporting datasets often studied in isolation; however, the visually-grounded language understanding skills required for success at these tasks overlap significantly. In this work, we investigate these relationships between vision-and-language tasks by developing a large-scale, multi-task model. Our approach culminates in a single model on 12 datasets from four broad categories of task including visual question answering, caption-based image retrieval, grounding referring expressions, and multimodal verification. Compared to independently trained single-task models, this represents a reduction from approximately 3 billion parameters to 270 million while simultaneously improving performance by 2.05 points on average across tasks. We use our multi-task framework to perform in-depth analysis of the effect of joint training diverse tasks. Further, we show that finetuning task-specific models from our single multi-task model can lead to further improvements, achieving performance at or above the state-of-the-art.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

UNITER: Learning UNiversal Image-TExt Representations (2019, methodology): “Compared with LXMERT (Tan & Bansal, 2019) and ViLBERT (Lu et al., 2019) that use two streams (one Transformer for each modality), our UNITER model can learn joint contextualized representations for image regions and textual words through a single Transformer.”
TAP: Text-Aware Pre-training for Text-VQA and Text-Caption (2020, background): “For the former, we reﬁne the pre-training tasks in VLP [37, 34] to support the extra scene text input.”
ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph (2020, methodology): “As illustrated in Table 3, ERINE-VIL-large acheives better downstream task performances on 5 tasks compared to UNITER, 12-in-1 [38] , OSCAR [39] and VILLA [40].”
Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, background): “More recent studies leverage multi-task learning [39] to enhance finetuning and use detected image tags [35] to further enhance pre-training.”
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks (2020, neutral): “GQA As shown in Table 9, compared with existing VLP works (LXMERT [37] and 12-in-1 [24]), our OscarB results on GQA gain 0.”
Scaling Up Vision-Language Pretraining for Image Captioning (2021, background): “Along the journey of VLP, researchers have investigated different training strategies [11,31], robustness [25], compression [9, 10, 42], probing analysis [3, 27], and the extension to video-text modeling [21, 24, 38, 39, 50].”
LiT: Zero-Shot Transfer with Locked-image text Tuning (2021, methodology): “The models have been applied to diverse downstream tasks, including visual question answering [23], visual commonsense reasoning [67] and image captioning [40, 41, 55].”
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation (2021, background): “The first category focuses on modelling the interactions between image and text features with transformerbased multimodal encoders [10, 11, 12, 13, 1, 14, 15, 2, 3, 16, 8, 17, 18].”
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts (2021, methodology): “Based directly upon BERT, V+L pre-training research has largely been focused on V+L understanding [52, 46, 21, 72, 3, 69, 45, 53], with classiﬁcation or regression tasks that do not involve generation.”
Unifying Vision-and-Language Tasks via Text Generation (2021, methodology): “Task-specific heads: We also experiment with the multi-task finetuning setup of ViLBERTMT (Lu et al., 2020), where a task-specific head is finetuned for each of the 7 downstream tasks while sharing backbone.”
Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “Several recent methods [5, 15, 18, 32, 45, 47, 49, 53, 54, 70, 73, 98] pretrain multi-modal vision-language representations, such as transformers, using datasets with image captions, e.”
Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training (2020, methodology): “Pioneering works such as ViLBERT (Lu et al., 2019a) and LXMERT (Tan and Bansal, 2019) propose to encode image and text modalities by two separate Transformers, with a third Transformer for later multimodal fusion.”
ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision (2021, methodology): “The pretrain-and-finetune scheme has been expanded to a joint domain of vision and language, giving birth to the category of Vision-and-Language Pretraining (VLP) models (Lu et al., 2019; Chen et al., 2019; Su et al., 2019; Li et al., 2019; Tan & Bansal, 2019; Li et al., 2020a; Lu et al., 2020; Qi e”
MDETR - Modulated Detection for End-to-End Multi-Modal Understanding (2021, methodology): “These methods can be divided into single stream [6, 24, 65, 22] and two-stream [47, 28, 29, 46] architectures depending on whether the text and images are processed by a single combined transformer or two separate transformers followed by some cross attention layers.”
M6: A Chinese Multimodal Pretrainer (2021, methodology): “A number of studies [4, 11, 17, 22, 24, 25, 28, 29, 38, 51] have created new state-of-the-art performances for various cross-modal downstream tasks.”