Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers

By Lisa Anne Hendricks, John F. J. Mellor, R. Schneider, Jean-Baptiste Alayrac, Aida Nematzadeh. 2021

134 citations recorded by the source corpus.

Abstract Recently, multimodal transformer models have gained popularity because their performance on downstream tasks suggests they learn rich visual-linguistic representations. Focusing on zero-shot image retrieval tasks, we study three important factors that can impact the quality of learned representations: pretraining data, the attention mechanism, and loss functions. By pretraining models on six datasets, we observe that dataset noise and language similarity to our downstream task are important indicators of model performance. Through architectural analysis, we learn that models with a multimodal attention mechanism can outperform deeper models with modality-specific attention mechanisms. Finally, we show that successful contrastive losses used in the self-supervised learning literature do not yield similar performance gains when used in multimodal transformers.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Model Architectures

In Optimization & Training

Citations in the corpus

Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”
Generalized Decoding for Pixel, Image, and Language (2022, background): “8 ? ? - ? ? - - ? ZegFormer (B) [30] 3 3 7 7 3 3 7 - - ? 8.”
Multimodal Learning With Transformers: A Survey (2022, background): “Why and how Transformers perform so well in multimodal learning has been investigated [106], [299], [300], [301], [302], [303], [304], [305], [306].”
Cross-Modal Implicit Relation Reasoning and Aligning for Text-to-Image Person Retrieval (2023, methodology): “To achieve full interaction between image and text modalities, We design an efficient multimodal interaction encoder to fuse the image and text embeddings, compared to two other popular multimodal interaction modules [9, 16], our design is more computationally efficient, as illustrated in Fig.”
Separating Skills and Concepts for Novel Visual Question Answering (2021, unknown): “Moreover, we focus on data-efficient methods and do not use prodigious amounts of data external to VQA, like pre-training approaches [9, 41, 54], which is expensive to obtain and can require prior knowledge of the domain and/or concepts in order to perform well [20, 51].”
Zero-Shot Video Question Answering via Frozen Bidirectional Language Models (2022, unknown): “Its success inspired numerous works to adapt it to train multi-modal transformer models on paired visual-linguistic data [15, 24, 25, 29, 34, 44, 54, 57, 62, 60, 65, 58, 56, 69, 70, 90, 91, 96, 97, 99, 104, 106, 118, 121, 126, 127].”
Attention Bottlenecks for Multimodal Fusion (2021, methodology): “[28] Lisa Anne Hendricks, John Mellor, Rosalia Schneider, Jean-Baptiste Alayrac, and Aida Nematzadeh.”
Multimodal biomedical AI (2022, background): “A study from DeepMind, in fact, suggested that curating higher-quality image–text datasets may be more important than generating large single-modality datasets, and other aspects of algorithm development and training 119 .”
ClimaX: A foundation model for weather and climate (2023, unknown): “Transformers have shown favorable scaling properties for language (Kaplan et al., 2020; Hoffmann et al., 2022), vision (Zhai et al., 2022), or even multi-modal tasks (Henighan et al., 2020; Hendricks et al., 2021; Reed et al., 2022b).”