Are we pretraining it right? Digging deeper into visio-linguistic pretraining

By Amanpreet Singh, Vedanuj Goswami, Devi Parikh. 2020

48 citations recorded by the source corpus.

Numerous recent works have proposed pretraining generic visio-linguistic representations and then finetuning them for downstream vision and language tasks. While architecture and objective function design choices have received attention, the choice of pretraining datasets has received little attention. In this work, we question some of the default choices made in literature. For instance, we systematically study how varying similarity between the pretraining dataset domain (textual and visual) and the downstream domain affects performance. Surprisingly, we show that automatically generated data in a domain closer to the downstream task (e.g., VQA v2) is a better choice for pretraining than "natural" data but of a slightly different domain (e.g., Conceptual Captions). On the other hand, some seemingly reasonable choices of pretraining datasets were found to be entirely ineffective for some downstream tasks. This suggests that despite the numerous recent efforts, vision & language pretraining does not quite work "out of the box" yet. Overall, as a by-product of our study, we find that simple design choices in pretraining can help us achieve close to state-of-art results on downstream tasks without any architectural changes.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, neutral): “Interestingly, we can see that the difference between unimodally pretrained models and multimodally pretrained models pretraining is relatively small, corroborating findings recently reported in [70] and indicating that multimodal pretraining can probably be improved further.”
Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers (2021, background): “Recent analysis work (Singh et al., 2020; Cao et al., 2020) has shed light on different aspects of multimodal transformer models.”
InfographicVQA (2021, methodology): “This is inline with observation by Singh et al . that pretraining on data similar to the data for downstream task is highly beneﬁcial in visio-linguistic pretraining [44].”
Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality (2022, background): “Despite the impressive performance of pretrained vision and language transformers on a wide variety of multimodal tasks [47,51,56], they remain poorly understood [8,19,46, 67].”
Multi-Modal Open-Domain Dialogue (2020, methodology): “Faster R-CNN Finally, we consider Faster RCNN features (Ren et al., 2017), using models trained in the Detectron framework (Girshick et al., 2018); specifically, we use a ResNeXt-152 backbone trained on the Visual Genome dataset (Krishna et al., 2016) with the attribute head (Singh et al., 2020a) 5.”
KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQA (2020, background): “The first two are Masked COCO and Masked VQA, introduced in [70].”
Natural Language Rationales with Full-Stack Visual Reasoning: From Pixels to Semantic Frames to Commonsense Graphs (2020, background): “A similar point was recently raised by Singh et al. (2020).”
Separating Skills and Concepts for Novel Visual Question Answering (2021, unknown): “Moreover, we focus on data-efﬁcient methods and do not use prodigious amounts of data external to VQA, like pre-training approaches [9, 41, 54], which is expensive to obtain and can require prior knowledge of the domain and/or concepts in order to perform well [20, 51].”