A Corpus for Reasoning about Natural Language Grounded in Photographs

By Alane Suhr, Stephanie Zhou, Iris Zhang, Huajun Bai, Yoav Artzi. 2018

733 citations recorded by the source corpus.

We introduce a new dataset for joint reasoning about natural language and images, with a focus on semantic diversity, compositionality, and visual reasoning challenges. The data contains 107,292 examples of English sentences paired with web photographs. The task is to determine whether a natural language caption is true about a pair of photographs. We crowdsource the data using sets of visually rich images and a compare-and-contrast task to elicit linguistically diverse language. Qualitative analysis shows the data requires compositional joint reasoning, including about quantities, comparisons, and relations. Evaluation using state-of-the-art visual reasoning methods shows the data presents a strong challenge.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

How Can We Accelerate Progress Towards Human-like Linguistic Generalization? (2020, methodology): “…this observation as a carte blanche to ignore sample efﬁciency, then, we should address it experimentally, by collecting multimodal data sets (Suhr et al., 2019; Hudson and Manning, 2019), developing models that learn from them efﬁciently, and using the Generalization Leaderboard to measure…”
Experience Grounds Language (2020, methodology): “…et al., 2012), a myriad of tasks on visual question answering (Antol et al., 2015; Das et al., 2018), natural language and visual reasoning (Suhr et al., 2019b), visual commonsense (Zellers et al., 2019a), and multilingual captioning and translation via video (Wang et al., 2019b) have…”
UNITER: Learning UNiversal Image-TExt Representations (2019, neutral): “…on six V+L tasks across nine datasets, including: ( i ) VQA; ( ii ) Visual Commonsense Reasoning (VCR) (Zellers et al., 2019); ( iii ) NLVR 2 (Suhr et al., 2019); ( iv ) Visual Entailment (Xie et al., 2019); ( v ) Image-Text Retrieval (including zero-shot setting) (Lee et al., 2018); and (…”
Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies (2021, background): “…over multiple parts of their input has attracted substantial attention recently, leading to the creation of many multi-step reasoning Question Answering (QA) benchmarks (Welbl et al., 2018; Talmor and Berant, 2018; Khashabi et al., 2018; Yang et al., 2018; Dua et al., 2019; Suhr et al., 2019).”
Break It Down: A Question Understanding Benchmark (2020, background): “• Visual Question Answering (VQA): Questions over images that require both visual and numerical reasoning skills (Johnson et al., 2017; Suhr et al., 2019).”
VisualBERT: A Simple and Performant Baseline for Vision and Language (2019, background): “(2019)), (3) natural language for visual reasoning (NLVR 2 , Suhr et al. (2019)), and (4) region-to-phrase grounding (Flickr30K, Plummer et al. (2015)).”
A Multimodal Framework for the Detection of Hateful Memes (2020, background): “Multimodal representation learning has recently gained traction due to the poor performance of existing (unimodal) models on multimodal tasks such as Visual Question Answering [1, 12] and Visual Reasoning [33].”
VirTex: Learning Visual Representations from Textual Annotations (2020, background): “Vision-and-Language Pretraining attempts to learn joint representations of image-text paired data that can be transferred to multimodal downstream tasks such as visual question answering [13, 14, 59, 60], visual reasoning [61, 62], referring expressions [63], and language-based image retrieval [35].”
Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, background): “For NLVR(2) [13], given a pair of images and a natural language description, the model judges the correctness of the description based on the visual clues in the image pair.”
The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, background): “[56] for a survey), with great progress on important problems such as visual question answering [3, 26] and image caption generation and retrieval [8, 88, 43, 68, 28], with offshoot tasks focusing specifically on visual reasoning [36], referring expressions [39], visual storytelling [57, 32], visual”
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks (2020, background): “GQA As shown in Table 9, compared with existing VLP works (LXMERT [37] and 12-in-1 [24]), our OscarB results on GQA gain 0.”
LXMERT: Learning Cross-Modality Encoder Representations from Transformers (2019, methodology): “…of our pre-trained model, we ﬁne-tune LXMERT on a challenging visual reasoning task, Natural Language for Visual Reasoning for Real (NLVR 2 ) (Suhr et al., 2019), where we do not use the natural images in their dataset for our pre-training, but ﬁne-tune and evaluate on these challenging,…”
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation (2022, methodology): “NLVR2.”
UFO: A UniFied TransfOrmer for Vision-Language Representation Learning (2021, background): “The task’s input is a pair of images and a natural description, and the goal [40] is to predict whether the description is true about the image pair.”
Achieving Human Parity on Visual Question Answering (2021, background): “Inspired by the breakthrough of language pre-training in NLP, the research community begins to pay more attention to vision-language pre-training on large-scale image-text pairing data, which achieves state-of-the-art performance across a variety of vision-language (VL) tasks [4, 30, 73].”
VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts (2021, methodology): “Natural Language for Visual Reasoning (NLVR2) The NLVR2 [42] dataset requires the model to predict whether a text description is true about a pair of images.”
SimVLM: Simple Visual Language Model Pretraining with Weak Supervision (2021, background): “In addition, SimVLM also consistently outperforms prior methods on NLVR2 and SNLI-VE, illustrating its capability of processing more complex visual-linguistic reasoning.”
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation (2021, background): “NLVR2 [19], VQA [20]), but most of them require high-resolution input images and pre-trained object detectors.”
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts (2021, background): “• NLVR2 [35] An instance of text comes from the caption associated with each pair of images of the training split.”