A Corpus of Natural Language for Visual Reasoning

By Alane Suhr, Michael Lewis, James Yeh, Yoav Artzi. 2017

277 citations recorded by the source corpus.

We present a new visual reasoning language dataset, containing 92,244 pairs of examples of natural statements grounded in synthetic images with 3,962 unique sentences. We describe a method of crowdsourcing linguistically-diverse data, and present an analysis of our data. The data demonstrates a broad set of linguistic phenomena, requiring visual and set-theoretic reasoning. We experiment with various models, and show the data presents a strong challenge for future research.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Towards VQA Models That Can Read (2019, methodology): “To study visual reasoning independent of language, non-photo-realistic VQA datasets have been introduced such as CLEVR [18], NLVR [42] and FigureQA [21].”
Learning with Latent Language (2017, neutral): “These kinds of reasoning problems have been well-studied in visual question answering settings (Johnson et al., 2017; Suhr et al., 2017).”
A Corpus for Reasoning about Natural Language Grounded in Photographs (2018, methodology): “Our approach is inspired by the collection of NLVR, where workers were shown a set of similar images and asked to write a sentence true for some images, but false for the others (Suhr et al., 2017).”
TabFact: A Large-scale Dataset for Table-based Fact Verification (2019, methodology): “, 2018) and multimodal language reasoning (NLVR/NLVR2) (Suhr et al., 2017; 2019).”
TOUCHDOWN: Natural Language Navigation and Spatial Reasoning in Visual Street Environments (2018, background): “The panoramas are connected in a graph-like structure with undirected edges connecting neighboring panoramas.”
Learning What Makes a Difference from Counterfactual Examples and Gradient Supervision (2020, background): “Most interestingly, we show that they can be also be generated from existing annotations [18,25,36,61,62,70].”
MIMIC-IT: Multi-Modal In-Context Instruction Tuning (2023, methodology): “The notion of instruction tuning in multi-modal models was initially introduced in the work called Multi-Instruct [50], which encompassed a wide range of multi-modal tasks [18, 56, 41, 27, 12] involving visual understanding and multi-modal reasoning, such as Visual Question Answering [18, 56, 23].”
Obtaining Faithful Interpretations from Compositional Neural Networks (2020, background): “or a table) have been recently gaining increased attention, leading to the creation of multiple datasets that require reasoning in both the visual and textual domain (Johnson et al., 2016; Suhr et al., 2017; Talmor and Berant, 2018; Yang et al., 2018a; Suhr et al., 2019; Hudson and Manning, 2019; Du”
NLVR2 Visual Bias Analysis (2019, background): “NLVR (Suhr et al., 2017) and NLVR2 Suhr et al. (2019) are two recently proposed benchmarks for visual reasoning with natural language.”
When and why vision-language models behave like bags-of-words, and what to do about it? (2022, methodology): “Other VQA datasets such as NVLR (Suhr et al., 2017; 2018) cover a broad range of linguistic phenoma including compositionality.”
Challenges and Prospects in Vision and Language Research (2019, unknown): “Similarly, NLVR is difﬁcult, but it lacks additional annotations to aid in the measurement of why a model fails, or eventually, why it succeeds.”
Robots That Use Language (2020, background): “Cornell NLVR (189) Pairs of images and logical statements about them that are true or false http://lic.”
PlotQA: Reasoning over Scientific Plots (2019, methodology): “On the other hand, datasets such as CLEVR [11] and NVLR [35] contain complex reasoning based questions on synthetic images having 2D and 3D geometric objects.”
FindIt: Generalized Localization with Natural Language Queries (2022, background): “Large amounts of work have been done to learn across image and language, including visualgrounding [16,81,49,96,91,62], image captioning [6,1,5], visual question and answering (VQA) [33,12], visual dialog [11,39], visual reasoning [75,93,85], text-toimage and image-to-text retrieval [32,66,74], and ”
CHALET: Cornell House Agent Learning Environment (2018, unknown): “Simulators have also been used to evaluate natural language instruction following [14, 3, 13, 9, 8] and question answering [7, 6, 10, 19].”
Evaluating Text-to-Visual Generation with Image-to-Text Generation (2024, methodology): “While visual programming achieves decent performance on classic benchmarks like GQA [28] and NLVR [71], we find that they rely heavily on hand-crafted in-context prompts (e.g., exemplar programs) and struggle on more challenging compositional tasks like Winoground [75].”
Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, background): “Video-language inference (VLI) is proposed by Liu et al.[238] in year 2020, which aims at understanding the video and text multi-modal data.”
Intrinsic Evaluation of Summarization Datasets (2020, background): “Nonethe-less, several recent natural language understanding datasets (Bowman et al., 2015; Rajpurkar et al., 2016; Suhr et al., 2017) institute explicit quality-control procedures in crowd-sourcing dataset construction (Zaidan and Callison-Burch, 2011; Yan et al., 2014; Callison-Burch et al.,…”
IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning (2021, background): “Unlike natural images (VQA [3], CLEVR [21]) or abstract diagrams like scenes, charts, plots, and illustrations (VQA-Abstract [3], DVQA [22], NLVR [48], AI2D [26], Geometry3K [37]), IconQA features icon images and covers the largest object set of 388 classes.”
VLP: A Survey on Vision-language Pre-training (2022, background): “Natural Language for Visual Reasoning (NLVR) [72, 73] .”