Don't just listen, use your imagination: Leveraging visual common sense for non-visual tasks

By Xiaoyu Lin, Devi Parikh. 2015

68 citations recorded by the source corpus.

Artificial agents today can answer factual questions. But they fall short on questions that require common sense reasoning. Perhaps this is because most existing common sense databases rely on text to learn and represent knowledge. But much of common sense knowledge is unwritten - partly because it tends not to be interesting enough to talk about, and partly because some common sense is unnatural to articulate in text. While unwritten, it is not unseen. In this paper we leverage semantic common sense knowledge learned from images - i.e. visual common sense - in two textual tasks: fill-in-the-blank and visual paraphrasing. We propose to “imagine” the scene behind the text, and leverage visual cues from the “imagined” scenes in addition to textual cues while answering these questions. We imagine the scenes as a visual abstraction. Our approach outperforms a strong text-only baseline on these tasks. Our proposed tasks can serve as benchmarks to quantitatively evaluate progress in solving tasks that go “beyond recognition”. Our code and datasets are publicly available.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books (2015, background): “artly due to the creation of CoCo [18], Microsoft’s large-scale captioned image dataset. The ﬁeld has tackled a diverse set of tasks such as captioning [13,11,36,35,21], alignment [11,15,34], Q&A [20,19], visual model learning from textual descriptions [8,26], and semantic visual search with nat”
VQA: Visual Question Answering (2015, methodology): “In this paper, we present both an open-ended answering task and a multiple-choice task [38, 27].”
Visual Madlibs: Fill in the blank Image Generation and Question Answering (2015, methodology): “Lin et al. [22] take an interesting multi-modal approach to question-answering.”
Image Captioning and Visual Question Answering Based on Attributes and Their Related External Knowledge (2016, background): “Using visually-sourced information is a promising approach to solve this problem [65], [66], but has a way to go before it might be usefully applied within our approach.”
Building a Large-scale Multimodal Knowledge Base for Visual Question Answering (2015, background): “While Geman et al. [9] and Malinowski and Fritz [23] focus on proposing a visual QA test for object detection problems, Lin and Parikh [22] have looked into richer reasoning of image descriptions using cartoon data.”
Visual question answering: A survey of methods and datasets (2016, methodology): “This type of images has been used before for capturing common sense [20, 46, 75], learning models of interactions between people [4], generating scenes from natural language descriptions [103], and learning the semantic relevance of visual features [102, 104].”
Video Question Answering with Spatio-Temporal Reasoning (2019, methodology): “Somedatasets contains natural images (e.g., Malinowski and Fritz (2014); Antol et al. (2015);Yu et al. (2015)),while others are based on synthetic images [e.g., Lin and Parikh (2015); Johnson et al. (2017)].”
Leveraging Visual Question Answering for Image-Caption Ranking (2016, background): “Recently there has been a surge of interest in visual reasoning tasks that require high-level reasoning such as physical reason-4 ing [34,35], future prediction [36,37,38], object affordance prediction [39] and textual tasks that require visual knowledge [40,41,42].”
Generative Models of Visually Grounded Imagination (2017, background): “(Lin and Parikh, 2015) imagine abstract clipart scenes underlying text, and show improvements on the tasks of visual paraphrasing (determining if two sentences correspond to the same image) and fill-in-the-blanks.”
Learning Common Sense through Visual Abstraction (2015, background): “Lin and Parikh [26] “imagine” abstract scenes corresponding to text, and use the common sense depicted in these imagined scenes to solve textual tasks such as ﬁll-in-the-blanks and paraphrasing.”
Seeing Voices and Hearing Faces: Cross-Modal Biometric Matching (2018, background): “In particular, works have demonstrated coupling between an individual’s idiosyncratic speaking style, the sound of their voice and the manner in which their face moves [13, 27, 53], suggesting the presence of dynamic information which can be exploited to solve the matching task.”
Visual Madlibs: Fill in the Blank Description Generation and Question Answering (2015, methodology): “[24] take an interesting multimodal approach to question-answering.”
Automatic Understanding of Image and Video Advertisements (2017, unknown): “[41] use the machine’s “imagination” to answer questions about images.”
Visual Commonsense R-CNN (2020, background): “However, despite the major challenge in trading off between annotation cost and noisy multimodal pairs, common sense is not always recorded in text due to the reporting bias [66, 37], e.”
Leveraging Visual Knowledge in Language Tasks: An Empirical Study on Intermediate Pre-training for Cross-Modal Knowledge Transfer (2022, background): “(VP) (Lin and Parikh, 2015) that needs scene understanding.”
Uncovering the Temporal Context for Video Question Answering (2017, methodology): “Lin and Parikh (2015) introduced an interesting multiple-choice fill-in-the-blank question answering task on abstract scenes, and Yu et al. (2015) applied the task to natural images using various question templates.”
Building a Large-scale Multimodal Knowledge Base System for Answering Visual Queries (2015, background): “Learning with Vision and Language Previous work on joint learning with vision and language abounds [23, 30, 41, 42, 60].”