Learning Common Sense through Visual Abstraction

By Ramakrishna Vedantam, Xiaoyu Lin, Tanmay Batra, C. Lawrence Zitnick, Devi Parikh. 2015

92 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

VQA: Visual Question Answering (2015, background): “[40] and [44] use visual information to assess the plausibility of common sense assertions.”
MovieQA: Understanding Stories in Movies through Question-Answering (2015, background): “An interesting direction, particularly for the goals of our paper, is also the task of learning common sense knowledge from captioned images [40].”
Yin and Yang: Balancing and Answering Binary Visual Questions (2015, background): “[34, 38] reason about the plausibility of commonsense assertions like (men, ride, elephants) by gathering visual evidence for them in real images [34] and abstract scenes [38].”
Neural Scene De-rendering (2017, background): “Abstract scenes have been an emerging research topic in computer vision [39], natural language processing [24], and reasoning [31].”
Obj2Text: Generating Visually Descriptive Language from Object Layouts (2017, unknown): “The works of Vedantam et al. (2015b); Eysenbach et al. (2016) proposed methods to learn common-sense knowledge from clipart scenes, while the method of Yatskar et al. (2016), similar to our work, leverages object annotations for natural images.”
Text2Scene: Generating Compositional Scenes From Textual Descriptions (2018, result): “Our work is also related to prior research on using abstract scenes to mirror and analyze complex situations in the real world [40, 41, 7, 30].”
Visual Commonsense in Pretrained Unimodal and Multimodal Models (2022, background): “…fill-in-the-blank problem for both image and text.100 2.2 Visual Commonsense101 In one of the early attempts at learning visual com-102 monsense, Vedantam et al. (2015) measure the plau-103 sibility of a commonsense assertion in the form of104 (obj1, relation, obj2) based on how similar it…”
Leveraging Visual Question Answering for Image-Caption Ranking (2016, background): “[41,42] evaluate the plausibility of commonsense assertions by verifying them on collections of abstract scenes and real images, respectively, to leverage the visual common sense in those collections.”
A Diagram is Worth a Dozen Images (2016, background): “The domain of abstract images has also received a considerable amount of interest over the past couple of years [8,9,10,11].”
What Is More Likely to Happen Next? Video-and-Language Future Event Prediction (2020, background): “Recently, commonsense reasoning has emerged as an important topic in both language (Zellers et al., 2018, 2019b; Sap et al., 2019) and vision (Vedantam et al., 2015b; Zellers et al., 2019a; Zadeh et al., 2019; Fang et al., 2020) communities.”
Human-Centered Artificial Intelligence and Machine Learning (2019, background): “There are a number of sources from which intelligent systems might acquire common knowledge, including machine vision applied to cartoons (Vedantam et al., 2015), images (Sadeghi et al.”
Measuring Machine Intelligence Through Visual Question Answering (2016, background): “The abstract scenes may be useful for a variety of other tasks as well, such as learning common sense knowledge [35, 2, 7, 10, 31].”
Visual question answering: A survey of methods and datasets (2016, methodology): “This type of images has been used before for capturing common sense [20, 46, 75], learning models of interactions between people [4], generating scenes from natural language descriptions [103], and learning the semantic relevance of visual features [102, 104].”
Visual Commonsense R-CNN (2020, methodology): “However, despite the major challenge in trading off between annotation cost and noisy multimodal pairs, common sense is not always recorded in text due to the reporting bias [65, 37], e.g ., most may say “people walking on road” but few will point out “people walking with legs”.”
iPerceive: Applying Common-Sense Reasoning to Multi-Modal Dense Video Captioning and Video Question Answering (2020, background): “ptic segmentation [21], etc. 2. Related Work 2.1. Visual Common-Sense Current research in the ﬁeld of building a common-sense knowledge base mainly falls into two categories: (i) learning from images [61, 55, 66] and (ii) learning actions from videos [9]. While the former limits learning to humanann”
Video2Commonsense: Generating Commonsense Descriptions to Enrich Video Captioning (2020, unknown): “Commonsense caption generation has been approached on abstract scenes and clipart images in [30].”
Machine Common Sense Concept Paper (2018, background): “6 Approved for Public Release, Distribution Unlimited These researchers are discovering new techniques for extracting commonsense knowledge from language [42][43][44][45], vision [46][47][48][49], and robotics [50][51][52].”
The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoning (2022, background): “VCR [75] 110K QA X X VisualCOMET [44] 59K If/Then KB X X Visual7W [80] 47K QA X partial Visual Madlibs [72] 11K FiTB X partial X Abstract Scenes [65] 4.”