Inferring the Why in Images

By Hamed Pirsiavash, Carl Vondrick, Antonio Torralba. 2014

43 citations recorded by the source corpus.

Abstract : Humans have the remarkable capability to infer the motivations of other people's actions, likely due to cognitive skills known in psychophysics as the theory of mind. In this paper, we strive to build a computational model that predicts the motivation behind the actions of people from images. To our knowledge, this challenging problem has not yet been extensively explored in computer vision. We present a novel learning based framework that uses high-level visual recognition to infer why people are performing an actions in images. However, the information in an image alone may not be sufficient to automatically solve this task. Since humans can rely on their own experiences to infer motivation, we propose to give computer vision systems access to some of these experiences by using recently developed natural language models to mine knowledge stored in massive amounts of text. While we are still far away from automatically inferring motivation, our results suggest that transferring knowledge from language into vision can help machines understand why a person might be performing an action in an image.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mohit Bansal
2. Yejin Choi
3. Devi Parikh
4. Ali Farhadi
5. Jianfeng Gao
6. Li Fei-Fei
7. Trevor Darrell
8. Abhinav Kumar Gupta

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Trevor Darrell
3. Dacheng Tao
4. Li Fei-Fei
5. Xiaogang Wang
6. Jitendra Malik
7. Andrew Zisserman
8. Yi Yang

In Multimodal Learning

1. Devi Parikh
2. Marcus Rohrbach
3. Li Fei-Fei
4. Mohit Bansal
5. Dhruv Batra
6. Ali Farhadi
7. Trevor Darrell
8. Qi Wu

Citations in the corpus

• Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books (2015, background): “In [27], the authors go beyond describing what is happening in an image and provide explanations about why something is happening.”
• From Recognition to Cognition: Visual Commonsense Reasoning (2018, background): “However, 12As we find in Appendix D, including additional language context tends to boost model performance. the space of commonsense inferences is often limited by the underlying dataset chosen (synthetic [79] or COCO [58] scenes).”
• Visual7W: Grounded Question Answering in Images (2015, background): “The 7W questions roughly correspond to an array of standard vision tasks: what [9, 15, 39], where [24, 50], when [30, 32], who [35, 42], why [33], how [23, 31] and which [16, 17].”
• MovieQA: Understanding Stories in Movies through Question-Answering (2015, background): “These topics have only begun to be explored in the literature [27, 49].”
• Don't just listen, use your imagination: Leveraging visual common sense for non-visual tasks (2015, background): “common sense learnt from non-visual sources) benefits physical reasoning [23, 49], reasoning about intentions [40] and object functionality [50].”
• Asynchronous Temporal Fields for Action Recognition (2016, background): “Moving beyond appearance and reasoning about the state of agents in the images requires understanding human intentions [16, 31].”
• Social LSTM: Human Trajectory Prediction in Crowded Spaces (2016, background): “More approaches can be found in [45] [52] [34] [3] [16] [33].”
• Learning Common Sense through Visual Abstraction (2015, background): “Recently, there has been a surge in interest in high-level “beyond recognition” tasks which can benefit from external knowledge beyond what is depicted in the image [3, 19, 23, 30, 31].”
• Leveraging Visual Question Answering for Image-Caption Ranking (2016, background): “Recently there has been a surge of interest in visual reasoning tasks that require high-level reasoning such as physical reason-4 ing [34,35], future prediction [36,37,38], object affordance prediction [39] and textual tasks that require visual knowledge [40,41,42].”
• Video2Commonsense: Generating Commonsense Descriptions to Enrich Video Captioning (2020, background): “Work for inferring motivations of human actions in static images by incorporating commonsense knowledge are reflected in Pirsiavash et al. (2014); Vondrick et al. (2016).”
• The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoning (2022, background): “3K KB Why In Images [49] 792 KB X BD2BB [48] 3.”