Discovering Causal Signals in Images

By David Lopez-Paz, Robert Nishihara, Soumith Chintala, Bernhard Schölkopf, L. Eon Bottou. 2016

256 citations recorded by the source corpus.

This paper establishes the existence of observable footprints that reveal the causal dispositions of the object categories appearing in collections of images. We achieve this goal in two steps. First, we take a learning approach to observational causal discovery, and build a classifier that achieves state-of-the-art performance on finding the causal direction between pairs of random variables, given samples from their joint distribution. Second, we use our causal direction classifier to effectively distinguish between features of objects and features of their contexts in collections of static images. Our experiments demonstrate the existence of a relation between the direction of causality and the difference between objects and their contexts, and by the same token, the existence of observable signals that reveal the causal dispositions of objects.

Subfields: Computer Vision and Pattern Recognition, Causal Inference & Discovery

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Causal Inference & Discovery

Citations in the corpus

Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, positive): “It was also shown that the use of counterfactual explanations can help the user to understand the decision of a model [40,42,351] .”
Toward Causal Representation Learning (2021, background): “However, real-world observations are usually not structured into those units, to begin with, for example, objects in images [162].”
Causality for Machine Learning (2019, background): “Leon, one of the original leaders of the field of deep learning, has since taken a strong interest in causality (Lopez-Paz et al., 2017). other.”
Invariant Risk Minimization (2019, background): “This suggests that invariant descriptions of objects relate to the causal explanation of the object itself (“Why is it a cow?”) [32].”
Conditional variance penalties and domain shift robustness (2017, background): “the problems of causeeffect inference and generative adversarial networks (Chalupka et al., 2014; Lopez-Paz et al., 2017; Lopez-Paz and Oquab, 2017; Goudet et al., 2017; Bahadori et al., 2017; Besserve et al., 2018; Kocaoglu et al., 2018).”
Set Transformer (2018, background): “, 2015), discovering causality (Lopez-Paz et al., 2017), learning the statistics of a set (Edwards & Storkey, 2017), few-shot image classification (Snell et al.”
Continual Normalization: Rethinking Batch Normalization for Online Continual Learning (2022, background): “Such properties are crucial for continual learning, especially in the online setting (Lopez-Paz et al., 2017; Riemer et al., 2019), but not satisfied by existing normalization layers.”
CausalGAN: Learning Causal Implicit Generative Models with Adversarial Training (2017, background): “In [25], the authors propose using a neural network in order to discover the causal relation between image class labels based on static images.”
Interventional Few-Shot Learning (2020, background): “[67] proposes a retrospective for image captioning and other applications include image classification [10, 36], imitation learning [13], long-tailed recognition [56] and semantic segmentation [73].”
COPHY: Counterfactual Learning of Physical Dynamics (2019, background): “In the ML community, causal reasoning gained mainstream attention relatively recently (Lopez-Paz et al., 2017; Lopez-Paz & Oquab, 2017; Kocaoglu et al., 2018; Rojas-Carulla et al., 2018; Mooij et al., 2016; Sch¨olkopf et al., 2012), due to limitations of statistical learning becoming increasingly…”
MUREL: Multimodal Relational Reasoning for Visual Question Answering (2019, background): “Recently, the deep learning community started to tackle complex visual reasoning problems such as relationship detection [26], object recognition [9], abstract reasoning [33] or visual causality [25], while more theoretical work attempt to formalize relational reasoning [7].”
Adaptively Connected Neural Networks (2019, background): “This phenomenon has also been observed experimentally by [58] and theoretically by [28], confirming that the over-globalization is a shortcoming of NLN.”
Visual Commonsense R-CNN (2020, methodology): “To this end, we apply NCC [40] to remove possible colliders from Z .”
iPerceive: Applying Common-Sense Reasoning to Multi-Modal Dense Video Captioning and Video Question Answering (2020, background): “The union of these ﬁelds has been explored in several contexts, including image classiﬁcation [34, 4], reinforcement learning [39], adversarial learning [26], visual dialog [44], image captioning [65] and scene/knowledge graph generation [51, 40].”
Automated Machine Learning: Methods, Systems, Challenges (2019, unknown): “[30]: whenever Pcause,effect changed it may often be the case that only Pcause or Peffect|cause changed.”
Counterfactual Attention Learning for Fine-Grained Visual Categorization and Re-identification (2021, unknown): “Some efforts also used causality as an effective tool to alleviate the effects of dataset bias in vision tasks, including image classification [33], scene graph gen-”