Visual Causal Feature Learning

By Krzysztof Chalupka, Pietro Perona, Frederick Eberhardt. 2014

160 citations recorded by the source corpus.

We provide a rigorous definition of the visual cause of a behavior that is broadly applicable to the visually driven behavior in humans, animals, neurons, robots and other perceiving systems. Our framework generalizes standard accounts of causal learning to settings in which the causal variables need to be constructed from micro-variables. We prove the Causal Coarsening Theorem, which allows us to gain causal knowledge from observational data with minimal experimental effort. The theorem provides a connection to standard inference techniques in machine learning that identify features of an image that correlate with, but may not cause, the target behavior. Finally, we propose an active learning scheme to learn a manipulator function that performs optimal manipulations on the image to automatically identify the visual cause of a target behavior. We illustrate our inference and learning algorithms in experiments based on both synthetic and real data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Explaining and Harnessing Adversarial Examples (2014, background): “Indeed, Gu & Rigazio (2014) and Chalupka et al. (2014) have already begun the ﬁrst steps toward designing models that resist adversarial perturbation, though no model has yet succesfully done so while maintaining state of the art accuracy on clean inputs.”
Analysis of classifiers’ robustness to adversarial perturbations (2015, background): “Following the original paper of Szegedy et al. (2014), several attempts have been made to make deep networks robust to adversarial perturbations Chalupka et al. (2014); Gu & Rigazio (2014), and a related phenomenon has been explored in Nguyen et al. (2014).”
Discovering Causal Signals in Images (2016, background): “[2] is closer to our work because it addresses causation issues in images.”
On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation (2015, background): “For a treatment on how to discriminate structures which influence predictions from correlations with hidden variables such that the hidden variables have an impact on the predictor, see [26].”
A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms (2019, background): “, 2019), and studied in the causal setting (Chalupka et al., 2015; 2017) and domain adaptation (Magliacane et al.”
Conditional variance penalties and domain shift robustness (2017, background): “the problems of causeeffect inference and generative adversarial networks (Chalupka et al., 2014; Lopez-Paz et al., 2017; Lopez-Paz and Oquab, 2017; Goudet et al., 2017; Bahadori et al., 2017; Besserve et al., 2018; Kocaoglu et al., 2018).”
Multi-Level Cause-Effect Systems (2015, methodology): “Chalupka et al. (2015) provide a method to discover from image pixels the macro-level visual cause of a pre-defined macro-level “target behavior”.”
Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learning (2019, methodology): “Since many semantic concepts may map to the same set of pixels, it may be difﬁcult to identify the functional relationship between changes in pixels and changes in network output according to semantic concepts or game state (Chalupka et al., 2015).”
Distilling Causal Effect of Data in Class-Incremental Learning (2021, methodology): “It has been recently introduced to various computer vision fields, including feature learning [3, 46], few-shot classification [49], long-tailed recognition [40], semantic segmentation [8] and other high-level vision tasks[41, 32, 48, 27].”
Measuring Neural Net Robustness with Constraints (2016, background): “In practice, X = Xtest is taken to be the test set.”
Interventional Few-Shot Learning (2020, background): “[67] proposes a retrospective for image captioning and other applications include image classification [10, 36], imitation learning [13], long-tailed recognition [56] and semantic segmentation [73].”
Causal feature learning: an overview (2017, unknown): “Causal feature learning (CFL) is an unsupervised machine learning and causal inference framework with two goals: (1) the formation of high-level causal hypotheses using low-level input data, and (2) efficient testing of these hypotheses (Chalupka et al. 2015, 2016a, b).”
Fundamental limits on adversarial robustness (2015, background): “Following the original paper (Szegedy et al., 2014), several attempts have been made to make deep networks robust to adverarial perturbations (Chalupka et al., 2014; Gu & Rigazio, 2014).”
Causal Induction from Visual Observations for Goal Directed Tasks (2019, unknown): “Following Chalupka et al. (2014), we assume that cause and effect in our problems can be deﬁned on a handful of causal macro-variables.”
iPerceive: Applying Common-Sense Reasoning to Multi-Modal Dense Video Captioning and Video Question Answering (2020, background): “The union of these ﬁelds has been explored in several contexts, including image classiﬁcation [34, 4], reinforcement learning [39], adversarial learning [26], visual dialog [44], image captioning [65] and scene/knowledge graph generation [51, 40].”
Representation Learning via Invariant Causal Mechanisms (2020, unknown): “To better understand contrastive learning and motivate this proxy task, we generalize instance discrimination using the causal concept of refinements (Chalupka et al., 2014).”
UvA-DARE ( Digital Academic Repository ) Causal Consistency of Structural Equation Models (2017, unknown): “(b) Micro-level models versus macro-level models in which the macro-variables are aggregate features of the micro-variables [4, 5, 9, 11, 17]; e.”
On the Robustness of Semantic Segmentation Models to Adversarial Attacks (2017, background): “However, these defenses are not universal; they have frequently been found to be vulnerable to other types of attacks [11, 9, 10, 34] and/or come at the cost of performance penalties on clean inputs [12, 31, 50].”
No Subclass Left Behind: Fine-Grained Robustness in Coarse-Grained Classification Problems (2020, background): “Many of the guarantees from statistical learning theory break down in the presence of such non-IID data [7], suggesting that models trained using traditional Empirical Risk Minimization (ERM) are likely to be vulnerable to hidden stratification.”