Counterfactual Generative Networks

By Axel Sauer, Andreas Geiger. 2021

151 citations recorded by the source corpus.

Neural networks are prone to learning shortcuts -- they often model simple correlations, ignoring more complex ones that potentially generalize better. Prior works on image classification show that instead of learning a connection to object shape, deep classifiers tend to exploit spurious correlations with low-level texture or the background for solving the classification task. In this work, we take a step towards more robust and interpretable classifiers that explicitly expose the task's causal structure. Building on current advances in deep generative modeling, we propose to decompose the image generation process into independent causal mechanisms that we train without direct supervision. By exploiting appropriate inductive biases, these mechanisms disentangle object shape, object texture, and background; hence, they allow for generating counterfactual images. We demonstrate the ability of our model to generate such images on MNIST and ImageNet. Further, we show that the counterfactual images can improve out-of-distribution robustness with a marginal drop in performance on the original classification task, despite being synthetic. Lastly, our generative model can be trained efficiently on a single GPU, exploiting common pre-trained models as inductive biases.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Causal Inference & Discovery

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Causal Inference & Discovery

Citations in the corpus

Projected GANs Converge Faster (2021, neutral): “Pretrained models can be used as a guiding mechanism to disentangle causal generative factors [54], for text-driven image manipulation [44], matching the generator activations to inverted classifiers [19, 56], or to generate images via gradient ascent in the latent space of a generator [41].”
StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets (2022, background): “StyleGAN-XL inherits this ability and, to a certain extent, even generates out-of-domain combinations between different classes, akin to counterfactual images presented in [53].”
On the link between conscious function and general intelligence in humans and machines (2022, background): “While scaling datasets and model architectures has worked well, more structured approaches could lead to more data-eﬃcient generalization: for instance, generative models with explicit machinery for counterfactual reasoning (Sauer & Geiger, 2021).”
Ensembling with Deep Generative Views (2021, methodology): “Augmentation using GANs can be applied at training time for robustness or test-time for ensembling; while we primarily focus on the latter, concurrent works [35, 50] investigate the benefits of the former approach while [56] investigates intermediate GAN representations for few-shot segmentation.”
Learning to Imagine: Integrating Counterfactual Thinking in Neural Discrete Reasoning (2022, background): “A few studies along this line also generate counter-factual samples with neural networks (Sauer and Geiger, 2021; Yue et al., 2021).”