Generalization in anti-causal learning

By Niki Kilbertus, Giambattista Parascandolo, Bernhard Schölkopf. 2018

56 citations recorded by the source corpus.

The ability to learn and act in novel situations is still a prerogative of animate intelligence, as current machine learning methods mostly fail when moving beyond the standard i.i.d. setting. What is the reason for this discrepancy? Most machine learning tasks are anti-causal, i.e., we infer causes (labels) from effects (observations). Typically, in supervised learning we build systems that try to directly invert causal mechanisms. Instead, in this paper we argue that strong generalization capabilities crucially hinge on searching and validating meaningful hypotheses, requiring access to a causal model. In such a framework, we want to find a cause that leads to the observed effect. Anti-causal models are used to drive this search, but a causal model is required for validation. We investigate the fundamental differences between causal and anti-causal tasks, discuss implications for topics ranging from adversarial attacks to disentangling factors of variation, and provide extensive evidence from the literature to substantiate our view. We advocate for incorporating causal models in supervised learning to shift the paradigm from inference only, to search and validation.

Subfields: Artificial Intelligence, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Scaling & Learning Theory

Citations in the corpus

Shortcut learning in deep neural networks (2020, background): “We can easily see the connection to machine learning—transferring knowledge to novel problems corresponds to testing generalisation beyond the narrowly learned setting [89, 90, 91].”
Toward Causal Representation Learning (2021, background): “One may, thus, expect that if the predictor approximates the causal mechanism that is inherently transferable and robust, adversarial examples should be harder to ﬁnd [133], [216].”
Towards Causal Representation Learning (2021, background): “that is inherently transferable and robust, adversarial examples should be harder to find [216, 134].”
Causality for Machine Learning (2019, background): “One may thus hypothesize that for causal learning problems (predicing effect from cause), adversarial examples should be impossible, or at least harder to find (Schölkopf, 2017; Kilbertus et al., 2018).”
Invariant Risk Minimization (2019, background): “I attending a lecture by Professor Sch¨olkopf [22] where he mentioned that P ( Y e | X e ) is often invariant across environments when X e is a cause of Y e , and that it often varies when X e is an eﬀect of Y e .”
MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models (2023, background): “MetaMath aims to transfer the underlying meta-knowledge to enable strong generalization [34].”