Invariant Risk Minimization

By Martín Arjovsky, L. Eon Bottou, Ishaan Gulrajani, David Lopez-Paz. 2019

2,885 citations recorded by the source corpus.

We introduce Invariant Risk Minimization (IRM), a learning paradigm to estimate invariant correlations across multiple training distributions. To achieve this goal, IRM learns a data representation such that the optimal classifier, on top of that data representation, matches for all training distributions. Through theory and experiments, we show how the invariances learned by IRM relate to the causal structures governing the data and enable out-of-distribution generalization.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Shortcut learning in deep neural networks (2020, background): “Under certain assumptions the intended (or even causal) solution can be learned from multiple domains and environments [131, 39, 111].”
Underspecification Presents Challenges for Credibility in Modern Machine Learning (2020, methodology): “As a result, concerns regarding “spurious correlations” and “shortcut learning” in trained models are now widespread (e.g., Geirhos et al., 2020; Arjovsky et al., 2019).”
An Analysis of the Adaptation Speed of Causal Models (2020, background): “This work has set a fruitful line of research for robust machine learning (Heinze-Deml et al., 2018b,a; Rothenhäusler et al., 2019; Arjovsky et al., 2019).”
Shaking the foundations: delusions in sequence models for interaction and control (2021, background): “2The self-delusions discussed here are unrelated to the problem arising from observational confounding [2, 8, 25].”
ZeroPrompt: Scaling Prompt-Based Pretraining to 1, 000 Tasks Improves Zero-Shot Generalization (2022, background): “Previous approaches in domain generalization are mainly based on gradient operations (Gulrajani and Lopez-Paz, 2020; Arjovsky et al., 2019; Mansilla et al., 2021; Kim et al., 2021) or disentanglement learning (Wang et al.”
Adversarial Filters of Dataset Biases (2020, background): “Arjovsky et al. (2019) propose Invariant Risk Minimization as an objective that promotes learning representations of the data which are stable across environments.”
Toward Causal Representation Learning (2021, methodology): “To formalize this general intuition, one can consider a form of out-of-distribution generalization, which can be optimized by minimizing the empirical risk over a class of distributions induced by a causal model of the data [5], [169], [187], [204], [220].”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “…techniques from causal inference (Schölkopf, 2019), uncertainty estimation (Gal and Ghahramani, 2016; Kendall and Gal, 2017), density estimation and generative modeling (Kingma et al., 2014), distributional robustness (Sinha et al., 2017; Sagawa et al., 2019) and invariance (Arjovsky et al., 2019).”
Towards Causal Representation Learning (2021, background): “environments E does not coincide with the set of possible environments PG, we have an additional estimation error that may be arbitrarily large in the worst case [5, 20].”
Learning explanations that are hard to vary (2020, methodology): “, 2016], a method specifically designed to address domain adaptation by obfuscating the domain information with an adversarial classifier; and (ii) Invariant Risk Minimization (IRM) [Arjovsky et al., 2019], which we discuss in detail in the appendix.”
Subspace Regularizers for Few-Shot Class Incremental Learning (2021, background): “Features independently learned for novel classes from small numbers of examples are likely to capture spurious correlations (unrelated to the true causal structure of the prediction problem) as a result of dataset biases (Arjovsky et al., 2019).”
Total Recall: a Customized Continual Learning Method for Neural Semantic Parsers (2021, methodology): “This training algorithm is closely related to Invariant Risk Minimization (Arjovsky et al., 2019), which learns invariant structures across different training environments.”
Continuous Coordination As a Realistic Scenario for Lifelong Learning (2021, methodology): “To this end, we take inspiration from the recent Invariant Risk Minimization (IRM) (Arjovsky et al., 2019) that improves the Out-of-Distribution (OOD) generalization by training an algorithm on multiple environments.”
SWAD: Domain Generalization by Seeking Flat Minima (2021, methodology): “12], obtained well-transferable model parameters by the meta-learning framework [13–16], data augmentation [17–19], or capturing causal relation [20, 21].”
Continual Reinforcement Learning with Multi-Timescale Replay (2020, methodology): “Invariant risk minimisation [Arjovsky et al., 2019] is an approach that seeks to improve out-of-distribution generalisation in machine learning models by training them to learn invariant or stable predictors that avoid spurious correlations in the training data, a common problem with the framework…”
In Search of Lost Domain Generalization (2020, methodology): “Akuzawa et al. (2019) extend DANN by considering cases where there exists an statistical dependence between the domain and the class label variables. Albuquerque et al. (2019) extend DANN by considering one-versus-all adversaries that try to predict to which training domain does each of the examples”
WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, neutral): “Numerous methods have been proposed for domain generalization, including domain-invariant learning (Ganin et al., 2016; Sun and Saenko, 2016), invariant risk minimization (IRM) (Arjovsky et al., 2019), and meta-learning-based methods (Li et al., 2017; Dou et al., 2019).”
Inductive biases for deep learning of higher-level cognition (2020, neutral): “, 2016), spurious correlations (Krueger et al., 2021; Beery et al., 2018; Arjovsky et al., 2019) etc.”
An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models (2020, methodology): “To enable extrapolation, we might need additional inductive bias (Nye et al., 2019) or new learning algorithms (Arjovsky et al., 2019).”