The Risks of Invariant Risk Minimization

By Elan Rosenfeld, Pradeep Ravikumar, Andrej Risteski. 2020

369 citations recorded by the source corpus.

Invariant Causal Prediction (Peters et al., 2016) is a technique for out-of-distribution generalization which assumes that some aspects of the data distribution vary across the training set but that the underlying causal mechanisms remain constant. Recently, Arjovsky et al. (2019) proposed Invariant Risk Minimization (IRM), an objective based on this idea for learning deep, invariant features of data which are a complex function of latent variables; many alternatives have subsequently been suggested. However, formal guarantees for all of these works are severely lacking. In this paper, we present the first analysis of classification under the IRM objective$-$as well as these recently proposed alternatives$-$under a fairly natural and general model. In the linear case, we show simple conditions under which the optimal solution succeeds or, more often, fails to recover the optimal invariant predictor. We furthermore present the very first results in the non-linear regime: we demonstrate that IRM can fail catastrophically unless the test data are sufficiently similar to the training distribution$-$this is precisely the issue that it was intended to solve. Thus, in this setting we find that IRM and its alternatives fundamentally do not improve over standard Empirical Risk Minimization.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Continual learning: a feature extraction formalization, an efficient algorithm, and fundamental obstructions (2022, background): “The work of [31, 30] formulate the problem of out-of-distribution generalization and provide theoretical guarantee, similarly, under the assumption of a good representation.”
Discovering Invariant Rationales for Graph Neural Networks (2022, background): “However, concerns about its feasibility (Rosenfeld et al., 2021; Ahuja et al., 2021) and optimality (Kamath et al., 2021) have been discussed recently.”
Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond (2021, background): “However, it may be necessary to observe data from a very large number of environments to disentangle the true causal structure (Rosenfeld et al., 2021).”
Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution (2022, neutral): “…but just require the OOD span T = Range(Σ) to have some directions outside the training span: dim(T \ S) > k. 6Prior work on distribution shift (Rosenfeld et al., 2021; Kamath et al., 2021; Chen et al., 2021b) often considers a worst case loss over some set—we can equivalently write Lood…”
Causally motivated shortcut removal using auxiliary labels (2021, background): “As others (e.g.,Guo et al. [2021], Rosenfeld et al. [2020]) noted, when the family of functions is as ﬂexible as DNNs, it is possible to ﬁnd a predictor that achieves the objective of IRM but is not robust.”
How Neural Networks Extrapolate: From Feedforward to Graph Neural Networks (2020, background): “If the model indeed learns these invariances, which can happen in the linear case and when there are confounders or anti-causal variables (Ahuja et al., 2021; Rosenfeld et al., 2021), this may essentially increase the training data range, since variations in the invariant features may be ignored by ”
Environment Inference for Invariant Learning (2020, background): “Despite the analysis in Arjovsky et al. (2019), this link is not completely clear (Kamath et al., 2021; Rosenfeld et al., 2021).”
Domain Generalization Under Posterior Drift (2025, unknown): “• Rosenfeld et al. [2021] (theory): “Invariant Risk Minimization and its alternatives fundamentally do not improve over standard ERM.””