Anchor regression: Heterogeneous data meet causality

By Dominik Rothenhausler, Nicolai Meinshausen, Peter Buhlmann, Jan Peters. 2018

266 citations recorded by the source corpus.

We consider the problem of predicting a response variable from a set of covariates on a data set that differs in distribution from the training data. Causal parameters are optimal in terms of predictive accuracy if in the new distribution either many variables are affected by interventions or only some variables are affected, but the perturbations are strong. If the training and test distributions differ by a shift, causal parameters might be too conservative to perform well on the above task. This motivates anchor regression, a method that makes use of exogenous variables to solve a relaxation of the ‘causal’ minimax problem by considering a modification of the least‐squares loss. The procedure naturally provides an interpolation between the solutions of ordinary least squares (OLS) and two‐stage least squares. We prove that the estimator satisfies predictive guarantees in terms of distributional robustness against shifts in a linear class; these guarantees are valid even if the instrumental variable assumptions are violated. If anchor regression and least squares provide the same answer (‘anchor stability’), we establish that OLS parameters are invariant under certain distributional changes. Anchor regression is shown empirically to improve replicability and protect against distributional shifts.

Subfields: Artificial Intelligence, Statistics and Probability, Causal Inference & Discovery

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

In Causal Inference & Discovery

Citations in the corpus

Causal Regularization (2019, background): “Accordingly, [12] describes regularization techniques for linear regression that help generalizing across certain shift perturbations.”
Conditional variance penalties and domain shift robustness (2017, neutral): “A similar result is shown in Rothenhäusler et al. (2018) who propose “anchor regression”.”
Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization (2019, background): “…prediction models, mostly from the causal inference literature, similarly aim to achieve high performance on a range of test distributions (Peters et al., 2016; B¨uhlmann & Meinshausen, 2016; Heinze-Deml & Meinshausen, 2017; Rothenh¨ausler et al., 2018; Yang et al., 2019; Arjovsky et al., 2019).”
TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second (2022, background): “It has been demonstrated that causal knowledge can facilitate various ML tasks, including semisupervised learning, transfer learning and out-of-distribution generalization (Schölkopf et al., 2012; Janzing, 2020; Rothenhäusler et al., 2018).”
Robustness to Spurious Correlations via Human Annotations (2020, unknown): “…from domain adaptation (Mansour et al., 2009a;b; Ben-David et al., 2006) causal invariance (Peters et al., 2016; Meinshausen & B¨uhlmann, 2015; Rothenh¨ausler et al., 2018), and robust optimization (Ben-Tal et al., 2013; Duchi & Namkoong, 2018; Bertsimas et al., 2018; Lam & Zhou, 2015), few…”
Out-of-Distribution Generalization via Risk Extrapolation (REx) (2020, background): “ Other approaches focus on adversarially learning representations that are invariant with respect to domainspeciﬁc features (Tzeng et al.,2017;Long et al.,2018;Li et al.,2018;Albuquerque et al.,2019).Rothenhusler et al. (2018) note that OoD generalization can be viewed as the solution to a minimax o”
Preventing Failures Due to Dataset Shift: Learning Predictive Models That Transport (2018, methodology): “On real data we also compare against Anchor Regression (AR), a distributionally robust method for bounded magnitude shift interventions (Rothenhäusler et al., 2018) which requires a causal “anchor” variable with no parents in the graph.”
Fairness Violations and Mitigation under Covariate Shift (2019, methodology): “Results on another method, anchor regression [56], are included in Appendix H.”