Stable Prediction across Unknown Environments

By Kun Kuang, Ruoxuan Xiong, Peng Cui, Susan Athey, Bo Li. 2018

191 citations recorded by the source corpus.

In many important machine learning applications, the training distribution used to learn a probabilistic classifier differs from the distribution on which the classifier will be used to make predictions. Traditional methods correct the distribution shift by reweighting training data with the ratio of the density between test and training data. However, in many applications training takes place without prior knowledge of the testing distribution. Recently, methods have been proposed to address the shift by learning the underlying causal structure, but those methods rely on diversity arising from multiple training data sets, and they further have complexity limitations in high dimensions. In this paper, we propose a novel Deep Global Balancing Regression (DGBR) algorithm to jointly optimize a deep auto-encoder model for feature selection and a global balancing model for stable prediction across unknown environments. The global balancing model constructs balancing weights that facilitate estimation of partial effects of features (holding fixed all other features), a problem that is challenging in high dimensions, and thus helps to identify stable, causal relationships between features and outcomes. The deep auto-encoder model is designed to reduce the dimensionality of the feature space, thus making global balancing easier. We show, both theoretically and with empirical experiments, that our algorithm can make stable predictions across unknown environments. Our experiments on both synthetic and real datasets demonstrate that our algorithm outperforms the state-of-the-art methods for stable prediction across unknown environments.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Invariant Risk Minimization (2019, background): “Some works in machine learning [45, 18, 21, 26, 36, 43, 34, 7] pursue similar questions.”
A Survey on Causal Inference (2020, methodology): “addition to the decision support, various balancing methods that can handle the selection bias ( mentioned in Section 3), can also be extended to improve the stability of machine learning methods. In [67], the reweighting method is adopted to improve the generalization ability of learned models for ”
Deep Stable Learning for Out-Of-Distribution Generalization (2021, background): “With this goal, a strand of research on stable learning are proposed [50, 28].”
Towards Out-Of-Distribution Generalization: A Survey (2021, background): “[120] propose StableNet, as shown in figure 2, which extends former linear frameworks [4], [12], [13] to incorporate deep models.”
A Survey of Learning Causality with Data: Problems and Methods (2018, result): “Similar results for domain generalization have been obtained through a global balancing approach [73] and a causal feature selection method [98].”
Explaining The Efficacy of Counterfactually-Augmented Data (2020, background): “…where the relationship between a target variable and its direct causal ancestors remains invariant across foreseeable shifts (Peters et al., 2016; Ghassami et al., 2017; Rojas-Carulla et al., 2018; Kuang et al., 2018; Magliacane et al., 2018; Christiansen & Peters, 2020; Weichwald & Peters, 2020).”
A Survey of Learning Causality with Data (2018, background): “Intuitively, the lower the probability of a feature to be a cause, the more penalty will be added to its corresponding weight, which eventually encourages the model to pay more attention to those features that are more likely to be causes of the label.”
Towards Non-I.I.D. image classification: A dataset and baselines (2019, unknown): “Lately [42] leverages causality for predictive modeling.”