Environment Inference for Invariant Learning

By Elliot Creager, Jörn-Henrik Jacobsen, Richard S. Zemel. International Conference on Machine Learning, 2020

469 citations recorded by the source corpus.

Learning models that gracefully handle distribution shifts is central to research on domain generalization, robust optimization, and fairness. A promising formulation is domain-invariant learning, which identifies the key issue of learning which features are domain-specific versus domain-invariant. An important assumption in this area is that the training examples are partitioned into"domains"or"environments". Our focus is on the more common setting where such partitions are not provided. We propose EIIL, a general framework for domain-invariant learning that incorporates Environment Inference to directly infer partitions that are maximally informative for downstream Invariant Learning. We show that EIIL outperforms invariant learning methods on the CMNIST benchmark without using environment labels, and significantly outperforms ERM on worst-group performance in the Waterbirds and CivilComments datasets. Finally, we establish connections between EIIL and algorithmic fairness, which enables EIIL to improve accuracy and calibration in a fair prediction problem.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/00325cb5408da77827951abd3fa93ec3bd019608

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations (2022, methodology): “Several methods have been proposed for the scenario where group labels are not known, and need to be inferred from the data; these methods typically train a pair of networks, where the first model is used to identify the challenging minority examples and define a weighted loss for the second model [”
Fairness without Demographics through Adversarially Reweighted Learning (2020, background): “[12] propose an invariant risk minimization approach for domain generalization where environment partitions are not provided.”
Discovering Invariant Rationales for Graph Neural Networks (2022, methodology): “, 2020) or adversarial environment inference (Creager et al., 2021; Wang et al., 2021b).”
Towards Out-Of-Distribution Generalization: A Survey (2021, background): “And Creager et al. targets on the problem of missing environment labels of IRM, and raise Environment Inference for Invariant Learning(EIIL [3]) to learn environments that maximize the IRM’s penalty.”
Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamics (2022, background): “…et al., 2020; Jiang et al., 2020), difficult examples (Agarwal et al., 2021), minority-group examples (Liu et al., 2021; Sagawa et al., 2020; 2019; Zhang et al., 2022), points worth training (Mindermann et al., 2022), or domain identity in a domain generalization setting (Creager et al., 2021).”
Few-Shot Learning With a Strong Teacher (2021, background): “…leveraged the graph propagation to improve meta-learning; [75], [76] constructed a generalized FSL objective towards adaptive calibration strategies between base and novel classes; [77] investigated specialized data augmentations; [78] studied how to identify invariant features in meta-learning.”
Just Train Twice: Improving Group Robustness without Training Group Information (2021, methodology): “Empirically, despite its simplicity, J TT performs better than LfF. Concurrently, Creager et al. (2021) also proposed a method that leverages similar intuition to J TT .”
Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models (2024, unknown): “…2022), automatically or manually reweighting data between groups (Liu et al., 2021; Nam et al., 2020; Yaghoobzadeh et al., 2021; Utama et al., 2020; Creager et al., 2021; Idrissi et al., 2022; Orgad & Belinkov, 2023), training classifiers with more favorable inductive biases (Kirichenko et al.,…”