Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalization

By Kartik Ahuja, Ethan Caballero, Dinghuai Zhang, Yoshua Bengio, Ioannis Mitliagkas, Irina Rish. Neural Information Processing Systems, 2021

353 citations recorded by the source corpus.

The invariance principle from causality is at the heart of notable approaches such as invariant risk minimization (IRM) that seek to address out-of-distribution (OOD) generalization failures. Despite the promising theory, invariance principle-based approaches fail in common classification tasks, where invariant (causal) features capture all the information about the label. Are these failures due to the methods failing to capture the invariance? Or is the invariance principle itself insufficient? To answer these questions, we revisit the fundamental assumptions in linear regression tasks, where invariance-based approaches were shown to provably generalize OOD. In contrast to the linear regression tasks, we show that for linear classification tasks we need much stronger restrictions on the distribution shifts, or otherwise OOD generalization is impossible. Furthermore, even with appropriate restrictions on distribution shifts in place, we show that the invariance principle alone is insufficient. We prove that a form of the information bottleneck constraint along with invariance helps address key failures when invariant features capture all the information about the label and also retains the existing success when they do not. We propose an approach that incorporates both of these principles and demonstrate its effectiveness in several experiments.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/4390d210bfd4cd7b646f13f287f44f9620a4f214

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mingsheng Long
2. Jianmin Wang
3. Jindong Wang
4. Tao Qin
5. Yidong Ouyang
6. Haixu Wu
7. Yong Liu
8. Chang Liu

Citations in the corpus

• Generalizing to Unseen Domains: A Survey on Domain Generalization (2021, background): “[105] found the invariance of f alone is not sufficient.”
• Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting (2022, background): “Besides, it is a fundamental problem to make deep models generalize well on a varying distribution [26, 19, 5].”
• HyQE: Ranking Contexts with Hypothetical Query Embeddings (2024, background): “It is widely known that generative models that respect the causality relationships are more robust to distribution shifts because they can avoid learning spurious relationships between random variables (Ahuja et al., 2021; Schölkopf et al., 2021; Lu et al., 2022).”