Towards Out-Of-Distribution Generalization: A Survey

By Zheyan Shen, Jiashuo Liu, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, Peng Cui. arXiv.org, 2021

709 citations recorded by the source corpus.

Traditional machine learning paradigms are based on the assumption that both training and test data follow the same statistical pattern, which is mathematically referred to as Independent and Identically Distributed ($i.i.d.$). However, in real-world applications, this $i.i.d.$ assumption often fails to hold due to unforeseen distributional shifts, leading to considerable degradation in model performance upon deployment. This observed discrepancy indicates the significance of investigating the Out-of-Distribution (OOD) generalization problem. OOD generalization is an emerging topic of machine learning research that focuses on complex scenarios wherein the distributions of the test data differ from those of the training data. This paper represents the first comprehensive, systematic review of OOD generalization, encompassing a spectrum of aspects from problem definition, methodological development, and evaluation procedures, to the implications and future directions of the field. Our discussion begins with a precise, formal characterization of the OOD generalization problem. Following that, we categorize existing methodologies into three segments: unsupervised representation learning, supervised model learning, and optimization, according to their positions within the overarching learning process. We provide an in-depth discussion on representative methodologies for each category, further elucidating the theoretical links between them. Subsequently, we outline the prevailing benchmark datasets employed in OOD generalization studies. To conclude, we overview the existing body of work in this domain and suggest potential avenues for future research on OOD generalization. A summary of the OOD generalization methodologies surveyed in this paper can be accessed at http://out-of-distribution-generalization.com.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/e5b2e2a284db5ba7c2c011daba9769d2c56b6586

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xiangnan He
2. Xiang Wang
3. Fuli Feng
4. Meng Wang
5. Tat-seng Chua
6. Xia Hu
7. Fan Yang
8. Jiawei Chen

Citations in the corpus

• Data-centric Artificial Intelligence: A Survey (2023, positive): “Although modern machine learning techniques generally perform well on in-distribution datasets, the distribution of data in the deployment environment may not align with the training data [203].”
• Self-Supervised Remote Sensing Feature Learning: Learning Paradigms, Challenges, and Future Works (2022, background): “performance collapse of the model [48], [49], [50].”
• Data collection and quality challenges in deep learning: a data-centric AI perspective (2021, background): “For more details, the reader can refer to [40,146].”
• On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective (2023, background): “The optimal classifier can be learned by optimizing the following objective [27]: Out-of-distribution robustness On the other hand, OOD robustness (generalization) [42, 55] aims to learn an optimal classifier on an unseen distribution by training on existing data.”
• Generalizing to Unseen Domains: A Survey on Domain Generalization (2021, background): “A more recent survey paper is on out-of-distribution (OOD) generalization by Shen et al. [4].”
• Bias and Debias in Recommender System: A Survey and Future Directions (2020, background): “There are some surveys on the bias and fairness of general machine learning and artiicial intelligence systems [43, 119, 124, 149].”