In Search of Lost Domain Generalization

By Ishaan Gulrajani, David Lopez-Paz. 2020

1,484 citations recorded by the source corpus.

The goal of domain generalization algorithms is to predict well on distributions different from those seen during training. While a myriad of domain generalization algorithms exist, inconsistencies in experimental conditions -- datasets, architectures, and model selection criteria -- render fair and realistic comparisons difficult. In this paper, we are interested in understanding how useful domain generalization algorithms are in realistic settings. As a first step, we realize that model selection is non-trivial for domain generalization tasks. Contrary to prior work, we argue that domain generalization algorithms without a model selection strategy should be regarded as incomplete. Next, we implement DomainBed, a testbed for domain generalization including seven multi-domain datasets, nine baseline algorithms, and three model selection criteria. We conduct extensive experiments using DomainBed and find that, when carefully implemented, empirical risk minimization shows state-of-the-art performance across all datasets. Looking forward, we hope that the release of DomainBed, along with contributions from fellow researchers, will streamline reproducible and rigorous research in domain generalization.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Evaluation & Benchmarks

Citations in the corpus

ZeroPrompt: Scaling Prompt-Based Pretraining to 1, 000 Tasks Improves Zero-Shot Generalization (2022, background): “Previous approaches in domain generalization are mainly based on gradient operations (Gulrajani and Lopez-Paz, 2020; Arjovsky et al., 2019; Mansilla et al., 2021; Kim et al., 2021) or disentanglement learning (Wang et al.”
ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models (2022, background): “Open-set recognition allows new concepts in evaluation, while typically remains the same visual domain [87, 93]; Domain shift allows new visual domain in evaluation, while typically remains the same concept pool [63, 28].”
On Generalizing Beyond Domains in Cross-Domain Continual Learning (2022, methodology): “This straightforward idea from supervised learning is effective for domain generalization as also shown in [13].”
SWAD: Domain Generalization by Seeking Flat Minima (2021, methodology): “[22] (denoted with † ).”
WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, positive): “In particular, they show empirically that on the DomainBed dataset, using ID validation accuracy instead of OOD validation accuracy leads to higher OOD accuracy.”
MSeg: A Composite Dataset for Multi-Domain Semantic Segmentation (2020, neutral): “An extensive collection of these methods have recently been summarized and benchmarked in various settings [29], [30], and we refer the interested reader to these studies.”
Generalizing to Unseen Domains: A Survey on Domain Generalization (2021, methodology): “6) Performance Evaluation for DG: The recent work [87] pointed out that on several datasets, the performance of some DG methods is almost the same as the baseline (i.”
Adaptive Risk Minimization: A Meta-Learning Approach for Tackling Group Shift (2020, methodology): “…primarily been tested in settings with an order of magnitude fewer source domains – usually 4 to 6, rather than tens or hundreds – and more data per domain [18], and employing techniques such as an adversarial domain classiﬁer may be less effective in general as the number of domains increases.”
Gradient Matching for Domain Generalization (2021, positive): “Datasets While W ILDS is a challenging benchmark capturing realistic distribution shift, to test our model under the synthetic-to-real transfer setting and provide more comparisons to SOTA methods, we also performed experiments on the D OMAIN B ED benchmark (Gulrajani and Lopez-Paz, 2020).”
Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Development (2021, methodology): “Second, although the best performed methods are MMD 206 and CORAL, the standard training strategy has similar performances as current ML SOTA domain 207 generalization algorithms, which confirms with the systematic study conducted by [46], highlighting 208 a demand for robust domain generalization m”
Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond (2021, background): “well across a set of causally-compatible domains, known as domain generalization (Ghifary et al., 2015; Gulrajani and Lopez-Paz, 2020).”
DORO: Distributional and Outlier Robust Optimization (2021, neutral): “(Gulrajani & Lopez-Paz, 2021) also pointed out the difﬁ-culty of model selection in domain-oblivious distributional shift tasks.”
Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalization (2021, methodology): “…distribution as train domain and does limited model queries (we set 25 queries), b) oracle test-domain validation set hyperparameter tuning procedure (Gulrajani and Lopez-Paz, 2021), which takes samples from the same distribution as test domain and does limited model queries (we set 25 queries).”
Domain Generalization using Causal Matching (2020, methodology): “The results for DANN (Ganin et al., 2016), CDANN (Li et al., 2018d), MMD, C-MMD (Li et al., 2018b), CORAL, C-CORAL (Sun & Saenko, 2016) were computed by using their implementations in DomainBed (Gulrajani & Lopez-Paz, 2020).”
Domain Generalization via Gradient Surgery (2021, methodology): “Learning a model with data from different domains and then applying it to a new domain not seen during training entails a domain generalization (DG) problem [8].”