WILDS: A Benchmark of in-the-Wild Distribution Shifts

By Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard L. Phillips, Irena Gao and 14 more. 2020

1,913 citations recorded by the source corpus.

Distribution shifts can cause significant degradation in a broad range of machine learning (ML) systems deployed in the wild. However, many widely-used datasets in the ML community today were not designed for evaluating distribution shifts. These datasets typically have training and test sets drawn from the same distribution, and prior work on retrofitting them with distribution shifts has generally relied on artificial shifts that need not represent the kinds of shifts encountered in the wild. In this paper, we present WILDS, a benchmark of in-the-wild distribution shifts spanning diverse data modalities and applications, from tumor identification to wildlife monitoring to poverty mapping. WILDS builds on top of recent data collection efforts by domain experts in these applications and provides a unified collection of datasets with evaluation metrics and train/test splits that are representative of real-world distribution shifts. These datasets reflect distribution shifts arising from training and testing on different hospitals, cameras, countries, time periods, demographics, molecular scaffolds, etc., all of which cause substantial performance drops in our baseline models. Finally, we survey other applications that would be promising additions to the benchmark but for which we did not manage to find appropriate datasets; we discuss their associated challenges and detail datasets and shifts where we did not see an appreciable performance drop. By unifying datasets from a variety of application areas and making them accessible to the ML community, we hope to encourage the development of general-purpose methods that are anchored to real-world distribution shifts and that work well across different applications and problem settings. Data loaders, default models, and leaderboards are available at this https URL.

Subfields: Artificial Intelligence, Radiology, Nuclear Medicine and Imaging, Evaluation & Benchmarks, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Radiology, Nuclear Medicine and Imaging

In Evaluation & Benchmarks

In Safety, Robustness & Interpretability

Citations in the corpus

Break-It-Fix-It: Unsupervised Learning for Program Repair (2021, background): “Domain adaptation aims to address the mismatch in data distribution between training and test domains (Daume III & Marcu, 2006; Quionero-Candela et al., 2009; Koh et al., 2021).”
ORBIT: A Real-World Few-Shot Dataset for Teachable Object Recognition (2021, background): “Datasets captured by users in real-world settings are naturally high-variation [1, 12, 7, 21, 27, 18, 41, 13], but none collected thus far explicitly target few-shot object recognition.”
Unsolved Problems in ML Safety (2021, background): “In addition to existing robustness benchmarks [78, 102, 75], researchers could create more environments and benchmarks to stress-test systems, ﬁnd their breaking points, and determine whether they will function appropriately in potential future scenarios.”
FLEX: Unifying Evaluation for Few-Shot NLP (2021, background): “We believe it is crucial to continually evolve the suite of datasets to remain challenging for the best models [36] and to tackle real-world challenges [1].”
Inductive biases for deep learning of higher-level cognition (2020, background): “Instead, current systems are generally not robust to changes in distribution (Peters et al., 2017b; Geirhos et al., 2020; Hendrycks et al., 2021; Koh et al., 2021; Schneider et al., 2020), adversarial examples (Goodfellow et al.”
Robust fine-tuning of zero-shot models (2021, methodology): “In addition to the aforementioned ImageNet distribution shifts, WiSE-FT consistently improves robustness on a diverse set of six further distribution shifts including: (i) geographic shifts in satellite imagery and wildlife recognition (WILDS-FMoW, WILDS-iWildCam) (Koh et al., 2021; Christie et al.,”
MSeg: A Composite Dataset for Multi-Domain Semantic Segmentation (2020, neutral): “An extensive collection of these methods have recently been summarized and benchmarked in various settings [29], [30], and we refer the interested reader to these studies.”
Visual Prompting: Modifying Pixel Space to Adapt Pre-trained Models (2022, methodology): “training distribution di ers from the test distribution, we evaluate on three image classi cation datasets in WILDS [35]: Camelyon17 [36], FMoW [37], and iWildCAM [38].”
Characterizing Generalization under Out-Of-Distribution Shifts in Deep Metric Learning (2021, background): “It thus remains unclear how well these methods generalize in more realistic settings [28] under potentially much more challenging, different train-to-test distribution shifts, which we investigate in this work.”
Evaluation Gaps in Machine Learning Practice (2022, methodology): “Our framework for analyzing the gaps builds upon and complements other streams of work on ML evaluation practices, including addressing distribution shifts between development data and application data [34, 94, 160], and robustness to perturbations in test items [118, 132, 173 ?”
The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning (2021, background): “Common examples include data captured in a different environment, like time of day or geographical location [17]; noise or small corruptions of the input data [12, 7]; or adversarial examples created to explicitly fool neural networks into making incorrect predictions [40, 3].”
Text Data Augmentation for Deep Learning (2021, methodology): “The original Transformer [92] uses sine and cosine functions to integrate positional information into text sequences.”
On the Importance of Gradients for Detecting Distributional Shifts in the Wild (2021, background): “Distributional shifts Distributional shifts have attracted increasing research interests [20].”
Conformal prediction with conditional guarantees (2023, positive): “Our next experiment examines the RxRx1 dataset (Taylor et al. (2019)) from the WILDS repository (Koh et al. (2021)).”
Generalizing to Unseen Domains: A Survey on Domain Generalization (2021, methodology): “The last ten datasets are from WILDS [152].”
Data-centric Artificial Intelligence: A Survey (2023, neutral): “Inference data development In-distribution Data slicing [53], algorithmic recourse [120] Out-of-distribution Generating adversarial samples [165], generating samples with distribution shift [125] Prompt engineeringManual prompt engineering [206], automated prompt engineering [238]”
On the Paradox of Learning to Reason from Data (2022, background): “On the contrary, discussions on OOD generalization often involve open problem space (Lin et al., 2019; Gontier et al., 2020; Wald et al., 2021) and domain/concept mismatch between training and testing distribution (Yin et al., 2021; Koh et al., 2021).”