Underspecification Presents Challenges for Credibility in Modern Machine Learning

By A. D'Amour, Katherine A. Heller, Dan I. Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D. Hoffman and 30 more. 2020

897 citations recorded by the source corpus.

ML models often exhibit unexpectedly poor behavior when they are deployed in real-world domains. We identify underspecification as a key reason for these failures. An ML pipeline is underspecified when it can return many predictors with equivalently strong held-out performance in the training domain. Underspecification is common in modern ML pipelines, such as those based on deep learning. Predictors returned by underspecified pipelines are often treated as equivalent based on their training domain performance, but we show here that such predictors can behave very differently in deployment domains. This ambiguity can lead to instability and poor model behavior in practice, and is a distinct failure mode from previously identified issues arising from structural mismatch between training and deployment domains. We show that this problem appears in a wide variety of practical ML pipelines, using examples from computer vision, medical imaging, natural language processing, clinical risk prediction based on electronic health records, and medical genomics. Our results show the need to explicitly account for underspecification in modeling pipelines that are intended for real-world deployment in any domain.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP (2021, background): “Modern Hopﬁeld networks are a novel concept for contrastive learning to extract more covariance structure.”
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023, background): “In fact, it has been suggested that even the choice of hyperparameters, such as the data ordering, can have an effect on the resulting bias (D’Amour et al., 2020).”
Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, background): “inductive biases that are undesirable [74].”
WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, positive): “Others have studied proprietary datasets with realistic distribution shifts, such as the StreetView StoreFronts dataset (Hendrycks et al., 2020) or diabetic retinopathy datasets (D’Amour et al., 2020).”
TRAK: Attributing Model Behavior at Scale (2023, neutral): “…analysis ignores the fact that in many modern settings, training is non-deterministic: applying the same learning algorithm to the same training dataset (i.e., varying only the random seed) can yield models with (often significantly) differing behavior (Nguyen et al., 2021b; D’Amour et al., 2020).”
Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research (2021, background): “There is 37 also growing concern about the limitations of existing datasets and standard metrics for evaluating 38 model behavior in real-world settings and assessing scientific progress in a problem domain [13, 14].”
How Reliable are Model Diagnostics? (2021, background): “This paper’s findings can be linked to the problems caused by underspecification in machine learning (D’Amour et al., 2020), i.e., when multiple unique predictors trained with the same configuration have the same performance but differ in subtle ways.”
Causally motivated shortcut removal using auxiliary labels (2021, background): “…learning is successful: in learning setups where there are many distinct ways to construct predictors that perform well on held-out data (i.e., when the learning problem is underspeciﬁed [D’Amour et al., 2020]), the inﬂuence of correlated factors is diﬃcult to disentangle a r X i v : 2105 .”
Dawn of the Transformer Era in Speech Emotion Recognition: Closing the Valence Gap (2022, neutral): “This could be a manifestation of the underspecification phenomenon plaguing machine learning architectures [62], as models which have similar performance on the entire test set, nevertheless behave differently across different subsets of it.”
Model Multiplicity: Opportunities, Concerns, and Solutions (2022, background): “Model selection on the basis of accuracy alone is an underspecified [23] selection process.”