Hidden stratification causes clinically meaningful failures in machine learning for medical imaging

By Luke Oakden-Rayner, Jared A. Dunnmon, Gustavo Carneiro, Christopher R'e. 2019

531 citations recorded by the source corpus.

Machine learning models for medical image analysis often suffer from poor performance on important subsets of a population that are not identified during training or testing. For example, overall performance of a cancer detection model may be high, but the model may still consistently miss a rare but aggressive cancer subtype. We refer to this problem as hidden stratification, and observe that it results from incompletely describing the meaningful variation in a dataset. While hidden stratification can substantially reduce the clinical efficacy of machine learning models, its effects remain difficult to measure. In this work, we assess the utility of several possible techniques for measuring hidden stratification effects, and characterize these effects both via synthetic experiments on the CIFAR-100 benchmark dataset and on multiple real-world medical imaging datasets. Using these measurement techniques, we find evidence that hidden stratification can occur in unidentified imaging subsets with low prevalence, low label quality, subtle distinguishing features, or spurious correlates, and that it can result in relative performance differences of over 20% on clinically important subsets. Finally, we discuss the clinical implications of our findings, and suggest that evaluation of hidden stratification should be a critical component of any machine learning deployment in medical imaging.

Subfields: Artificial Intelligence, Health Informatics, Radiology, Nuclear Medicine and Imaging, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Informatics

In Radiology, Nuclear Medicine and Imaging

In Safety, Robustness & Interpretability

Citations in the corpus

Underspecification Presents Challenges for Credibility in Modern Machine Learning (2020, background): “…socially salient characteristics like skin type and gender (Buolamwini and Gebru, 2018); the ML for healthcare literature (Obermeyer et al., 2019; Oakden-Rayner et al., 2020), where examples are stratified by subpopulations; and the natural language processing and computer vision literatures…”
Characterising Bias in Compressed Models (2020, background): “…constrained but sensitive domains such as hiring (Dastin, 2018; Harwell, 2019), health care diagnostics (Xie et al., 2019; Gruetzemacher et al., 2018; Badgeley et al., 2019; Oakden-Rayner et al., 2019), self-driving cars (NHTSA, 2017) and facial recognition software (Buolamwini & Gebru, 2018b).”
Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization (2019, background): “…shifts is important future work, existing work has identiﬁed many distributional shifts that can be expressed with pre-speciﬁed groups, e.g., batch effects in biology (Leek et al., 2010), or image artifacts (Oakden-Rayner et al., 2019) and patient demographics (Badgeley et al., 2019) in medicine.”
Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL (2020, background): “Although identifying hidden stratification is straightforward to identify in hindsight given the hierarchical organization of the diagnostic labels, [55] also demonstrated how to identify groups of samples exhibiting hidden stratification for a given class label under consideration using an unsuperv”
Surgical data science – from concepts toward clinical translation (2020, background): “Other examples of confounders (also called hidden variables) are chest drains and skin markings in the context of pneumothorax (Oakden-Rayner et al., 2020) and melanoma diagnosis (Winkler et al.”
Assessing the (Un)Trustworthiness of Saliency Maps for Localizing Abnormalities in Medical Imaging (2020, background): “For example, a chest tube in an image would imply the presence of a pneumothorax, but much of the chest tube may not be in the region-of-interest [53].”
Slice-based Learning: A Programming Model for Residual Learning in Critical Data Slices (2019, background): “bone X-rays associated with degenerative joint disease [27]).”
Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology with the Fitzpatrick 17k Dataset (2021, background): “…within a population has been shown to lead deep neural networks to produce large accuracy disparities across gender and skin color for facial recognition [11], across images with and without surgical markings in dermatology [56, 10], and across treated and untreated conditions in radiology [40].”
The Fallacy of AI Functionality (2022, unknown): “read medical images [66, 132], and misdiagnose [180, 203].”
Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension (2020, unknown): “53 Therefore, reporting cases of error and defining risk-mitigation strategies are important for informing when, and for which populations, the intervention can be safely implemented.”
No Subclass Left Behind: Fine-Grained Robustness in Coarse-Grained Classification Problems (2020, methodology): “Nevertheless, recent empirical evidence [36] encouragingly suggests that feature representations of deep neural networks often carry information about unlabeled subclasses (see Figure 1).”
Ethical Machine Learning in Healthcare (2020, background): “Clinical analysis of sources of error in disease labels can also guide improvements (89) and identify affected groups (56).”
Gradient Starvation: A Learning Proclivity in Neural Networks (2020, background): “Recent work aims at identifying the reasons behind this phenomenon [97, 70, 22, 73, 51, 76, 100, 92, 83, 105, 42, 79, 4], while complementary work quantifies resulting shortcomings, including poor generalization to out-of-distribution (OOD) test data, reliance upon spurious correlations, and lack of”
Selective Classification Can Magnify Disparities Across Groups (2020, background): “, 2020), spurious clinical variables (Badgeley et al., 2019; Oakden-Rayner et al., 2020), or linguistic artifacts (Gururangan et al.”