Deep learning predicts hip fracture using confounding patient and healthcare variables

By Marcus A. Badgeley, John R. Zech, L. Oakden-Rayner, Benjamin S. Glicksberg, Manway Liu, William Gale, M. McConnell, Bethany Percha, T. Snyder, Joel T. Dudley. 2018

286 citations recorded by the source corpus.

Hip fractures are a leading cause of death and disability among older adults. Hip fractures are also the most commonly missed diagnosis on pelvic radiographs, and delayed diagnosis leads to higher cost and worse outcomes. Computer-aided diagnosis (CAD) algorithms have shown promise for helping radiologists detect fractures, but the image features underpinning their predictions are notoriously difficult to understand. In this study, we trained deep-learning models on 17,587 radiographs to classify fracture, 5 patient traits, and 14 hospital process variables. All 20 variables could be individually predicted from a radiograph, with the best performances on scanner model (AUC = 1.00), scanner brand (AUC = 0.98), and whether the order was marked “priority” (AUC = 0.79). Fracture was predicted moderately well from the image (AUC = 0.78) and better when combining image features with patient data (AUC = 0.86, DeLong paired AUC comparison, p = 2e-9) or patient data plus hospital process features (AUC = 0.91, p = 1e-21). Fracture prediction on a test set that balanced fracture risk across patient variables was significantly lower than a random test set (AUC = 0.67, DeLong unpaired AUC comparison, p = 0.003); and on a test set with fracture risk balanced across patient and hospital process variables, the model performed randomly (AUC = 0.52, 95% CI 0.46–0.58), indicating that these variables were the main source of the model’s fracture predictions. A single model that directly combines image features, patient, and hospital process data outperforms a Naive Bayes ensemble of an image-only model prediction, patient, and hospital process data. If CAD algorithms are inexplicably leveraging patient and process variables in their predictions, it is unclear how radiologists should interpret their predictions in the context of other known patient data. Further research is needed to illuminate deep-learning decision processes so that computers and clinicians can effectively cooperate.

Subfields: Biomedical Engineering, Surgery, Medical & Health AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Surgery

In Medical & Health AI

Citations in the corpus

Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, background): “Black box models often predict the right answer for the wrong reason (the “Clever Hans” phenomenon), leading to excellent performance in training but poor performance in practice [268, 172, 225, 331, 19, 119].”
A case-based interpretable deep learning model for classification of mass lesions in digital mammography (2021, methodology): “If this reasoning process were correct, it would lead to a higher chance that: (1) the model could generalize beyond its finite training set; (2) the model’s reasoning process could be useful information for doctors, even if its prediction is sometimes incorrect; (3) it would be much easier to troub”
WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, positive): “Prior work has shown that individuals—technicians for medical imaging in this case—can not only be identified from data, but also are predictive of the diagnosis, highlighting the risk of learning to classify technicians rather than the medical condition (Badgeley et al., 2019).”
Characterising Bias in Compressed Models (2020, background): “…constrained but sensitive domains such as hiring (Dastin, 2018; Harwell, 2019), health care diagnostics (Xie et al., 2019; Gruetzemacher et al., 2018; Badgeley et al., 2019; Oakden-Rayner et al., 2019), self-driving cars (NHTSA, 2017) and facial recognition software (Buolamwini & Gebru, 2018b).”
Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization (2019, background): “…shifts is important future work, existing work has identiﬁed many distributional shifts that can be expressed with pre-speciﬁed groups, e.g., batch effects in biology (Leek et al., 2010), or image artifacts (Oakden-Rayner et al., 2019) and patient demographics (Badgeley et al., 2019) in medicine.”
COVID-19 Image Data Collection: Prospective Predictions Are the Future (2020, background): “This can be seen by checking if the predictive regions of the image lie outside of the region of interest [Ross et al., 2017; Badgeley et al., 2019; Viviano et al., 2019].”
Debugging Tests for Model Explanations (2020, methodology): “A model trained on this dataset can learn to associate blue sky backgrounds with the bird class; such dataset biases frequently occur in practice [7, 49].”
Just Train Twice: Improving Group Robustness without Training Group Information (2021, background): “Similar performance disparities due to spurious correlations occur in many other applications, including other language tasks, facial recognition, and medical imaging (Gururan-gan et al., 2018; McCoy et al., 2019; Badgeley et al., 2019; Sagawa et al., 2020a; Oakden-Rayner et al., 2020).”
Randomness In Neural Network Training: Characterizing The Impact of Tooling (2021, background): “54 Our results suggest that deterministic tooling is critical for ensuring AI safety in sensitive domains 55 such as credit scoring, health care diagnostics [82, 19, 4, 59] and autonomous driving [54].”
Artificial intelligence with multi-functional machine learning platform development for better healthcare and precision medicine (2020, background): “4’ (249) for analyzing mammogram images for breast density assessment; ‘OsteoDetect’ (84,250,251) for detecting distal radius fractures in the wrist; cloud-based deep neural network algorithm for diagnosing cardiac anomalies and segmentation of lung and liver tumors by analyzing CT scans (56); and A”
Surgical data science – from concepts toward clinical translation (2020, background): “Recent work in biomedical image analysis (Badgeley et al., 2019; Roberts et al., 2021; Dietrich et al., 2021) showed that knowledge of confounding variables is crucial to the development of successful predictive models.”
Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamics (2022, background): “In safety-critical or otherwise sensitive domains such as health care diagnostics [70, 19, 7, 46], self-driving cars [44], hiring [13, 20], and many others, providing tools for domain experts to audit models is of great importance to ensure scalable oversight.”
An adversarial approach for the robust classification of pneumonia from chest radiographs (2020, background): “[1] demonstrated that a deep learning hip fracture classier was leveraging patient-level variables (such as age and gender) and process-level variables (such as scanner model and hospital department) in its predictions.”
A Survey of Deep Learning for Scientific Discovery (2020, background): “Deep learning models are notorious for relying on spurious correlations in the data to perform their predictions [8, 162, 247].”
Selective Classification Can Magnify Disparities Across Groups (2020, background): “els latch onto spurious correlations, e.g., demographics (Buolamwini & Gebru, 2018; Borkan et al., 2019), image backgrounds (Ribeiro et al., 2016; Xiao et al., 2020), spurious clinical variables (Badgeley et al., 2019; Oakden-Rayner et al., 2020), or linguistic artifacts (Gururangan et al., 2018”