AI for radiographic COVID-19 detection selects shortcuts over signal

By Alex J. DeGrave, Joseph D. Janizek, Su-In Lee. Nature Machine Intelligence, 2020

727 citations recorded by the source corpus.

Artificial intelligence (AI) researchers and radiologists have recently reported AI systems that accurately detect COVID-19 in chest radiographs. However, the robustness of these systems remains unclear. Using state-of-the-art techniques in explainable AI, we demonstrate that recent deep learning systems to detect COVID-19 from chest radiographs rely on confounding factors rather than medical pathology, creating an alarming situation in which the systems appear accurate, but fail when tested in new hospitals. We observe that the approach to obtain training data for these AI systems introduces a nearly ideal scenario for AI to learn these spurious ‘shortcuts’. Because this approach to data collection has also been used to obtain training data for the detection of COVID-19 in computed tomography scans and for medical imaging tasks related to other diseases, our study reveals a far-reaching problem in medical-imaging AI. In addition, we show that evaluation of a model on external data is insufficient to ensure AI systems rely on medically relevant pathology, because the undesired ‘shortcuts’ learned by AI systems may not impair performance in new hospitals. These findings demonstrate that explainable AI should be seen as a prerequisite to clinical deployment of machine-learning healthcare models. The urgency of the developing COVID-19 epidemic has led to a large number of novel diagnostic approaches, many of which use machine learning. DeGrave and colleagues use explainable AI techniques to analyse a selection of these approaches and find that the methods frequently learn to identify features unrelated to the actual disease.

Subfields: Health Informatics, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/6e62a903dac8e643160bd1337d6cb8b09bf2f062

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Health Informatics

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Algorithm fairness in artificial intelligence for medicine and healthcare (2023, background): “Still, saliency mapping can be used to detect spurious bugs or artefacts in the input feature space (although they do not always detect them in ‘contaminated’ models 323,324 ).”
Deep Learning for Chest X-ray Analysis: A Survey (2021, neutral): “Such concerns about dataset construction for COVID-19 studies have been discussed in several other works (López-Cabrera et al., 2021; DeGrave et al., 2020; Cruz et al., 2021; Maguolo and Nanni, 2020; Tartaglione et al., 2020).”
Obtaining genetics insights from deep learning via explainable artificial intelligence (2022, background): “Interpretations can provide a way to identify such spurious correlations; for example, in a medical imaging study, model interpretation enabled the researchers to identify hidden batch effects that were being used by the model to make predictions 84 .”
AI in health and medicine (2022, methodology): “We therefore encourage researchers to be wary of technological bias, even when using data from diverse sources 97 .”
WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, positive): “…hospitals and deployment sites has also been shown to degrade model accuracy in other medical applications such as diabetic retinopathy (Beede et al., 2020) and chest radiographs (Zech et al., 2018; Phillips et al., 2020), including recent work on COVID-19 detection (DeGrave et al., 2020).”
FUTURE-AI: international consensus guideline for trustworthy and deployable artificial intelligence in healthcare (2023, neutral): “72 The evaluations should also identify any limitations of the AI explanations (eg, they are clinically incoherent 73 or sensitive to noise or adversarial attacks, 74 they unreasonably increase the confidence in the AI generated results 75 ).”
Distilling Model Failures as Directions in Latent Space (2022, neutral): “…on speciﬁc subpopulations in the data if that subpopulation was mislabeled (Northcutt et al., 2021; Stock & Cisse, 2018; Beyer et al., 2020; Vasudevan et al., 2022), underrepresented (Sagawa et al., 2020; Santurkar et al., 2021), or corrupted (Hendrycks & Dietterich, 2019; Hendrycks et al., 2020).”
AI recognition of patient race in medical imaging: a modelling study (2021, neutral): “…potential explanations of race detection that could target the known shortcut mechanisms that deep models might be using as proxies for race 33 by evaluating, first, frequency domain differences in the high frequency image features (ie, textural) and low frequency image features (ie,…”
COVID-19 Image Data Collection: Prospective Predictions Are the Future (2020, background): “Additionally, much research is presented without appropriate evaluations, potentially leading to overﬁtting and performance overestimation [Maguolo and Nanni, 2020; Tartaglione et al., 2020; DeGrave et al., 2020].”