Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans

By M. Roberts, D. Driggs, Matthew Thorpe, Julian D. Gilbey, Michael Yeung, S. Ursprung, Angelica I. Avilés-Rivero, Christian Etmann, C. McCague, L. Beer and 43 more. Nature Machine Intelligence, 2020

936 citations recorded by the source corpus.

Machine learning methods offer great promise for fast and accurate detection and prognostication of coronavirus disease 2019 (COVID-19) from standard-of-care chest radiographs (CXR) and chest computed tomography (CT) images. Many articles have been published in 2020 describing new machine learning-based models for both of these tasks, but it is unclear which are of potential clinical utility. In this systematic review, we consider all published papers and preprints, for the period from 1 January 2020 to 3 October 2020, which describe new machine learning models for the diagnosis or prognosis of COVID-19 from CXR or CT images. All manuscripts uploaded to bioRxiv, medRxiv and arXiv along with all entries in EMBASE and MEDLINE in this timeframe are considered. Our search identified 2,212 studies, of which 415 were included after initial screening and, after quality screening, 62 studies were included in this systematic review. Our review finds that none of the models identified are of potential clinical use due to methodological flaws and/or underlying biases. This is a major weakness, given the urgency with which validated COVID-19 models are needed. To address this, we give many recommendations which, if followed, will solve these issues and lead to higher-quality model development and well-documented manuscripts. Many machine learning-based approaches have been developed for the prognosis and diagnosis of COVID-19 from medical images and this Analysis identifies over 2,200 relevant published papers and preprints in this area. After initial screening, 62 studies are analysed and the authors find they all have methodological flaws standing in the way of clinical utility. The authors have several recommendations to address these issues.

Subfields: Health Informatics, Infectious Diseases, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/69d49a06f09cf934310ccbf3bb2a360fa719272d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Health Informatics

In Radiology, Nuclear Medicine and Imaging

Reabal Najjar

Citations in the corpus

Computational approaches streamlining drug discovery (2023, background): “It has been shown that DL models, especially based on limited datasets lacking negative data, are prone to overtraining and spurious performance, sometimes leading to whole classes of models deemed ‘useless’ 88 or severely biased by subjective factors defining the training dataset 89 .”
Sources of bias in artificial intelligence that perpetuate healthcare disparities—A global review (2022, background): “None of the studies performed a proper assessment of model bias [48].”
Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies (2021, result): “This misalignment is analogous to the discrepancies between the recent burst of COVID-related machine learning papers and clinical practices [116].”
Redefining Radiology: A Review of Artificial Intelligence Integration in Medical Imaging (2023, background): “…mathematical mirroring of reality, depends not only on their training datasets and precision but also on calibration in interpreting medical images depends on comprehensive datasets that accurately reﬂect varied patient demographics, including age, sex, ethnicity, and disease stages [94,95].”
The Fallacy of AI Functionality (2022, background): “Notably, the COVID-19 technology which resulted from innovation policy and democratization efforts mentioned in section 3 was later shown to be completely unsuitable for clinical deployment after the fact [85, 97, 161, 206].”