Design Characteristics of Studies Reporting the Performance of Artificial Intelligence Algorithms for Diagnostic Analysis of Medical Images: Results from Recently Published Papers

By Dong Wook Kim, H. Jang, K. Kim, Youngbin Shin, S. Park. Korean Journal of Radiology, 2019

410 citations recorded by the source corpus.

Objective To evaluate the design characteristics of studies that evaluated the performance of artificial intelligence (AI) algorithms for the diagnostic analysis of medical images. Materials and Methods PubMed MEDLINE and Embase databases were searched to identify original research articles published between January 1, 2018 and August 17, 2018 that investigated the performance of AI algorithms that analyze medical images to provide diagnostic decisions. Eligible articles were evaluated to determine 1) whether the study used external validation rather than internal validation, and in case of external validation, whether the data for validation were collected, 2) with diagnostic cohort design instead of diagnostic case-control design, 3) from multiple institutions, and 4) in a prospective manner. These are fundamental methodologic features recommended for clinical validation of AI performance in real-world practice. The studies that fulfilled the above criteria were identified. We classified the publishing journals into medical vs. non-medical journal groups. Then, the results were compared between medical and non-medical journals. Results Of 516 eligible published studies, only 6% (31 studies) performed external validation. None of the 31 studies adopted all three design features: diagnostic cohort design, the inclusion of multiple institutions, and prospective data collection for external validation. No significant difference was found between medical and non-medical journals. Conclusion Nearly all of the studies published in the study period that evaluated the performance of AI algorithms for diagnostic analysis of medical images were designed as proof-of-concept technical feasibility studies and did not have the design features that are recommended for robust validation of the real-world clinical performance of AI algorithms.

View Paper: https://www.semanticscholar.org/paper/4693db07bd0fc78f5f862f29711f575652be670c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

B. van Ginneken
Alan Karthikesalingam
Dominic King
Christopher J. Kelly
Greg C. Corrado
Matthieu Rutten
K. V. van Leeuwen
Mustafa Suleyman

Citations in the corpus

Key challenges for delivering clinical impact with artificial intelligence (2019, background): “A recent systematic review of studies that evaluated AI algorithms for the diagnostic analysis of medical imaging found that only 6% of 516 eligible published studies performed external validation [77].”
The ‘Digital Twin’ to enable the vision of precision cardiology (2020, unknown): “57 The quality of datasets also needs to be thoroughly validated to avoid possible biases before the models developed from them can be integrated in clinical decision-making.”
Artificial intelligence in radiology: 100 commercially available products and their scientific evidence (2021, background): “Only 6% of the recent papers onmedical deep learning algorithms included a validation on independent external data [6].”