Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy
By Jonathan Krause, Varun Gulshan, Ehsan Rahimy, Peter Karth, Kasumi Widner, Greg S. Corrado, Lily H. Peng, Dale R. Webster. 2017
503 citations recorded by the source corpus.
PURPOSE Use adjudication to quantify errors in diabetic retinopathy (DR) grading based on individual graders and majority decision, and to train an improved automated algorithm for DR grading. DESIGN Retrospective analysis. PARTICIPANTS Retinal fundus images from DR screening programs. METHODS Images were each graded by the algorithm, U.S. board-certified ophthalmologists, and retinal specialists. The adjudicated consensus of the retinal specialists served as the reference standard. MAIN OUTCOME MEASURES For agreement between different graders as well as between the graders and the algorithm, we measured the (quadratic-weighted) kappa score. To compare the performance of different forms of manual grading and the algorithm for various DR severity cutoffs (e.g., mild or worse DR, moderate or worse DR), we measured area under the curve (AUC), sensitivity, and specificity. RESULTS Of the 193 discrepancies between adjudication by retinal specialists and majority decision of ophthalmologists, the most common were missing microaneurysm (MAs) (36%), artifacts (20%), and misclassified hemorrhages (16%). Relative to the reference standard, the kappa for individual retinal specialists, ophthalmologists, and algorithm ranged from 0.82 to 0.91, 0.80 to 0.84, and 0.84, respectively. For moderate or worse DR, the majority decision of ophthalmologists had a sensitivity of 0.838 and specificity of 0.981. The algorithm had a sensitivity of 0.971, specificity of 0.923, and AUC of 0.986. For mild or worse DR, the algorithm had a sensitivity of 0.970, specificity of 0.917, and AUC of 0.986. By using a small number of adjudicated consensus grades as a tuning dataset and higher-resolution images as input, the algorithm improved in AUC from 0.934 to 0.986 for moderate or worse DR. CONCLUSIONS Adjudication reduces the errors in DR grading. A small set of adjudicated DR grades allows substantial improvements in algorithm performance. The resulting algorithm's performance was on par with that of individual U.S. Board-Certified ophthalmologists and retinal specialists.
Subfields: Ophthalmology, Radiology, Nuclear Medicine and Imaging, Evaluation & Benchmarks, Medical Imaging
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Ophthalmology
- András Hajdu
- William T. Freeman
- Arunkumar Rajendran
- Renzo Perfetti
- Elisa Ricci
- Palanivel Karthigaikumar
In Radiology, Nuclear Medicine and Imaging
- Balázs Harangi
- P. Heng
- Lequan Yu
- Woong Bae
- Michal Irani
- Ana Maria Mendonça
- Chi-Wing Fu
- Fabrice Mériaudeau
In Evaluation & Benchmarks
- Phillip Isola
- William T. Freeman
- Greg S. Corrado
- Michal Irani
- Chi-Wing Fu
- Lei Zhu
- Gal Elidan
- Baocai Yin
In Medical Imaging
Citations in the corpus
- Explaining in Style: Training a GAN to explain a classifier in StyleSpace (2021, methodology): “er to its prediction on this image. Dataset Classifier AFHQ [2] Cats / dogs AFHQ Wild cats species FFHQ [14] Perceived gender FFHQ Perceived age Plant-Village [13] Healthy / sick leaves Retinal Fundus [17] DME / non-DME Table 1. List of datasets used in this paper. 4. Evaluation and Results We test o”
- Unbox the black-box for the medical explainable AI via multi-modal and multi-centre data fusion: A mini-review, two showcases and beyond (2021, background): “Where the diagnostic role has an incomplete inter-expert agreement, it has been shown that consensus diagnostics could greatly boost the efficiency of the training of the deep learning based models [56].”
- CANet: Cross-Disease Attention Network for Joint Diabetic Retinopathy and Diabetic Macular Edema Grading (2019, methodology): “Convolutional neural networks (CNNs) have been proven to be a powerful tool to learn features for DR [3]–[5] and DME [6], [7] grading.”
- Ensuring Fairness in Machine Learning to Advance Health Equity (2018, background): “Machine learning can identify the statistical patterns of data generated by tens of thousands of physicians and billions of patients to train computers to perform specific tasks with sometimes superhuman ability, such as detecting diabetic eye disease better than retinal specialists (1).”
- Artificial intelligence in retina. (2018, methodology): “Introduction of an automated algorithm for DR grading compared with retinal specialist gradings achieved substantial improvements in correct adjudication, including evident DR features such as microaneurysms (Krause et al., 2018).”
- IDRiD: Diabetic Retinopathy - Segmentation and Grading Challenge. (2020, methodology): “As confirmed by recent studies ( Krause et al., 2018; Son et al., 2019 ), we hypothesized that algorithms developed using images with fine visibility and images having high resolution with ad-judicated consensus grades yield better performance when compared to datasets consisting of poor-quality…”