Diagnostic Concordance Among Pathologists Interpreting Breast Biopsy Specimens

By J. Elmore, G. Longton, P. Carney, B. Geller, T. Onega, A. Tosteson, H. Nelson, M. Pepe, K. Allison, S. Schnitt and 2 more. Journal of the American Medical Association (JAMA), 2015

707 citations recorded by the source corpus.

IMPORTANCE A breast pathology diagnosis provides the basis for clinical treatment and management decisions; however, its accuracy is inadequately understood. OBJECTIVES To quantify the magnitude of diagnostic disagreement among pathologists compared with a consensus panel reference diagnosis and to evaluate associated patient and pathologist characteristics. DESIGN, SETTING, AND PARTICIPANTS Study of pathologists who interpret breast biopsies in clinical practices in 8 US states. EXPOSURES Participants independently interpreted slides between November 2011 and May 2014 from test sets of 60 breast biopsies (240 total cases, 1 slide per case), including 23 cases of invasive breast cancer, 73 ductal carcinoma in situ (DCIS), 72 with atypical hyperplasia (atypia), and 72 benign cases without atypia. Participants were blinded to the interpretations of other study pathologists and consensus panel members. Among the 3 consensus panel members, unanimous agreement of their independent diagnoses was 75%, and concordance with the consensus-derived reference diagnoses was 90.3%. MAIN OUTCOMES AND MEASURES The proportions of diagnoses overinterpreted and underinterpreted relative to the consensus-derived reference diagnoses were assessed. RESULTS Sixty-five percent of invited, responding pathologists were eligible and consented to participate. Of these, 91% (N = 115) completed the study, providing 6900 individual case diagnoses. Compared with the consensus-derived reference diagnosis, the overall concordance rate of diagnostic interpretations of participating pathologists was 75.3% (95% CI, 73.4%–77.0%; 5194 of 6900 interpretations). Consensus ReferenceDiagnosis Pathologist Interpretation vs Consensus-Derived Reference Diagnosis, % (95% CI) No. ofInterpretations Overall ConcordanceRate OverinterpretationRate UnderinterpretationRate Benign without atypia 2070 87 (85–89) 13 (11–15) Atypia 2070 48 (44–52) 17 (15–21) 35 (31–39) DCIS 2097 84 (82–86) 3 (2–4) 13 (12–15) Invasive carcinoma 663 96 (94–97) 4 (3–6) Disagreement with the reference diagnosis was statistically significantly higher among biopsies from women with higher (n = 122) vs lower (n = 118) breast density on prior mammograms (overall concordance rate, 73% [95% CI, 71%–75%] for higher vs 77% [95% CI, 75%–80%] for lower, P < .001), and among pathologists who interpreted lower weekly case volumes (P < .001) or worked in smaller practices (P = .034) or nonacademic settings (P = .007). CONCLUSIONS AND RELEVANCE In this study of pathologists, in which diagnostic interpretation was based on a single breast biopsy slide, overall agreement between the individual pathologists’ interpretations and the expert consensus–derived reference diagnoses was 75.3%, with the highest level of concordance for invasive carcinoma and lower levels of concordance for DCIS and atypia. Further research is needed to understand the relationship of these findings with patient management.

Subfields: Artificial Intelligence, Oncology, Pathology and Forensic Medicine

View Paper: https://www.semanticscholar.org/paper/d91e200343fbaa6f68e147bd4a65d4b83a1e36af

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Oncology

In Pathology and Forensic Medicine

Andrew H. Beck

Citations in the corpus

Breast Tumor Cellularity Assessment using Deep Neural Networks (2019, methodology): “The subjectivity in visual tissue assessment motivates the use of computer-aided methods to improve the diagnosis accuracy, reduce human error and increase inter-observer agreement and reproducibility [27, 11].”
Artificial Intelligence and Digital Pathology: Challenges and Opportunities (2018, neutral): “Such language has ramifications for potential monitoring and treatment.[19] Hence, binary language may only be desirable in easy, obvious cases.”
Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy (2017, background): “This is not surprising as grader variability is a well-known issue with human interpretation of imaging in other medical fields such as radiology [6] or pathology [7].”
High resolution mapping of the tumor microenvironment using integrated single-cell, spatial and in situ analysis (2023, neutral): “It has been reported that pathology diagnoses are only con-cordant75%ofthetime 24 .”
BACH: Grand challenge on breast cancer histology images (2018, background): “The analysis of breast cancer WSIs is non-trivial due to the large amount of data to visualize and the complexity of the task [4].”
Deep Learning for Identifying Metastatic Breast Cancer (2016, background): “Limitations of the qualitative visual analysis of microscopic images includes lack of standardization, diagnostic errors, and the significant cognitive load required to manually evaluate millions of cells across hundreds of slides in a typical pathologist’s workday [15, 17, 7].”
The impact of pre- and post-image processing techniques on deep learning frameworks: A comprehensive review for digital pathology image analysis (2020, background): “Manual examination of breast histopathological images calls for an intense workload and, on average, there is only a 75% diagnostic concordance between specialists [112].”
Direct Uncertainty Prediction for Medical Second Opinions (2018, background): “Other examples are given by (Daniel, 2004), a study of tuberculosis diagnosis, showing that radiologists disagree with colleagues 25% of the time, and with themselves 20% of the time, and (Elmore et al., 2015), studying disagreement on cancer diagnosis from breast biopsies.”