Evaluation of symptom checkers for self diagnosis and triage: audit study
By Hannah L Semigran, J. Linder, C. Gidengil, Ateev Mehrotra. 2015
482 citations recorded by the source corpus.
Objective To determine the diagnostic and triage accuracy of online symptom checkers (tools that use computer algorithms to help patients with self diagnosis or self triage). Design Audit study. Setting Publicly available, free symptom checkers. Participants 23 symptom checkers that were in English and provided advice across a range of conditions. 45 standardized patient vignettes were compiled and equally divided into three categories of triage urgency: emergent care required (for example, pulmonary embolism), non-emergent care reasonable (for example, otitis media), and self care reasonable (for example, viral upper respiratory tract infection). Main outcome measures For symptom checkers that provided a diagnosis, our main outcomes were whether the symptom checker listed the correct diagnosis first or within the first 20 potential diagnoses (n=770 standardized patient evaluations). For symptom checkers that provided a triage recommendation, our main outcomes were whether the symptom checker correctly recommended emergent care, non-emergent care, or self care (n=532 standardized patient evaluations). Results The 23 symptom checkers provided the correct diagnosis first in 34% (95% confidence interval 31% to 37%) of standardized patient evaluations, listed the correct diagnosis within the top 20 diagnoses given in 58% (55% to 62%) of standardized patient evaluations, and provided the appropriate triage advice in 57% (52% to 61%) of standardized patient evaluations. Triage performance varied by urgency of condition, with appropriate triage advice provided in 80% (95% confidence interval 75% to 86%) of emergent cases, 55% (47% to 63%) of non-emergent cases, and 33% (26% to 40%) of self care cases (P<0.001). Performance on appropriate triage advice across the 23 individual symptom checkers ranged from 33% (95% confidence interval 19% to 48%) to 78% (64% to 91%) of standardized patient evaluations. Conclusions Symptom checkers had deficits in both triage and diagnosis. Triage advice from symptom checkers is generally risk averse, encouraging users to seek care for conditions where self care is reasonable.
Subfields: Family Practice, Health Informatics, Health Information Management
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Family Practice
In Health Informatics
In Health Information Management
Citations in the corpus
- Improving the accuracy of medical diagnosis with causal machine learning (2020, methodology): “…real data sets such as EHRs, as the task of simulating a disease given its known properties is simpler than performing a differential diagnosis, and has been found to be effective for evaluating human doctors 73 – 76 and comparing the accuracy of doctors to symptom checker algorithms 17,23,77,78 .”
- Mobile technology and the digitization of healthcare. (2016, background): “Several concerns have been raised into the approval of technologies that have not included outcomes data, and we are learning that some health-related apps and devices may not work adequately in the real world.(62,63) Seminal observations have emerged into the high cost of care with mHealth in a uni”
- Sorting out symptoms: design and evaluation of the 'babylon check' automated triage system (2016, methodology): “As is common practice we grouped the outcomes for each vignette into three categories [2]: self-care (self-management / pharmacy), GP, and emergency care (A&E).”
- A comparative study of artificial intelligence and human doctors for the purpose of triage and diagnosis (2018, result): “Importantly, this evaluation assessed the accuracy and safety of both the AI and human doctors and, unlike previous studies, also takes in account the information gathering process of both agents [1, 2].”
- Context-Aware Symptom Checking for Disease Diagnosis Using Hierarchical Reinforcement Learning (2018, background): “According to a survey in 2012 (Semigran et al. 2015), 35% of U.S. adults in the U.S. have attempted to self-diagnose their ailments through online services This process often starts by searching a particular symptom on a search engine.”
- Towards conversational diagnostic artificial intelligence (2025, unknown): “Our study was significantly more challenging because it required the AI system to actively acquire relevant information through conversation, rather than relying on clinical information collated by human efforts 23 .”
- Inquire and Diagnose : Neural Symptom Checking Ensemble using Deep Reinforcement Learning (2016, background): “As indicated by [6], because of scarcity of information can be obtained from a user, accuracy achieved by existing symptom checkers is not high.”