A comparative study of artificial intelligence and human doctors for the purpose of triage and diagnosis
By S. Razzaki, Adam Baker, Yura N. Perov, Katherine Middleton, J. Baxter, D. Mullarkey, Davinder Sangar, Michael Taliercio, Mobasher Butt, Azeem Majeed and 3 more. 2018
96 citations recorded by the source corpus.
Online symptom checkers have significant potential to improve patient care, however their reliability and accuracy remain variable. We hypothesised that an artificial intelligence (AI) powered triage and diagnostic system would compare favourably with human doctors with respect to triage and diagnostic accuracy. We performed a prospective validation study of the accuracy and safety of an AI powered triage and diagnostic system. Identical cases were evaluated by both an AI system and human doctors. Differential diagnoses and triage outcomes were evaluated by an independent judge, who was blinded from knowing the source (AI system or human doctor) of the outcomes. Independently of these cases, vignettes from publicly available resources were also assessed to provide a benchmark to previous studies and the diagnostic component of the MRCGP exam. Overall we found that the Babylon AI powered Triage and Diagnostic System was able to identify the condition modelled by a clinical vignette with accuracy comparable to human doctors (in terms of precision and recall). In addition, we found that the triage advice recommended by the AI System was, on average, safer than that of human doctors, when compared to the ranges of acceptable triage provided by independent expert judges, with only a minimal reduction in appropriateness.
Subfields: Artificial Intelligence, Family Practice, Health Informatics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Family Practice
In Health Informatics
Citations in the corpus
- Improving the accuracy of medical diagnosis with causal machine learning (2020, methodology): “…real data sets such as EHRs, as the task of simulating a disease given its known properties is simpler than performing a differential diagnosis, and has been found to be effective for evaluating human doctors 73 – 76 and comparing the accuracy of doctors to symptom checker algorithms 17,23,77,78 .”
- Acceptability of artificial intelligence (AI)-led chatbot services in healthcare: A mixed-methods study (2019, background): “Other systems have also been proposed to act as a symptom checker, 14 online triage service 15 or health promotion assistant, 16 providing live feedback in an interactive way.”
- Shifting machine learning for healthcare from development to deployment and from models to data (2022, unknown): “…have also been used to improve clinical questioning and answering 141 , in clinical voice assistants 142,143 , in chatbots for patient triage 144,145 , and in medical-image-to-text translation and medical-image captioning 146 . structured data (such as laboratory values, or codes from the…”
- Conversational Agents in Health Care: Scoping Review and Conceptual Analysis (2020, methodology): “A variety of study designs were used in the included studies, comprising 20 case studies [44,48,51,61-63,66,69,71, 73-79,82,84,85,89], 4 surveys [55,56,59,65], 3 observational studies [53,86,87], 11 randomized controlled trials [46,49,50,57,64,67,72,80,81,83,88], 3 diagnostic accuracy studies [58,60”
- Physicians’ Perceptions of Chatbots in Health Care: Cross-Sectional Web-Based Survey (2018, unknown): “Early research has demonstrated the benefits of using health care chatbots, such as helping with diagnostic decision support [15,16], promoting and increasing physical activity [17], and cognitive behavioral therapy for psychiatric and somatic disorders [18-24], which provide effective, acceptable, ”
- An overview of clinical decision support systems: benefits, risks, and strategies for success (2020, unknown): “53,54 Diagnostics support: imaging .”