Diagnostic Accuracy of Differential-Diagnosis Lists Generated by Generative Pretrained Transformer 3 Chatbot for Clinical Vignettes with Common Chief Complaints: A Pilot Study
By Takanobu Hirosawa, Y. Harada, Masashi Yokose, Tetsu Sakamoto, Ren Kawamura, Taro Shimizu. International Journal of Environmental Research and Public Health, 2023
301 citations recorded by the source corpus.
The diagnostic accuracy of differential diagnoses generated by artificial intelligence (AI) chatbots, including the generative pretrained transformer 3 (GPT-3) chatbot (ChatGPT-3) is unknown. This study evaluated the accuracy of differential-diagnosis lists generated by ChatGPT-3 for clinical vignettes with common chief complaints. General internal medicine physicians created clinical cases, correct diagnoses, and five differential diagnoses for ten common chief complaints. The rate of correct diagnosis by ChatGPT-3 within the ten differential-diagnosis lists was 28/30 (93.3%). The rate of correct diagnosis by physicians was still superior to that by ChatGPT-3 within the five differential-diagnosis lists (98.3% vs. 83.3%, p = 0.03). The rate of correct diagnosis by physicians was also superior to that by ChatGPT-3 in the top diagnosis (53.3% vs. 93.3%, p < 0.001). The rate of consistent differential diagnoses among physicians within the ten differential-diagnosis lists generated by ChatGPT-3 was 62/88 (70.5%). In summary, this study demonstrates the high diagnostic accuracy of differential-diagnosis lists generated by ChatGPT-3 for clinical cases with common chief complaints. This suggests that AI chatbots such as ChatGPT-3 can generate a well-differentiated diagnosis list for common chief complaints. However, the order of these lists can be improved in the future.
Subfields: Artificial Intelligence, Health Informatics
View Paper: https://www.semanticscholar.org/paper/b026af1b2436fe064e02ca7a00b79713427bc3b2
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Health Informatics
- Jan Egger
- Nikolaos D. Tselikas
- Jianning Li
- Konstantinos I. Roumeliotis
- Amin Dada
- Jens Kleesiek
- Behrus Puladi
Citations in the corpus
- A study of generative large language model for medical research and healthcare (2023, unknown): “…about LLMs in the potential to facilitate documentation of patient reports (e.g., a progress report) 3,4 , improving diagnostic accuracy 5 , and assisting in various clinical care 6,7 , while at the same time concerning the hallucinations and fabrications 7,8 , bias and stereotype 9 ,…”
- Utility of ChatGPT in Clinical Practice (2023, unknown): “Current applications of ChatGPT in CDS include the following: • Differential-diagnosis lists: Hirosawa et al [5] evaluated ChatGPT-3 and general internal medicine physicians to generate clinical cases, correct diagnoses, and five differential diagnoses for 10 common chief complaints.”
- ChatGPT and Open-AI Models: A Preliminary Review (2023, background): “Reasoning/Critical Thinking Healthcare/Diagnosis ChatGPT provided correct diagnoses with more than 90% accuracy [40].”
- ChatGPT in Healthcare: A Taxonomy and Systematic Review (2023, background): “Other representative exclusions include [44, 55], which deal with CPT-3, and [56, 30, 90, 2], where the authors claimed that ChatGPT assisted with the writing of the papers or case reports, but did not provide any discussion of the appropriateness of the generated texts and how the texts were…”