Assessing the Accuracy and Reliability of AI-Generated Medical Responses: An Evaluation of the Chat-GPT Model
By Douglas B. Johnson, Rachel S Goodman, J. Patrinely, Cosby A. Stone, Eli Zimmerman, R. Donald, Sam S Chang, Sean T. Berkowitz, Avni P Finn, Eiman Jahangir and 24 more. Research Square, 2023
498 citations recorded by the source corpus.
Background: Natural language processing models such as ChatGPT can generate text-based content and are poised to become a major information source in medicine and beyond. The accuracy and completeness of ChatGPT for medical queries is not known. Methods: Thirty-three physicians across 17 specialties generated 284 medical questions that they subjectively classified as easy, medium, or hard with either binary (yes/no) or descriptive answers. The physicians then graded ChatGPT-generated answers to these questions for accuracy (6-point Likert scale; range 1 – completely incorrect to 6 – completely correct) and completeness (3-point Likert scale; range 1 – incomplete to 3 - complete plus additional context). Scores were summarized with descriptive statistics and compared using Mann-Whitney U or Kruskal-Wallis testing. Results: Across all questions (n=284), median accuracy score was 5.5 (between almost completely and completely correct) with mean score of 4.8 (between mostly and almost completely correct). Median completeness score was 3 (complete and comprehensive) with mean score of 2.5. For questions rated easy, medium, and hard, median accuracy scores were 6, 5.5, and 5 (mean 5.0, 4.7, and 4.6; p=0.05). Accuracy scores for binary and descriptive questions were similar (median 6 vs. 5; mean 4.9 vs. 4.7; p=0.07). Of 36 questions with scores of 1-2, 34 were re-queried/re-graded 8-17 days later with substantial improvement (median 2 vs. 4; p<0.01). Conclusions: ChatGPT generated largely accurate information to diverse medical queries as judged by academic physician specialists although with important limitations. Further research and model development are needed to correct inaccuracies and for validation.
Subfields: Artificial Intelligence, Family Practice, Health Informatics
View Paper: https://www.semanticscholar.org/paper/67655143fdd9dafe018870e1915149111c8c81b6
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Health Informatics
- Jan Egger
- Jianning Li
- Amin Dada
- Jens Kleesiek
- Behrus Puladi
- Mohamad Forouzanfar
- Shiva Maleki Varnosfaderani
Citations in the corpus
- A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics (2023, methodology): “Consequently, ethic issues related to data privacy and protection [324], [333] must be thoroughly considered in the evaluation”
- ChatGPT in Healthcare: A Taxonomy and Systematic Review (2023, background): “[52] consultation medical education ChatGPT answers 284 question from various medical specialities ChatGPT achieved overall high accuracies preprint (1)7”
- The Role of AI in Hospitals and Clinics: Transforming Healthcare in the 21st Century (2024, background): “AI in radiology is not only about detecting abnormalities; it also helps in quantifying disease progression [96], assessing response to treatment [97], and predicting patient outcomes [98].”
- A Survey on Evaluation of Large Language Models (2023, background): “, 2023), and many other QA situations (Hamidi and Roberts, 2023; Johnson et al., 2023).”