Evaluating the Performance of ChatGPT in Ophthalmology

By F. Antaki, Samir Touma, D. Milad, J. El-Khoury, R. Duval. medRxiv, 2023

475 citations recorded by the source corpus.

We tested the accuracy of ChatGPT, a large language model (LLM), in the ophthalmology question-answering space using two popular multiple choice question banks used for the high-stakes Ophthalmic Knowledge Assessment Program (OKAP) exam. The testing sets were of easy-to-moderate difficulty and were diversified, including recall, interpretation, practical and clinical decision-making problems. ChatGPT achieved 55.8% and 42.7% accuracy in the two 260-question simulated exams. Its performance varied across subspecialties, with the best results in general medicine and the worst in neuro-ophthalmology and ophthalmic pathology and intraocular tumors. These results are encouraging but suggest that specialising LLMs through domain-specific pre-training may be necessary to improve their performance in ophthalmic subspecialties.

Subfields: Family Practice, Health Informatics, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/35cdc00a4e2bc1f6c253a34a5e3a6f697050d1e9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Family Practice

1. C. Preiksaitis
2. Christian Rose

In Health Informatics

1. C. Preiksaitis
2. Joschka Haltaufderheide
3. Christian Rose
4. Robert Ranisch

In Radiology, Nuclear Medicine and Imaging

1. C. Preiksaitis
2. Joschka Haltaufderheide
3. Christian Rose
4. Robert Ranisch

Citations in the corpus

• Towards Expert-Level Medical Question Answering with Large Language Models (2023, unknown): “On a similar note, Duong & Solomon [23], Oh et al. [24], and Antaki et al. [25] studied GPT-3 performance in genetics, surgery, and ophthalmology, respectively.”
• Uncovering ChatGPT’s Capabilities in Recommender Systems (2023, background): “These LLMs possess a versatile nature and extensive world knowledge, allowing them to be applied not only in various NLP tasks [3, 5, 16], but also in domains such as education [11, 19, 22], medicine [1, 2, 29], search [20, 26, 37] and law [4, 8, 38].”
• Opportunities, Challenges, and Future Directions of Generative Artificial Intelligence in Medical Education: Scoping Review (2023, background): “A significant issue lies in data accuracy, with many authors drawing attention to this technology’s propensity to “hallucinate,” or create false information, and its knowledge being confined to the training data set [1,10,21,22,25,27,28,34,37,44].”
• Prompt engineering in consistency and reliability with the evidence-based guideline for LLMs (2024, unknown): “The study of Fares et al. 30 accessed GPT-4 via the API and set different temperatures (temperature = 0, 0.3, 0.7, 1) and found that the model set at a temperature of 0.3 performed better in answering ophthalmology-related questions.”
• The ethics of ChatGPT in medicine and healthcare: a systematic review on Large Language Models (LLMs) (2024, methodology): “With this, we aspire to not only to outline the current ethical discourse but also to inform future dialogues and policy-making in the intersection of LLMs and healthcare ethics.”