Variability in Large Language Models’ Responses to Medical Licensing and Certification Examinations. Comment on “How Does ChatGPT Perform on the United States Medical Licensing Examination? The…

By R. Epstein, F. Dexter. JMIR Medical Education, 2023

367 citations recorded by the source corpus.

Subfields: Family Practice, Health Informatics, Public Health, Environmental and Occupational Health

View Paper: https://www.semanticscholar.org/paper/428057300776bc5a6ed215b351b8d517d484ef1e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Family Practice

1. Gunther Eysenbach
2. Abdulhadi Shoufan
3. S. Reddy

In Health Informatics

1. S. Reddy
2. Matei A. Zaharia
3. James Y. Zou
4. Jan Egger
5. Gunther Eysenbach
6. Abdulhadi Shoufan
7. Jianning Li
8. Amin Dada

In Public Health, Environmental and Occupational Health

1. Gunther Eysenbach
2. Abdulhadi Shoufan
3. S. Reddy

Citations in the corpus

• The Role of ChatGPT, Generative Language Models, and Artificial Intelligence in Medical Education: A Conversation With ChatGPT and a Call for Papers (2023, background): “A recent study [3] published in JMIR Medical Education on February 8, 2023, evaluated the potential of ChatGPT as a medical education tool.”
• A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics (2023, neutral): “Consequently, the reliability of ChatGPT in tasks necessitating faithfulness is called into question.”
• How is ChatGPT's behavior changing over time? (2023, background): “Recent works (Gilson et al., 2023; Kung et al., 2023; Mbakwe et al., 2023) also show that LLMs such as ChatGPT can pass the USMLE medical exam.”
• ChatGPT in Healthcare: A Taxonomy and Systematic Review (2023, methodology): “[36] further showed that ChatGPT outperformed two other language models, InstructGPT and GPT-3, in the exam.”
• Impact of the Implementation of ChatGPT in Education: A Systematic Review (2023, background): “In addition, ChatGPT’s knowledge is still limited and has not yet been updated with information beyond 2021 [27].”
• Improving large language models for clinical named entity recognition via prompt engineering (2024, neutral): “15 16–23 With those motivations, this study aims to investigate the potential of GPT models for clinical NER tasks.”
• Exploring Students’ Perceptions of ChatGPT: Thematic Analysis and Follow-Up Survey (2023, methodology): “In [26], the authors evaluated ChatGPT’s performance on four sets of multiple-choice questions from the United States Medical Licensing Examination (USMLE).”
• Generative AI in healthcare: an implementation science informed translational path on application, integration and governance (2024, background): “In some studies, these models have shown near-passing performance on medical exams, demonstrating a level of understanding comparable to that of a medical student [29].”