How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment
By Aidan Gilson, C. Safranek, Thomas Huang, V. Socrates, Ling Chi, R. Taylor, David Chartash. JMIR Medical Education, 2023
1,716 citations recorded by the source corpus.
Background Chat Generative Pre-trained Transformer (ChatGPT) is a 175-billion-parameter natural language processing model that can generate conversation-style responses to user input. Objective This study aimed to evaluate the performance of ChatGPT on questions within the scope of the United States Medical Licensing Examination (USMLE) Step 1 and Step 2 exams, as well as to analyze responses for user interpretability. Methods We used 2 sets of multiple-choice questions to evaluate ChatGPT’s performance, each with questions pertaining to Step 1 and Step 2. The first set was derived from AMBOSS, a commonly used question bank for medical students, which also provides statistics on question difficulty and the performance on an exam relative to the user base. The second set was the National Board of Medical Examiners (NBME) free 120 questions. ChatGPT’s performance was compared to 2 other large language models, GPT-3 and InstructGPT. The text output of each ChatGPT response was evaluated across 3 qualitative metrics: logical justification of the answer selected, presence of information internal to the question, and presence of information external to the question. Results Of the 4 data sets, AMBOSS-Step1, AMBOSS-Step2, NBME-Free-Step1, and NBME-Free-Step2, ChatGPT achieved accuracies of 44% (44/100), 42% (42/100), 64.4% (56/87), and 57.8% (59/102), respectively. ChatGPT outperformed InstructGPT by 8.15% on average across all data sets, and GPT-3 performed similarly to random chance. The model demonstrated a significant decrease in performance as question difficulty increased (P=.01) within the AMBOSS-Step1 data set. We found that logical justification for ChatGPT’s answer selection was present in 100% of outputs of the NBME data sets. Internal information to the question was present in 96.8% (183/189) of all questions. The presence of information external to the question was 44.5% and 27% lower for incorrect answers relative to correct answers on the NBME-Free-Step1 (P<.001) and NBME-Free-Step2 (P=.001) data sets, respectively. Conclusions ChatGPT marks a significant improvement in natural language processing models on the tasks of medical question answering. By performing at a greater than 60% threshold on the NBME-Free-Step-1 data set, we show that the model achieves the equivalent of a passing score for a third-year medical student. Additionally, we highlight ChatGPT’s capacity to provide logic and informational context across the majority of answers. These facts taken together make a compelling case for the potential applications of ChatGPT as an interactive medical education tool to support learning.
Subfields: Artificial Intelligence, Health Informatics, Safety Research
View Paper: https://www.semanticscholar.org/paper/cd4d112f3f9120d0715f22a9de2ce4720822368c
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Lixiang Yan
- C. Preiksaitis
- Chi-Liang Liu
- Alaa A. Abd-alrazaq
- Wei Dang Lu
- Gero Strobel
- Kai Zhang
- Jindong Wang
In Health Informatics
- C. Preiksaitis
- Christian Rose
- Gero Strobel
- Bernd Carsten Stahl
- Leonardo Banh
- Damian Okaibedi Eke
- Gunther Eysenbach
- Malik Sallam
In Safety Research
- Gero Strobel
- Bernd Carsten Stahl
- Lixiang Yan
- Leonardo Banh
- Damian Okaibedi Eke
- Malik Sallam
- Dragan Gasevic
Citations in the corpus
- The ethics of ChatGPT - Exploring the ethical issues of an emerging technology (2024, background): “…difficult to identify as the output of an AI (Zhang et al., 2023); • Ability to engage in a dialogical interaction on a very broad array of topics (Gilson et al., 2023); • Ability to tailor its output to specific language styles (Short & Short, 2023); • While currently text based, we assume that…”
- HealthBench: Evaluating Large Language Models Towards Improved Human Health (2025, neutral): “Some works have focused on specialized medical models (Luo et al., 2022; Singhal et al., 2023; Li et al., 2023; Singhal et al., 2025; Saab et al., 2024), and others on general models (Nori et al., 2023; Gilson et al., 2023; Takagi et al., 2023).”
- Investigating the Impact of User Trust on the Adoption and Use of ChatGPT: Survey Analysis (2023, neutral): “This implies that health providers can leverage chatbots to provide health information and support to patients [8,51,52].”
- Generative artificial intelligence (2023, background): “Early studies have shown how the chatbot ChatGPT performed surprisingly well in medical exams, suggesting inherent knowledge similar to medical students (Bhayana et al., 2023; Gilson et al., 2023).”
- Practical and ethical challenges of large language models in education: A systematic scoping review (2023, background): “, 2023) and evaluating these LLMs’ capability to tackle educational assessments (Gilson et al., 2023).”
- Large language models in health care: Development, applications, and challenges (2023, background): “The interactive interface of LLMs can complement existing student ‐ directed learning, where Socratic style of teaching has been surveyed as preferable by students over didactic lectures [20, 43, 44].”
- HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledge (2023, background): “While the generated instructions are required to be diverse enough for unseen tasks (Wang et al., 2022) in the general domain, the correctness of the fact in the responses from the large language model is of more concern in the biomedical domain (Gilson et al., 2023).”
- ChatGPT and large language models in academia: opportunities and challenges (2023, neutral): “In higher education, LLM-based chatbots may be used to increase student engagement, facilitate group activities, create interactive learning tools, and provide immediate feedback and assessment [18–20].”
- Opportunities, Challenges, and Future Directions of Generative Artificial Intelligence in Medical Education: Scoping Review (2023, background): “Only a handful of these studies delved into the potential implications of generative AI's performance on these tests [8,33].”
- A Survey on Evaluation of Large Language Models (2023, methodology): “, 2023) X (Duong and Solomon, 2023) X (Gilson et al., 2023) X (Holmes et al.”