Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models
By Tiffany H. Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo and 1 more. medRxiv, 2022
3,505 citations recorded by the source corpus.
We evaluated the performance of a large language model called ChatGPT on the United States Medical Licensing Exam (USMLE), which consists of three exams: Step 1, Step 2CK, and Step 3. ChatGPT performed at or near the passing threshold for all three exams without any specialized training or reinforcement. Additionally, ChatGPT demonstrated a high level of concordance and insight in its explanations. These results suggest that large language models may have the potential to assist with medical education, and potentially, even clinical decision-making.
Subfields: Artificial Intelligence, General Health Professions, Health Informatics
View Paper: https://www.semanticscholar.org/paper/cf1f26e7cbed3958b3c2870656568c299fece6e3
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Kang Liu
- Zhiyuan Liu
- Thien Huu Nguyen
- Nikolaos D. Tselikas
- Konstantinos I. Roumeliotis
- Matei A. Zaharia
- Maosong Sun
- Shizhu He
In General Health Professions
- C. Preiksaitis
- Eric Horvitz
- Michael Agyemang Adarkwah
- Thien Huu Nguyen
- Christian Rose
- Boulus Shehata
- Ronghuai Huang
- Ahmed Tlili
In Health Informatics
- Nikolaos D. Tselikas
- Konstantinos I. Roumeliotis
- Matei A. Zaharia
- C. Preiksaitis
- H. Alkaissi
- James Y. Zou
- Eric Horvitz
- Jan Egger
Citations in the corpus
- Large language models propagate race-based medicine (2023, background): “While studies have assessed the applications of LLMs for answering medical questions 2,15 , much work remains to understand the pitfalls of these models in providing support to healthcare practitioners.”
- MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning (2023, neutral): “Although numerous attempts, particularly within math and coding, have been made to enhance prompting methods (Besta et al., 2023; Lála et al., 2023; Wang et al., 2023b), strategies used in the medical field have been shown to induce hallucination (Umapathi et al., 2023; Harris, 2023; Ji et al.,…”
- PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering (2023, background): “Notably, even in domains where expert knowledge is supposed to play a critical role, like medical diagnosis, these language models have also achieved impressive success, passing the United States Medical Licensing Examination (USMLE) [13, 17, 26, 32].”
- ChatGPT and large language models in academia: opportunities and challenges (2023, background): “In one case, ChatGPT was listed as an author in the preprint [9] but missing as an author in the final publication [4].”
- Comparing scientific abstracts generated by ChatGPT to real abstracts with detectors and blinded human reviewers (2023, neutral): “Articles are bemoaning the death of the traditional school essay assignment 4,6,7 , as ChatGPT has been shown to generate high-scoring papers 8 , correctly answer USMLE questions 9 , and even articulate critical thinking 10 .”
- How is ChatGPT's behavior changing over time? (2023, background): “This effectively means that the June versions corrected previous errors but also made additional mistakes. responses to USMLE questions became longer. s The task is to create a output grid corresponding to an input grid, based solely on a few similar examples.”
- The future landscape of large language models in medicine (2023, methodology): “Nonetheless, using LLMs for scienti fi c writing currently requires signi fi cant revisions by human authors due to inaccurate, shallow and repetitive outputs (Supplementary Data, example 6).”
- Comparing ChatGPT and GPT-4 performance in USMLE soft skill assessments (2023, neutral): “Another study that tested ChatGPT on the USMLE example test questions reported accuracies up to 55.8%, 59.1% and 61.3% for Step1, 2CK and 3, respectively 15 .”
- Artificial Hallucinations in ChatGPT: Implications in Scientific Writing (2023, background): “High internal concordance was also noticed, especially in correctly answering questions, with a concordance rate of up to 99% [3].”
- What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education (2023, background): “For instance, Kung et al. (2023) found the accuracy of ChatGPT to be around 60%, demanding careful assessment of its output before use.”
- BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset (2023, background): “The recent advent of large language models (LLMs) [1, 2, 3, 4, 5] promises vast transformative potential across multiple sectors, from healthcare [6, 7, 8] and education [9, 10, 11] to robotics [12, 13, 14] and commerce [15].”
- ChatGPT outperforms humans in emotional awareness evaluations (2023, background): “Although preliminary studies have revealed that ChatGPT can successfully generate information it has been trained on in multiple fields such as medical licensing examinations (Kung et al., 2023) and academic writing (Rudolph et al.”
- Performance of ChatGPT, GPT-4, and Google Bard on a Neurosurgery Oral Boards Preparation Question Bank (2023, background): “8 The goals of this study were to (1) assess the performance of 3 LLMs on a question bank with higher-order questions more representative of oral board topics and (2) elucidate differences in accuracy and performance by question characteristics between LLMs.”
- Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine (2023, neutral): “In [17], ChatGPT-3.”
- Investigating the Impact of User Trust on the Adoption and Use of ChatGPT: Survey Analysis (2023, neutral): “Many users appreciate the convenience and efficiency of this technology, particularly in various applications such as chatbots, virtual assistants, and customer service agents [11-14].”