Capabilities of GPT-4 on Medical Challenge Problems

By Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, Eric Horvitz. arXiv.org, 2023

1,324 citations recorded by the source corpus.

Large language models (LLMs) have demonstrated remarkable capabilities in natural language understanding and generation across various domains, including medicine. We present a comprehensive evaluation of GPT-4, a state-of-the-art LLM, on medical competency examinations and benchmark datasets. GPT-4 is a general-purpose model that is not specialized for medical problems through training or engineered to solve clinical tasks. Our analysis covers two sets of official practice materials for the USMLE, a three-step examination program used to assess clinical competency and grant licensure in the United States. We also evaluate performance on the MultiMedQA suite of benchmark datasets. Beyond measuring model performance, experiments were conducted to investigate the influence of test questions containing both text and images on model performance, probe for memorization of content during training, and study probability calibration, which is of critical importance in high-stakes applications like medicine. Our results show that GPT-4, without any specialized prompt crafting, exceeds the passing score on USMLE by over 20 points and outperforms earlier general-purpose models (GPT-3.5) as well as models specifically fine-tuned on medical knowledge (Med-PaLM, a prompt-tuned version of Flan-PaLM 540B). In addition, GPT-4 is significantly better calibrated than GPT-3.5, demonstrating a much-improved ability to predict the likelihood that its answers are correct. We also explore the behavior of the model qualitatively through a case study that shows the ability of GPT-4 to explain medical reasoning, personalize explanations to students, and interactively craft new counterfactual scenarios around a medical case. Implications of the findings are discussed for potential uses of GPT-4 in medical education, assessment, and clinical practice, with appropriate attention to challenges of accuracy and safety.

Subfields: Artificial Intelligence, Family Practice, Health Informatics

View Paper: https://www.semanticscholar.org/paper/348a1efa54376fa39053e5e25d52bd0eb6a0ba68

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Informatics

Harsha Nori

Citations in the corpus

Benchmarking Retrieval-Augmented Generation for Medicine (2024, methodology): “Evaluating medical QA systems using multi-choice questions is a widely adopted method that can be practically implemented for large-scale evaluation (Nori et al., 2023a,b; Singhal et al., 2023a; Liévin et al., 2022; Lála et al., 2023).”
MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts (2023, background): “…abilities to respond to the queries from various disciplines, grounded in text, such as QA, math, medicine, coding and science (Bubeck et al., 2023; Nori et al., 2023; Chen et al., 2021; Fu et al., 2023; Sun et al., 2023; Wang et al., 2023; Huang et al., 2023, 2022; Liu et al., 2023b On the…”
CodeS: Towards Building Open-source Language Models for Text-to-SQL (2024, background): “Therefore, we follow the general assumption that ChatGPT is approximately equivalent in size to GPT-3-175B [32] and GPT-4 significantly exceeds the size of ChatGPT [47, 48]. data (4.5GB), with the aim of enhancing both SQL generation capabilities and natural language comprehension.”
Towards accurate differential diagnosis with large language models (2023, background): “Both general-purpose LLMs (GPT-4) and medical domain-specialized LLMs (Med-PaLM 2) have demonstrated strong performance in standardized and multiple-choice medical benchmarks [10, 11].”
Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks (2023, background): “They have been shown to pass professional exams (Kung et al., 2023; Nori et al., 2023; Terwi-esch, 2023; i.a. ), exceed state-of-the-art methods on many traditional benchmarks (Sun et al., 2023; Sobania et al., 2023; Zhang et al., 2023a; Dhingra et al., 2023; i.a. ), and surpass human performance…”
Towards Expert-Level Medical Question Answering with Large Language Models (2023, neutral): “We believe this methodology is both simple and conservative, and when possible we recommend it over blackbox memorization testing techniques [2], which do not conclusively measure test set contamination.”
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (2023, background): “, 2022b), and more recent work can be divided into two main categories: intrinsic estimation, which uses language model probability (Si et al., 2023; Nori et al., 2023) and sampling (Kuhn et al.”
HealthBench: Evaluating Large Language Models Towards Improved Human Health (2025, neutral): “In particular, large language models (LLMs) can encode vast clinical knowledge, reason over complex inputs, and adapt flexibly to user needs (Singhal et al., 2023, 2025; Nori et al., 2023).”
Large language models in medicine (2023, background): “Performance is far from perfect even in medical student examinations, with no reported scores approaching 100% 14,15,38,43,44 .”
Toward expert-level medical question answering with large language models (2025, neutral): “We searched for overlapping text segments between multiple-choice questions in MultiMedQA and the corpus used to train the base LLM underlying Med-PaLM 2.”
ChatGPT: The End of Online Exam Integrity? (2022, neutral): “Indeed, studies [22,31,38,47,48] have identified this as a major obstacle in using LLMs for producing correct and reliable responses in various critical contexts, and this also translates to question-answering tasks within high-stakes online exams.”
GPT-4o System Card (2024, background): “In recent years, large language models have shown significant promise in biomedical settings, both in academic evaluations [39, 40, 41, 42, 43] and real-world use-cases such as clinical documentation [44, 45], patient messaging [46, 47], clinical trial recruitment [48, 49], and clinical decision…”
LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day (2023, background): “Consequently, although conversational generative AI has demonstrated great potential for biomedical applications [20, 33, 19], current investigations are often limited to unimodal text.”
PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering (2023, background): “Notably, even in domains where expert knowledge is supposed to play a critical role, like medical diagnosis, these language models have also achieved impressive success, passing the United States Medical Licensing Examination (USMLE) [13, 17, 26, 32].”