Toward expert-level medical question answering with large language models
By Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R. Pfohl, Heather Cole-Lewis and 25 more. Nature Medicine, 2025
1,038 citations recorded by the source corpus.
Large language models (LLMs) have shown promise in medical question answering, with Med-PaLM being the first to exceed a ‘passing’ score in United States Medical Licensing Examination style questions. However, challenges remain in long-form medical question answering and handling real-world workflows. Here, we present Med-PaLM 2, which bridges these gaps with a combination of base LLM improvements, medical domain fine-tuning and new strategies for improving reasoning and grounding through ensemble refinement and chain of retrieval. Med-PaLM 2 scores up to 86.5% on the MedQA dataset, improving upon Med-PaLM by over 19%, and demonstrates dramatic performance increases across MedMCQA, PubMedQA and MMLU clinical topics datasets. Our detailed human evaluations framework shows that physicians prefer Med-PaLM 2 answers to those from other physicians on eight of nine clinical axes. Med-PaLM 2 also demonstrates significant improvements over its predecessor across all evaluation metrics, particularly on new adversarial datasets designed to probe LLM limitations (P < 0.001). In a pilot study using real-world medical questions, specialists preferred Med-PaLM 2 answers to generalist physician answers 65% of the time. While specialist answers were still preferred overall, both specialists and generalists rated Med-PaLM 2 to be as safe as physician answers, demonstrating its growing potential in real-world medical applications. With an improved framework for model development and evaluation, a large language model is shown to provide answers to medical questions that are comparable or preferred with respect to those provided by human physicians.
Subfields: Artificial Intelligence
View Paper: https://www.semanticscholar.org/paper/b34c0b0169925a6c7f14e3de9764ed9505f30de3
Citations in the corpus
- HealthBench: Evaluating Large Language Models Towards Improved Human Health (2025, neutral): “Some works have focused on specialized medical models (Luo et al., 2022; Singhal et al., 2023; Li et al., 2023; Singhal et al., 2025; Saab et al., 2024), and others on general models (Nori et al., 2023; Gilson et al., 2023; Takagi et al., 2023).”
- Towards conversational diagnostic artificial intelligence (2025, neutral): “We also curated 191 MedQA questions from the training set where clinical experts had crafted step-by-step reasoning leading to the correct answer 13 .”
- Traj-CoA: Patient Trajectory Modeling via Chain-of-Agents for Lung Cancer Risk Prediction (2025, unknown): “While traditional approaches required complex feature engineering and task-specific models [4, 5, 6], modern Large Language Models (LLMs) promise a more generalizable, zero-shot paradigm for clinical prediction [7, 8, 9].”
- Data Kernel Perspective Space Performance Guarantees for Synthetic Data from Transformer Models (2026, unknown): “Transformer models have quickly achieved state-of-the-art results even on tasks of seemingly high cognitive complexity (e.g. question answering [13], [14], [15]).”