Large language models encode clinical knowledge
By K. Singhal, Shekoofeh Azizi, T. Tu, S. Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Kumar Tanwani, Heather J. Cole-Lewis, S. Pfohl and 20 more. 2022
4,703 citations recorded by the source corpus.
Med-PaLM, a state-of-the-art large language model for medicine, is introduced and evaluated across several medical question answering tasks, demonstrating the promise of these models in this domain. Large language models (LLMs) have demonstrated impressive capabilities, but the bar for clinical applications is high. Attempts to assess the clinical knowledge of models typically rely on automated evaluations based on limited benchmarks. Here, to address these limitations, we present MultiMedQA, a benchmark combining six existing medical question answering datasets spanning professional medicine, research and consumer queries and a new dataset of medical questions searched online, HealthSearchQA. We propose a human evaluation framework for model answers along multiple axes including factuality, comprehension, reasoning, possible harm and bias. In addition, we evaluate Pathways Language Model^ 1 (PaLM, a 540-billion parameter LLM) and its instruction-tuned variant, Flan-PaLM^ 2 on MultiMedQA. Using a combination of prompting strategies, Flan-PaLM achieves state-of-the-art accuracy on every MultiMedQA multiple-choice dataset (MedQA^ 3 , MedMCQA^ 4 , PubMedQA^ 5 and Measuring Massive Multitask Language Understanding (MMLU) clinical topics^ 6 ), including 67.6% accuracy on MedQA (US Medical Licensing Exam-style questions), surpassing the prior state of the art by more than 17%. However, human evaluation reveals key gaps. To resolve this, we introduce instruction prompt tuning, a parameter-efficient approach for aligning LLMs to new domains using a few exemplars. The resulting model, Med-PaLM, performs encouragingly, but remains inferior to clinicians. We show that comprehension, knowledge recall and reasoning improve with model scale and instruction prompt tuning, suggesting the potential utility of LLMs in medicine. Our human evaluations reveal limitations of today’s models, reinforcing the importance of both evaluation frameworks and method development in creating safe, helpful LLMs for clinical applications.
Subfields: Artificial Intelligence, Health Informatics, Large Language Models
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Percy Liang
- Sergey Levine
- Quoc V. Le
- Yoshua Bengio
- Luke S. Zettlemoyer
- Jianfeng Gao
- Ruslan Salakhutdinov
In Health Informatics
- Bertalan Meskó
- Stefan Harrer
- Joschka Haltaufderheide
- Robert Ranisch
- Pranav Rajpurkar
- Eric Horvitz
- Xuandong Zhao
- Zheng-Xin Yong
In Large Language Models
- Luke S. Zettlemoyer
- Yi Tay
- Yejin Choi
- Denny Zhou
- Kyunghyun Cho
- Quoc V. Le
- Sebastian Gehrmann
- Noah A. Smith
Citations in the corpus
- The Flan Collection: Designing Data and Methods for Effective Instruction Tuning (2023, background): “Further, Liu et al. (2022b), Vu et al. (2022), Wei et al. (2021), and Singhal et al. (2022) collectively show that both single-task and multi-task parameter-efficient tuning can be productively combined with instruction tuning, either before or after regular full-model instruction tuning.”
- Towards Expert-Level Medical Question Answering with Large Language Models (2023, positive): “In our prior work on Med-PaLM, we demonstrated the importance of a comprehensive benchmark for medical question-answering, human evaluation of model answers, and alignment strategies in the medical domain [1].”
- Understanding the Effects of Iterative Prompting on Truthfulness (2024, background): “This prowess has rendered them invaluable across diverse sectors, including finance, healthcare, and autonomous systems, revolutionizing conventional approaches to tasks in these domains [4–7].”
- Large language models in medicine (2023, background): “Performance is far from perfect even in medical student examinations, with no reported scores approaching 100% 14,15,38,43,44 .”
- Adapted large language models can outperform medical experts in clinical text summarization (2023, positive): “Previous work has demonstrated potential across clinical NLP tasks 22,23 , adapting to the medical domain by training a new model 24,25 , fine-tuning an existing model 26,27 or supplying task-specific examples in the model prompt 27,28 .”
- Large language models empowered agent-based modeling and simulation: a survey and perspectives (2023, positive): “Additionally, Med-PaLM (Singhal et al., 2023), an LLM for the medical fi eld, is trained based on a foundational model PaLM (Chowdhery et al., 2023).”
- Health system-scale language models are all-purpose prediction engines (2023, methodology): “…Nature Portfolio reporting summaries, source data, extended data, supplementary information, acknowl-edgements, peer review information; details of author contributions and competing interests; and statements of data and code availability are available at https://doi.org/10.1038/s41586-023-06160-y.”
- HealthBench: Evaluating Large Language Models Towards Improved Human Health (2025, positive): “More recent works have evaluated short, closed-form model responses (Kanjee et al., 2023; McDuff et al., 2025) and longer, open-ended model responses (Singhal et al., 2023; Fleming et al., 2024; Singhal et al., 2025; Tu et al., 2025).”
- A Survey on Evaluation of Large Language Models (2023, background): “, 2023c) Multilingual Multimodal Multilevel Specific downstream task Task-Specific Metrics MultiMedQA (Singhal et al., 2022) Medical Question Answering Specific downstream task Accuracy, Medical Knowledge, Reasoning SOCKET (Choi et al.”
- Towards accurate differential diagnosis with large language models (2023, neutral): “From the Health-SearchQA, LiveQA and MedicationQA datasets we used expert-crafted long-form responses to 64 questions.”
- Foundation models for generalist medical artificial intelligence (2023, background): “For example, by drawing on medical texts present within their training datasets, general-purpose models such as Flan-PaLM or ChatGPT can accurately answer medical questions, achieving passing scores on the United States Medical Licensing Exam 10,52,53 .”
- Benchmarking Retrieval-Augmented Generation for Medicine (2024, methodology): “As such, studies that evaluate open-ended QA in medicine are typically limited in scale, using only hundreds of questions (for example, Singhal et al. (2023a); Zakka et al. (2024)).”
- Knowledge Conflicts for LLMs: A Survey (2024, neutral): “While LLMs primarily acquire knowledge during pre-training (Zhou et al., 2023a; Kaddour et al., 2023; Naveed et al., 2023; Akyürek et al., 2022; Singhal et al., 2022), the vast and often unreliable nature of internet-sourced training data (Bender et al., 2021; Weidinger et al., 2021) can lead to…”