Towards Expert-Level Medical Question Answering with Large Language Models
By K. Singhal, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, S. Pfohl, Heather J. Cole-Lewis, Darlene Neal, Mike Schaekermann and 20 more. 2023
799 citations recorded by the source corpus.
Recent artificial intelligence (AI) systems have reached milestones in"grand challenges"ranging from Go to protein-folding. The capability to retrieve medical knowledge, reason over it, and answer medical questions comparably to physicians has long been viewed as one such grand challenge. Large language models (LLMs) have catalyzed significant progress in medical question answering; Med-PaLM was the first model to exceed a"passing"score in US Medical Licensing Examination (USMLE) style questions with a score of 67.2% on the MedQA dataset. However, this and other prior work suggested significant room for improvement, especially when models' answers were compared to clinicians' answers. Here we present Med-PaLM 2, which bridges these gaps by leveraging a combination of base LLM improvements (PaLM 2), medical domain finetuning, and prompting strategies including a novel ensemble refinement approach. Med-PaLM 2 scored up to 86.5% on the MedQA dataset, improving upon Med-PaLM by over 19% and setting a new state-of-the-art. We also observed performance approaching or exceeding state-of-the-art across MedMCQA, PubMedQA, and MMLU clinical topics datasets. We performed detailed human evaluations on long-form questions along multiple axes relevant to clinical applications. In pairwise comparative ranking of 1066 consumer medical questions, physicians preferred Med-PaLM 2 answers to those produced by physicians on eight of nine axes pertaining to clinical utility (p<0.001). We also observed significant improvements compared to Med-PaLM on every evaluation axis (p<0.001) on newly introduced datasets of 240 long-form"adversarial"questions to probe LLM limitations. While further studies are necessary to validate the efficacy of these models in real-world settings, these results highlight rapid progress towards physician-level performance in medical question answering.
Subfields: Artificial Intelligence, Health Informatics, Large Language Models, Question Answering & Reading Comprehension, Medical & Health AI
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Mohit Bansal
- Yoshua Bengio
- Percy Liang
- Quoc V. Le
- Pieter Abbeel
- Jianfeng Gao
- Ruslan Salakhutdinov
In Health Informatics
In Large Language Models
- Luke S. Zettlemoyer
- Yejin Choi
- Ilya Sutskever
- Nitish Shirish Keskar
- Dacheng Tao
- Jason Phang
- Michael Janner
- Erik Cambria
In Question Answering & Reading Comprehension
In Medical & Health AI
Citations in the corpus
- GPT-4o System Card (2024, methodology): “This exceeds the performance of existing specialized medical models using few-shot prompting [43, 42], e.g., 84.0% for Med-Gemini-L 1.0 and 79.7% for Med-PaLM 2.”
- Leveraging Large Language Models for Mental Health Prediction via Online Text Data (2023, background): “[78] finetuned PaLM-2 on medical domains and achieved 86.”
- Large Language Models: A Survey (2024, neutral): “Med-PaLM 2 improves Med-PaLM via med-domain finetuning and ensemble prompting [77].”
- Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunities (2024, methodology): “Unlike previous ML algorithms, these large-scale models with a huge amount of parameters have shown versatile comprehension and reasoning capabilities in various fields such as health care [7], law [8], finance [9], education, and so on [10].”
- A Survey on Knowledge Distillation of Large Language Models (2024, background): “The integration of LLMs carries substantial promise in fundamentally reshaping the landscape of medical data analysis, comprehension, and smart medical services (Singhal et al., 2023; Yang et al., 2024b).”
- OCRBench: on the hidden mystery of OCR in large multimodal models (2023, neutral): “Additionally, the scarcity of training data for this specific task further complicates the recognition process for LMMs.”
- Empowering Biomedical Discovery with AI Agents (2024, neutral): “…results in clinical medicine question-answering suggest that assigning specific roles, such as clinicians, to GPT-4 [61] can achieve better performance in terms of accuracy on multiple-choice benchmarks compared to using domain-specialized LLMs like BioGPT [63], NYUTron [64], and Med-PaLM [65, 66].”
- Large language models in medicine (2023, background): “Autonomous deployment of LLM applications is not currently feasible, and clinicians will remain responsible for delivering optimal and humane care for their patients 14,16 .”
- MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning (2023, neutral): “3 3 Details about all guideline prompts and roles are shown in Section D for clarification.”
- Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Health (2023, methodology): “Development of LLMs has been steadily setting new state-of-the-art performance on a variety of tasks in general NLP as well as in biomedical NLP specifically [8, 26, 28-30].”
- Towards Generalist Biomedical AI (2023, result): “On medical question answering tasks, we compared against the SOTA Med-PaLM 2 results [61] and observed higher performance of Med-PaLM 2.”
- Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation (2023, background): “…their capabilities to generate content in various modalities, asserting their dominance in other domains like audio (Agostinelli et al., 2023), speech (Rubenstein et al., 2023), code generation (Li et al., 2023), medical applications (Singhal et al., 2023) and robotics (Zitkovich et al., 2023).”
- A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics (2023, methodology): “For instance, Med-PaLM 2 [16], a medical domain LLM, achieved a score of up to 86.5% on the USMLE dataset, outperforming Med-PaLM [99] by over 19% and setting a new state-of-the-art.”