Can large language models reason about medical questions?

By Valentin Liévin, C. Hother, A. Motzfeldt, Ole Winther. Patterns, 2022

475 citations recorded by the source corpus.

Summary Although large language models often produce impressive outputs, it remains unclear how they perform in real-world scenarios requiring strong reasoning skills and expert domain knowledge. We set out to investigate whether closed- and open-source models (GPT-3.5, Llama 2, etc.) can be applied to answer and reason about difficult real-world-based questions. We focus on three popular medical benchmarks (MedQA-US Medical Licensing Examination [USMLE], MedMCQA, and PubMedQA) and multiple prompting scenarios: chain of thought (CoT; think step by step), few shot, and retrieval augmentation. Based on an expert annotation of the generated CoTs, we found that InstructGPT can often read, reason, and recall expert knowledge. Last, by leveraging advances in prompt engineering (few-shot and ensemble methods), we demonstrated that GPT-3.5 not only yields calibrated predictive distributions but also reaches the passing score on three datasets: MedQA-USMLE (60.2%), MedMCQA (62.7%), and PubMedQA (78.2%). Open-source models are closing the gap: Llama 2 70B also passed the MedQA-USMLE with 62.5% accuracy.

Subfields: Artificial Intelligence, Health Informatics

View Paper: https://www.semanticscholar.org/paper/d697b440dd0e65a05fe027e4c0ea85f62dcba033

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Toward expert-level medical question answering with large language models (2025, unknown): “In particular, these models have demonstrated impressive capabilities on multiple-choice research benchmarks 1–3 .”
A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics (2023, unknown): “As mentioned earlier, one notable difference between PLMs and LLMs is that PLMs are typically discriminative AI models, while LLMs are generative AI models.”
MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning (2023, background): “(ii) The demand for extensive domain knowledge (Schmidt and Rikers, 2007) and advanced reasoning skills (Liévin et al., 2022) makes eliciting medical expertise via simple prompting challenging (Kung et al., 2023; Singhal et al., 2023a).”
Evaluating the Performance of ChatGPT in Ophthalmology (2023, background): “There has been recent interest in evaluating the capabilities of LLMs for understanding and generating natural language in medicine 9 10 .”
Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Health (2023, background): “…Encoder Representations from Transformers) [17] and its variants, decoder-only LMs such as the GPT (Generative Pre-trained Transformer) family [18-20] and encoder-decoder LMs such as T5 (Text-to-Text Transfer Transformer) [21] and BART (Bidirectional and AutoRegressive Transformers) [22].”
Towards Expert-Level Medical Question Answering with Large Language Models (2023, unknown): “In particular, these models have demonstrated impressive capabilities on multiple-choice research benchmarks [1–3].”
Large language models encode clinical knowledge (2022, methodology): “[50] used Instruct GPT-3, an instruction-tuned LLM [63], and applied chain-of-thought prompting [91] on top to improve the results on the MedQA, MedMCQA, and PubMedQA datasets.”
Benchmarking Retrieval-Augmented Generation for Medicine (2024, methodology): “Evaluating medical QA systems using multi-choice questions is a widely adopted method that can be practically implemented for large-scale evaluation (Nori et al., 2023a,b; Singhal et al., 2023a; Liévin et al., 2022; Lála et al., 2023).”
A Categorical Archive of ChatGPT Failures (2023, unknown): “riddles, poetry, IQ, emotions [37, 39], medicine [46, 42], reading comprehension, etc.”
BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains (2024, methodology): “In the case of PubMedQA, the prompt includes a context before the question and the three answer option: yes , no , or maybe are formulated as a multiple choice question, where A is yes, B is no, and C is maybe, matching the testing method done by Liévin et al. (2023).”
A Survey on Evaluation of Large Language Models (2023, background): “In professional domain reasoning tasks, zeroshot InstructGPT and Codex are capable of complex medical reasoning tasks, but still need to be further improved (Liévin et al., 2022).”