Adapted large language models can outperform medical experts in clinical text summarization

By Dave Van Veen, Cara Van Uden, L. Blankemeier, Jean-Benoit Delbrouck, Asad Aali, Christian Blüthgen, Anuj Pareek, Malgorzata Polacin, William Collins, Neera Ahuja and 5 more. Nature Medicine, 2023

793 citations recorded by the source corpus.

Analyzing vast textual data and summarizing key information from electronic health records imposes a substantial burden on how clinicians allocate their time. Although large language models (LLMs) have shown promise in natural language processing (NLP) tasks, their effectiveness on a diverse range of clinical summarization tasks remains unproven. Here we applied adaptation methods to eight LLMs, spanning four distinct clinical summarization tasks: radiology reports, patient questions, progress notes and doctor–patient dialogue. Quantitative assessments with syntactic, semantic and conceptual NLP metrics reveal trade-offs between models and adaptation methods. A clinical reader study with 10 physicians evaluated summary completeness, correctness and conciseness; in most cases, summaries from our best-adapted LLMs were deemed either equivalent (45%) or superior (36%) compared with summaries from medical experts. The ensuing safety analysis highlights challenges faced by both LLMs and medical experts, as we connect errors to potential medical harm and categorize types of fabricated information. Our research provides evidence of LLMs outperforming medical experts in clinical text summarization across multiple tasks. This suggests that integrating LLMs into clinical workflows could alleviate documentation burden, allowing clinicians to focus more on patient care. Comparative performance assessment of large language models identified ChatGPT-4 as the best-adapted model across a diverse set of clinical text summarization tasks, and it outperformed 10 medical experts in a reader study.

Subfields: Artificial Intelligence, Molecular Biology

View Paper: https://www.semanticscholar.org/paper/007c3d9b8dab341d2c77c4ee764fd921f7f14956

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

Citations in the corpus

A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation (2025, positive): “In fact, the use of LLMs for clinical documentation, especially clinical note generation 4 or consultation summarisation 5,6 , is an active area of research.”
A generalist vision–language foundation model for diverse biomedical tasks (2023, neutral): “Although metrics such as CIDEr and ROUGE-L can measure the agreement between generated content and a gold standard, and are commonly used for model selection to further assess clinical applicability 35 , ensuring the factual accuracy of these outputs remains a concern.”
Evaluation and mitigation of the limitations of large language models in clinical decision-making (2024, neutral): “LLMs can summarize reports 3–5 , generate reports 2,4 , serve as diagnostic assistants 21,27 and could ultimately autonomously diagnose patients.”
GPT-4o System Card (2024, background): “…large language models have shown significant promise in biomedical settings, both in academic evaluations [39, 40, 41, 42, 43] and real-world use-cases such as clinical documentation [44, 45], patient messaging [46, 47], clinical trial recruitment [48, 49], and clinical decision support [50, 51].”