The shaky foundations of large language models and foundation models for electronic health records

By Michael Wornow, Yizhe Xu, Rahul Thapa, Birju S. Patel, E. Steinberg, S. Fleming, M. Pfeffer, J. Fries, N. Shah. npj Digit. Medicine, 2023

339 citations recorded by the source corpus.

The success of foundation models such as ChatGPT and AlphaFold has spurred significant interest in building similar models for electronic medical records (EMRs) to improve patient care and hospital operations. However, recent hype has obscured critical gaps in our understanding of these models’ capabilities. In this narrative review, we examine 84 foundation models trained on non-imaging EMR data (i.e., clinical text and/or structured data) and create a taxonomy delineating their architectures, training data, and potential use cases. We find that most models are trained on small, narrowly-scoped clinical datasets (e.g., MIMIC-III) or broad, public biomedical corpora (e.g., PubMed) and are evaluated on tasks that do not provide meaningful insights on their usefulness to health systems. Considering these findings, we propose an improved evaluation framework for measuring the benefits of clinical foundation models that is more closely grounded to metrics that matter in healthcare.

Subfields: Artificial Intelligence, Health Informatics

View Paper: https://www.semanticscholar.org/paper/5e6566354b23f7e849b1b439a24ba9d4c5cb655f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Anuj Pareek
2. Sergios Gatidis
3. Jean-Benoit Delbrouck

Citations in the corpus

• Adapted large language models can outperform medical experts in clinical text summarization (2023, positive): “Previous work has demonstrated potential across clinical NLP tasks 22,23 , adapting to the medical domain by training a new model 24,25 , fine-tuning an existing model 26,27 or supplying task-specific examples in the model prompt 27,28 .”
• Reasoning-Enhanced Healthcare Predictions with Knowledge Graph Community Retrieval (2024, unknown): “In the healthcare domain, LLMs hold the potential to transform clinical decision-making by providing insights derived from vast amounts of medical data (Wornow et al., 2023; Yang et al., 2022).”