Health system-scale language models are all-purpose prediction engines

By L. Jiang, X. Liu, Nima Pour Nejatian, Mustafa Nasir-Moin, Duo Wang, Anas Abidin, Kevin Eaton, H. Riina, Ilya Laufer, Paawan Punjabi and 18 more. Nature, 2023

481 citations recorded by the source corpus.

Physicians make critical time-constrained decisions every day. Clinical predictive models can help physicians and administrators make decisions by forecasting clinical and operational events. Existing structured data-based clinical predictive models have limited use in everyday practice owing to complexity in data processing, as well as model development and deployment^ 1 – 3 . Here we show that unstructured clinical notes from the electronic health record can enable the training of clinical language models, which can be used as all-purpose clinical predictive engines with low-resistance development and deployment. Our approach leverages recent advances in natural language processing^ 4 , 5 to train a large language model for medical language (NYUTron) and subsequently fine-tune it across a wide range of clinical and operational predictive tasks. We evaluated our approach within our health system for five such tasks: 30-day all-cause readmission prediction, in-hospital mortality prediction, comorbidity index prediction, length of stay prediction, and insurance denial prediction. We show that NYUTron has an area under the curve (AUC) of 78.7–94.9%, with an improvement of 5.36–14.7% in the AUC compared with traditional models. We additionally demonstrate the benefits of pretraining with clinical text, the potential for increasing generalizability to different sites through fine-tuning and the full deployment of our system in a prospective, single-arm trial. These results show the potential for using clinical language models in medicine to read alongside physicians and provide guidance at the point of care. A clinical language model trained on unstructured clinical notes from the electronic health record enhances prediction of clinical and operational events.

Subfields: Artificial Intelligence, Health Informatics

View Paper: https://www.semanticscholar.org/paper/1645e93f34ce34c0ff248a7349bf757a416c5312

Citations in the corpus

• Empowering Biomedical Discovery with AI Agents (2024, unknown): “…results in clinical medicine question-answering suggest that assigning specific roles, such as clinicians, to GPT-4 [61] can achieve better performance in terms of accuracy on multiple-choice benchmarks compared to using domain-specialized LLMs like BioGPT [63], NYUTron [64], and Med-PaLM [65, 66].”
• The TRIPOD-LLM reporting guideline for studies using large language models (2025, unknown): “The rapid advancements in LLMs have stretched existing regulatory and governance structures to their limits, exposing a patchwork of solutions that do not fully encompass the nuances of these all-purpose models 6–8 .”
• Foundation Models for Time Series Analysis: A Tutorial and Survey (2024, unknown): “…AuxMobLCast [102] , LLM-Mob [87] Event: LAMP [81] , Gunjal & Durrett et al. [38] Self-supervised Generative Mobility: GTM [55] Event: NYUTron [45] , GatorTron [105] non-Transformer-based(MLPRNNCNN) Self-supervised Generative Mobility: Trembr [33] Contrastive Mobility: MMTEC [56] Hybrid…”
• A study of generative large language model for medical research and healthcare (2023, unknown): “A recent study 32 examined an LLM developed at New York University, i.e., NYUTron, and our previously developed Gator-Tron 15 for prediction of readmission, in-hospital mortality, comorbidity, length of stay, and insurance denial, demonstrating the potential utility of LLMs in healthcare.”
• Large language models empowered agent-based modeling and simulation: a survey and perspectives (2023, unknown): “Moreover, pre-trained models can be further fi ne-tuned for speci fi c tasks, adapting to particular application scenarios (Jiang et al., 2023).”
• General-purpose large language models outperform specialized clinical AI tools on medical benchmarks (2026, unknown): “The path forward may ultimately lie with hospital-specific LLMs that leverage institutional data 13,14 to mitigate external harm 15 , along with careful use of frontier models for less-sensitive tasks 16 .”
• Large Language Models Predict Functional Outcomes after Acute Ischemic Stroke (2026, unknown): “14,23 Yet, to date, no study has evaluated how generative LLMs perform on mRS prediction using unstructured clinical documentation as the sole input.”