A large language model for electronic health records

By Xi Yang, Aokun Chen, Nima M. Pournejatian, Hoo-Chang Shin, Kaleb E. Smith, Christopher Parisien, Colin B. Compas, Cheryl Martin, Anthony B Costa, Mona G. Flores and 9 more. npj Digital Medicine, 2022

857 citations recorded by the source corpus.

There is an increasing interest in developing artificial intelligence (AI) systems to process and interpret electronic health records (EHRs). Natural language processing (NLP) powered by pretrained language models is the key technology for medical AI systems utilizing clinical narratives. However, there are few clinical language models, the largest of which trained in the clinical domain is comparatively small at 110 million parameters (compared with billions of parameters in the general domain). It is not clear how large clinical language models with billions of parameters can help medical AI systems utilize unstructured EHRs. In this study, we develop from scratch a large clinical language model—GatorTron—using >90 billion words of text (including >82 billion words of de-identified clinical text) and systematically evaluate it on five clinical NLP tasks including clinical concept extraction, medical relation extraction, semantic textual similarity, natural language inference (NLI), and medical question answering (MQA). We examine how (1) scaling up the number of parameters and (2) scaling up the size of the training data could benefit these NLP tasks. GatorTron models scale up the clinical language model from 110 million to 8.9 billion parameters and improve five clinical NLP tasks (e.g., 9.6% and 9.5% improvement in accuracy for NLI and MQA), which can be applied to medical AI systems to improve healthcare delivery. The GatorTron models are publicly available at: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/clara/models/gatortron_og .

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/332dc8b2ca9d49fad607c7282f3360bb2a9aacf3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

The Role of AI in Hospitals and Clinics: Transforming Healthcare in the 21st Century (2024, neutral): “An example of such a language model is the GatorTron [34].”
Safe RLHF: Safe Reinforcement Learning from Human Feedback (2023, neutral): “…that leverage LLMs are increasingly enhancing the efficiency of numerous human activities, such as coding (Chen et al., 2021; Gao et al., 2023b), medical assistance (Yang et al., 2022; Moor et al., 2023), education (Kasneci et al., 2023; Kung et al., 2023), law (Katz et al., 2023), and so forth.”
A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics (2023, methodology): “The training data used for GatorTronGPT comprises de-identified clinical text sourced from the University of Florida (UF) Health, along with 195 billion diverse English words obtained from the Pile dataset [87], [228].”
Using AI-generated suggestions from ChatGPT to optimize clinical decision support (2023, methodology): “For example, using a publicly available clinical language model, GatorTron, based on 90 billion words of deidentified clinical notes from the University of Florida Health, PubMed articles, and Wikipedia, might be a good start.(36) Another example is BioGPT, a GPT-like model trained on PubMed article”
Health system-scale language models are all-purpose prediction engines (2023, methodology): “, followed followed by the H&P note of the subsequent readmission.”
Large language models in medicine (2023, background): “Alternative applications have been developed that can access the internet when generating responses, such as BlenderBot 3 and Sparrow 26,27 .”
A study of generative large language model for medical research and healthcare (2023, neutral): “Following our previous study 15 , we performed a minimal preprocessing procedure.”
Large language models in health care: Development, applications, and challenges (2023, methodology): “In addition, GatorTron, the largest clinical language model available, was trained from scratch using over 90 billion words of text from the deidentified clinical notes of University of Florida Health, PubMed articles and Wikipedia [25].”
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, positive): “M-LLMs are instrumental in extracting meaningful insights from EHRs, which encompass diverse and nonregularly timed medical events [89].”
Generative AI in healthcare: an implementation science informed translational path on application, integration and governance (2024, methodology): “LLMs can be used for EHR analysis and as a clinical decision support tool [40].”
Large language models in healthcare and medical domain: A review (2023, methodology): “[123] The GatorTron-large model achieved the best F1 score of 0.9627 for medical relation extraction.”
BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset (2023, background): “The recent advent of large language models (LLMs) [1, 2, 3, 4, 5] promises vast transformative potential across multiple sectors, from healthcare [6, 7, 8] and education [9, 10, 11] to robotics [12, 13, 14] and commerce [15].”
Reasoning-Enhanced Healthcare Predictions with Knowledge Graph Community Retrieval (2024, unknown): “In the healthcare domain, LLMs hold the potential to transform clinical decision-making by providing insights derived from vast amounts of medical data (Wornow et al., 2023; Yang et al., 2022).”