Enhancing Clinical Concept Extraction with Contextual Embedding

By Yuqi Si, Jingqi Wang, Hua Xu, Kirk Roberts. J. Am. Medical Informatics Assoc., 2019

329 citations recorded by the source corpus.

OBJECTIVE Neural network-based representations ("embeddings") have dramatically advanced natural language processing (NLP) tasks, including clinical NLP tasks such as concept extraction. Recently, however, more advanced embedding methods and representations (eg, ELMo, BERT) have further pushed the state of the art in NLP, yet there are no common best practices for how to integrate these representations into clinical tasks. The purpose of this study, then, is to explore the space of possible options in utilizing these new models for clinical concept extraction, including comparing these to traditional word embedding methods (word2vec, GloVe, fastText). MATERIALS AND METHODS Both off-the-shelf, open-domain embeddings and pretrained clinical embeddings from MIMIC-III (Medical Information Mart for Intensive Care III) are evaluated. We explore a battery of embedding methods consisting of traditional word embeddings and contextual embeddings and compare these on 4 concept extraction corpora: i2b2 2010, i2b2 2012, SemEval 2014, and SemEval 2015. We also analyze the impact of the pretraining time of a large language model like ELMo or BERT on the extraction performance. Last, we present an intuitive way to understand the semantic information encoded by contextual embeddings. RESULTS Contextual embeddings pretrained on a large clinical corpus achieves new state-of-the-art performances across all concept extraction tasks. The best-performing model outperforms all state-of-the-art methods with respective F1-measures of 90.25, 93.18 (partial), 80.74, and 81.65. CONCLUSIONS We demonstrate the potential of contextual embeddings through the state-of-the-art performance these methods achieve on clinical concept extraction. Additionally, we demonstrate that contextual embeddings encode valuable semantic information not accounted for in traditional word representations.

Subfields: Artificial Intelligence, Molecular Biology

View Paper: https://www.semanticscholar.org/paper/06b36e744dca445863c9f9aefe76aea95ba95999

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

Citations in the corpus

Shifting machine learning for healthcare from development to deployment and from models to data (2022, methodology): “NLP has enabled the automated translation of doctor–patient interactions to notes 5,104,105 , the summarization of clinical notes 106 , the captioning of medical images 107,108 and the prediction of disease progression 6,7 .”
Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets (2019, methodology): “P: PubMed, P+M: PubMed + MIMIC-III For named entity recognition, we used a Bi-LSTM-CRF implementation as a sequence tag-ger (Huang et al., 2015; Si et al., 2019; Lample et al., 2016).”
Federated pretraining and fine tuning of BERT using clinical notes from multiple silos (2020, unknown): “Later in the year, Alsentzer et al. (2019) and Si et al. (2019)”
Hurtful words: quantifying biases in clinical contextual word embeddings (2020, methodology): “Several BERT models pretrained on MIMIC notes are publicly available [3, 29, 60].”
CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters (2020, background): “While current research mostly focuses on improving language representations for the default “generaldomain”, there seems to be a growing interest in building suitable word embeddings for more specialized domains (El Boukkouri et al., 2019; Si et al., 2019; Elwany et al., 2019).”