How to Train good Word Embeddings for Biomedical NLP

By Billy Chiu, Gamal K. O. Crichton, Anna Korhonen, Sampo Pyysalo. BioNLP@ACL, 2016

352 citations recorded by the source corpus.

The quality of word embeddings depends on the input corpora, model architectures, and hyper-parameter settings. Using the state-of-the-art neural embedding tool word2vec and both intrinsic and extrinsic evaluations, we present a comprehensive study of how the quality of embeddings changes according to these features. Apart from identifying the most inﬂuential hyper-parameters, we also observe one that creates contradictory re-sults between intrinsic and extrinsic evaluations. Furthermore, we ﬁnd that bigger corpora do not necessarily produce better biomedical domain word embeddings. We make our evaluation tools and resources as well as the created state-of-the-art word embeddings available under open licenses from https://github.com/ cambridgeltl/BioNLP-2016 .

Subfields: Artificial Intelligence, Molecular Biology

View Paper: https://www.semanticscholar.org/paper/8de470d16044429085b590f03216c0a8992e364c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

Citations in the corpus

Intrinsic Evaluation of Word Vectors Fails to Predict Extrinsic Performance (2016, positive): “In concurrent work, we have also observed a similar effect for biomedical domain tasks and word vectors (Chiu et al., 2016).”
A Neural Autoencoder Approach for Document Ranking and Query Refinement in Pharmacogenomic Information Retrieval (2018, background): “The quality of biomedical word embeddings was investigated by Th et al. (2015) and Chiu et al. (2016).”
A question-entailment approach to question answering (2019, background): “Word embeddings trained specifically for the medical domain can also play an important role in improving the performance [56], as well as different forms of embeddings aggregation that should be investigated when building aggregated sentence embeddings [57, 58].”
Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets (2019, background): “One established trend is a form of word embed-dings that represent the semantic, using high dimensional vectors (Chiu et al., 2016; Wang et al., 2018c; Zhang et al., 2019).”
From POS tagging to dependency parsing for biomedical event extraction (2018, methodology): “ssigned correctly. For the three BiLSTM-CRF-based models, StanfordNNdep, jPTDP and Stanford-Biane which utilizes pre-trained word embeddings, we employ 200- dimensional pre-trained word vectors from [32]. These pre-trained vectors were obtained by training the Word2Vec skip-gram model [33] on a PubM”
Deep learning meets ontologies: experiments to anchor the cardiovascular disease ontology in the biomedical literature (2018, methodology): “Chiu et al. [42] used the Pyysalo et al. [30] datasets and more than 10 M PubMed English abstracts from the BioASQ challenge [43] for an intrinsic evaluation of the Skip-gram and CBOW word embeddings with the Pakhomov et al. [36] word-pairs.”
Data-driven Summarization of Scientific Articles (2018, methodology): “com/moses-smt/mosesdecoder (6)We use the best-performing Word2Vec model from (Chiu et al., 2016), which is trained on PubMed and MEDLINE.”
Condensed Memory Networks for Clinical Diagnostic Inferencing (2016, background): “In the future, we will explore other knowledge sources and the recently proposed word vectors for the biomedical domain (Chiu et al. 2016).”
Mixed Dimension Embeddings with Application to Memory-Efficient Recommendation Systems (2019, background): “Embedding representations power state-of-the-art applications in diverse domains, including computer vision [7, 53], natural language processing [13, 33, 31, 49, 2, 43, 32], computational biology [5], and recommendation systems [12, 41, 54].”
Measuring Gender Bias in Word Embeddings across Domains and Discovering New Gender Bias Word Categories (2019, methodology): “…2015); (c) Skip-Gram embeddings trained on the PubMed Central Open Access subset (PMC) and PubMed 4 , with a vocabulary of about 2.2M word types (Chiu et al., 2016) and trained using two different sliding window sizes: 2 and 30 words; (d) FastText embeddings trained on the GAP corpus (Webster…”