How to Generate a Good Word Embedding

By Siwei Lai, Kang Liu, Shizhu He, Jun Zhao. IEEE Intelligent Systems, 2015

368 citations recorded by the source corpus.

The authors analyze three critical components in training word embeddings: model, corpus, and training parameters. They systematize existing neural-network-based word embedding methods and experimentally compare them using the same corpus. They then evaluate each word embedding in three ways: analyzing its semantic properties, using it as a feature for supervised tasks, and using it to initialize neural networks. They also provide several simple guidelines for training good word embeddings.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/67c299c030ea82d80018d8aca8c1f609be97424c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn’t. (2016, background): “…2 affix-derived word relations, animal collective nouns, and verb-object pairs. general accuracy on Google analogy test (Levy et al., 2015; Lai et al., 2015), but, to our knowledge, this is the first study to address the effect of model parameters on individual linguistic relations in the…”
The (too Many) Problems of Analogical Reasoning with Word Vectors (2017, background): “Roughly, the current VSMs score between 40% (Lai et al., 2016) and 75% (Pennington et al., 2014) on th Google test set (Mikolov t al., 2013 ).”
Investigating Language Universal and Specific Properties in Word Embeddings (2016, background): “…efforts in empirically interpreting word embedding and exploring the intrinsic/extrinsic factors in learning process (Andreas and Klein, 2014; Lai et al., 2015; K¨ohn, 2015; Melamud et al., 2016), it remains unknown whether embedding models are really immune to the structural variance of…”
Evaluating word embedding models: methods and experimental results (2019, background): “Formore detailed guidance on how to train a good word embedding, we refer to [59].”
Effects of Semantic Features on Machine Learning-Based Drug Name Recognition Systems: Word Embeddings vs. Manually Constructed Dictionaries (2015, methodology): “Following previous studies [15,56], the dimension of word embeddings is set to 50.”
Detecting weak and strong Islamophobic hate speech on social media (2018, neutral): “(Lai, Liu, He, & Zhao, 2016, p. 8).”
A Survey on Context Learning (2017, background): “Though it is hard to conclude which is the best as they were trained on different corpora and designed for different tasks, we can observe that a larger corpus of a suitable domain yields better word embeddings for specific tasks [66].”
Word Embeddings, Analogies, and Machine Learning: Beyond king - man + woman = queen (2016, background): “The performance of word embeddings can be drastically affected by their parameters (Levy et al., 2015; Lai et al., 2015), which prompts parameter searching for different tasks.”
Drug drug interaction extraction from biomedical literature using syntax convolutional neural network (2016, methodology): “Since the larger corpus will generate the better embedding (Lai et al. , 2015), besides the original DDI corpus, a total of 4 653 097 Medline abstracts were downloaded from PubMed website (http://www.ncbi.nlm. nih.gov/pubmed/) to learn the syntax word embedding with a query string ‘drug’.”