Improving Lexical Embeddings with Semantic Knowledge

By Mo Yu, Mark Dredze. Annual Meeting of the Association for Computational Linguistics, 2014

315 citations recorded by the source corpus.

Word embeddings learned on unlabeled data are a popular tool in semantics, but may not capture the desired semantics. We propose a new learning objective that incorporates both a neural language model objective (Mikolov et al., 2013) and prior knowledge from semantic resources to learn improved lexical semantic embeddings. We demonstrate that our embeddings improve over those learned solely on raw text in three settings: language modeling, measuring semantic similarity, and predicting human judgements.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/c8dfdb6bc17094fc1c35757a0020dea8d813b7b6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Embedding Words and Senses Together via Joint Knowledge-Enhanced Training (2016, background): “Recent approaches have utilized semantic networks to inject knowledge into existing word representations (Yu and Dredze, 2014; Faruqui et al., 2015; Goikoetxea et al., 2015; Speer and LowryDuda, 2017; Mrksic et al., 2017), but without solving the meaning conflation issue.”
Exploiting Domain Knowledge via Grouped Weight Sharing with Application to Text Categorization (2017, background): “There have been efforts to exploit such resources specifically to induce better word vectors (Yu and Dredze, 2014; Faruqui et al., 2014; Yu et al., 2016; Xu et al., 2014).”
Counter-fitting Word Vectors to Linguistic Constraints (2016, background): “This hypothesis supports unsupervised learning of meaningful word representations from large corpora (Curran, 2003; Ó Séaghdha and Korhonen, 2014; Mikolov et al., 2013; Pennington et al., 2014).”
Neural Path Hunter: Reducing Hallucination in Dialogue Systems via Path Grounding (2021, background): “, 2017), and natural language processing (Berant et al., 2013; Yu and Dredze, 2014), to name a few.”
PPDB 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification (2015, unknown): “, 2011), and improved lexical embeddings (Yu and Dredze, 2014; Rastogi et al., 2015; Faruqui et al., 2015).”
Common Sense or World Knowledge? Investigating Adapter-Based Knowledge Injection into Pretrained Transformers (2020, background): “Joint pretraining models, on the one hand, augment distributional LM objectives with additional objectives based on external resources (Yu and Dredze, 2014; Nguyen et al., 2016; Lauscher et al., 2019) and train the extended model from scratch.”
RC-NET: A General Framework for Incorporating Knowledge into Word Representations (2014, background): “Most recently, Yu et al. [25] attempted to use knowledge to improve word representations.”
Learning Domain-Specific Word Embeddings from Sparse Cybersecurity Texts (2017, background): “Since then, many extensions to Word2Vec have been proposed (Levy and Goldberg 2014), (Luong, Socher, and Manning 2013), (Yu and Dredze 2014), (Bian, Gao, and Liu 2014), (Xu et al. 2014), (Faruqui et al. 2014), (Bol-legala et al. 2016), (Le and Mikolov 2014).”
Named Entity Recognition for Chinese Social Media with Jointly Trained Embeddings (2015, background): “Yu and Dredze (2014) use multi-part objectives that include both standard unlabeled objectives, such as skip-gram models in word2vec, and task speciﬁc objectives.”
Semantic Specialisation of Distributional Word Vector Spaces using Monolingual and Cross-Lingual Constraints (2017, methodology): “Learning from Scratch: some methods modify the prior or the regularization of the original training procedure using the set of linguistic constraints (Yu and Dredze, 2014; Xu et al., 2014; Bian et al., 2014; Kiela et al., 2015; Aletras and Stevenson, 2015).”
Sparse Overcomplete Word Vector Representations (2015, methodology): “Unlike recent approaches of incorporating semantics in distributional word vectors (Yu and Dredze, 2014; Xu et al., 2014; Faruqui et al., 2015), the method does not rely on any external information source.”
Sensing spatial distribution of urban land use by integrating points-of-interest and Google Word2Vec model (2017, methodology): “Compared with the stochastic sampling of word pairs in the Skip-Gram model, the continuous input and training process of CBOW can better re ﬂ ect the context relationships characterizing words (Yu and Dredze 2014).”
Supersense Embeddings: A Unified Model for Supersense Interpretation, Prediction, and Utilization (2016, background): “An alternative approach focuses on altering the objective of the learning mechanism to capture relational and similarity information from knowledge bases (Bordes et al., 2011; Bordes et al., 2012; Yu and Dredze, 2014; Bian et al., 2014; Faruqui and Dyer, 2014; Goikoetxea et al., 2015).”
De-Conflated Semantic Representations (2016, background): “Another notable line of research incorporates knowledge from external resources, such as PPDB (Ganitkevitch et al., 2013) and WordNet, to improve word embeddings (Yu and Dredze, 2014; Faruqui et al., 2015).”
Directional Skip-Gram: Explicitly Distinguishing Left and Right Context for Word Embeddings (2018, background): “To improve original word embedding models, there are various studies leveraging external knowledge to update word embeddings with post processing (Faruqui et al., 2015; Kiela et al., 2015; Song et al., 2017) or supervised objectives (Yu and Dredze, 2014; Nguyen et al., 2016).”
Commonsense Knowledge Enhanced Embeddings for Solving Pronoun Disambiguation Problems in Winograd Schema Challenge (2016, background): “In the natural language processing community, there are some works that have been proposed to employ semantic knowledge to improve the learning of word embeddings [17, 18, 19, 20, 21].”
Assessing State-of-the-Art Sentiment Models on State-of-the-Art Sentiment Datasets (2017, methodology): “Yu and Dredze (2014) propose several methods which combine the CBOW architecture (Mikolov et al., 2013) and a second objective function which attempts to maximize the relations found within some semantic lexicon.”
From Word to Sense Embeddings: A Survey on Vector Representations of Meaning (2018, methodology): “…embeddings using lexical resources modiﬁed the objective function of a neural language model for learning word embeddings (e.g., Skip-gram of Word2vec) in order to integrate relations from lexical resources into the learning process (Xu, Bai, Bian, Gao, Wang, Liu, & Liu, 2014; Yu & Dredze, 2014).”
Learning Continuous Word Embedding with Metadata for Question Retrieval in Community Question Answering (2015, methodology): “Most recently, Yu et al. (2014) used semantic prior knowledge to improve word representations.”