Learning Semantic Word Embeddings based on Ordinal Knowledge Constraints

By QUAN LIU, Hui Jiang, Si Wei, Zhenhua Ling, Yu Hu. 2015

141 citations recorded by the source corpus.

In this paper, we propose a general framework to incorporate semantic knowledge into the popular data-driven learning process of word embeddings to improve the quality of them. Under this framework, we represent semantic knowledge as many ordinal ranking inequalities and formulate the learning of semantic word embeddings (SWE) as a constrained optimization problem, where the data-derived objective function is optimized subject to all ordinal knowledge inequality constraints extracted from available knowledge resources such as Thesaurus and WordNet. We have demonstrated that this constrained optimization problem can be efficiently solved by the stochastic gradient descent (SGD) algorithm, even for a large number of inequality constraints. Experimental results on four standard NLP tasks, including word similarity measure, sentence completion, name entity recognition, and the TOEFL synonym selection, have all demonstrated that the quality of learned word vectors can be significantly improved after semantic knowledge is incorporated as inequality constraints during the learning process of word embeddings.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

context2vec: Learning Generic Context Embedding with Bidirectional LSTM (2016, background): “…such representations were found useful for measuring context-sensitive similarity (Huang et al., 2012), word sense disambiguation (Chen et al., 2014), word sense induction (K˚ageb¨ack et al., 2015), lexical substitution (Melamud et al., 2015b), sentence completion (Liu et al., 2015) and more.”
Leverage Financial News to Predict Stock Price Movements Using Word Embeddings and Deep Neural Networks (2015, methodology): “In this paper, we propose to use the recent word embedding methods (Mikolov et al., 2013b; Liu et al., 2015; Chen et al., 2015) to select features from on-line financial news corpora, and employ deep neural networks (DNNs) to predict the future stock movements based on the extracted features.”
Ordinal Common-sense Inference (2016, background): “…similarity and bow features results in a com-uations in ordinal prediction tasks (Baccianella et al., 2009; Bennett and Lanning, 2007; Gaudette and Japkowicz, 2009; Agresti, 2003; Popescu and Dinu, 2009; Liu et al., 2015; Gella et al., 2013). parable performance drop to removing seq2seq features.”
Revisiting Word Embedding for Contrasting Meaning (2015, background): “Word Embedding Word embedding models learn continuous representations for words in a low dimensional space (Turney and Pantel, 2010; Hin-ton and Roweis, 2002; Collobert et al., 2011; Mikolov et al., 2013; Liu et al., 2015), which is not new.”
Semantic Specialisation of Distributional Word Vector Spaces using Monolingual and Cross-Lingual Constraints (2017, background): “Other ones modify the skip-gram (Mikolov et al., 2013b) objective function by introducing semantic constraints (Yih et al., 2012; Liu et al., 2015) to train word vectors which emphasise word similarity over relatedness.”
Combing Context and Commonsense Knowledge Through Neural Networks for Solving Winograd Schema Problems (2016, methodology): “The idea to learn word embedding based on constraints is similar to the work of (Liu et al. 2015).”
Intent detection using semantically enriched word embeddings (2016, methodology): “However, these enrichment approaches showed improved performance only on word-level tasks such as word similarity [8, 9, 11], antonym detection [10, 11, 16, 13], semantic hierarchies [12, 15], and semantic scale inference [14].”
Counter-fitting Word Vectors to Linguistic Constraints (2016, background): “As far as we are aware, there is no previous work on exploiting antonymy in dialogue systems”
Neural Natural Language Inference Models Enhanced with External Knowledge (2017, background): “…knowledge has shown to be effective in neural networks for other NLP tasks, including word embedding (Chen et al., 2015; Faruqui et al., 2015; Liu et al., 2015; Wieting et al., 2015; Mrksic et al., 2017), machine translation (Shi et al., 2016; Zhang et al., 2017b), language modeling (Ahn et…”
Commonsense Knowledge Enhanced Embeddings for Solving Pronoun Disambiguation Problems in Winograd Schema Challenge (2016, methodology): “In this work, we propose to use a method similar to the one (named SWE) proposed in [21].”
A Survey on Context Learning (2017, background): “[71] proposes a general framework to incorporate ordinal ranking inequalities into neural network based language models.”