A Simple Word Embedding Model for Lexical Substitution

By Oren Melamud, Omer Levy, Ido Dagan. 2015

113 citations recorded by the source corpus.

The lexical substitution task requires identifying meaning-preserving substitutes for a target word instance in a given sentential context. Since its introduction in SemEval-2007, various models addressed this challenge, mostly in an unsupervised setting. In this work we propose a simple model for lexical substitution, which is based on the popular skip-gram word embedding model. The novelty of our approach is in leveraging explicitly the context embeddings generated within the skip-gram model, which were so far considered only as an internal component of the learning process. Our model is efficient, very simple to implement, and at the same time achieves state-ofthe-art results on lexical substitution tasks in an unsupervised setting.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

context2vec: Learning Generic Context Embedding with Bidirectional LSTM (2016, methodology): “The second-best is due to Melamud et al. (2015b).”
Annotating Relation Inference in Context via Question Answering (2016, methodology): “We also tried to use the arguments’ embeddings to induce a context-sensitive measure of similarity, as suggested by Melamud et al. (2015); however, this method did not improve performance on our dataset.”
Obfuscating Gender in Social Media Writing (2016, methodology): “We use the measure below, adapted from Melamud et al. (2015), giving the substitutability of a for b in the context of a list of tokens C by averaging over b and the context: Unlike Melamud et al. (2015) who rely on the dependency-parse-based system throughout, we take Sem( a, c ) to be the cosine s”
SenticNet 5: Discovering Conceptual Primitives for Sentiment Analysis by Means of Context Embeddings (2018, background): “• Baseline 1 – Melamud et al. (Melamud et al. 2015) pro- posed a model to find lexical substitution of a target based on skipgram word embeddings and incorporating syntactic relations in the skipgram model.”
SenticNet 6: Ensemble Application of Symbolic and Subsymbolic AI for Sentiment Analysis (2020, background): “Baseline 2 is a model proposed by [35] to find lexical substitution of a target based on skipgram word embeddings and incorporating syntactic relations in the skipgram model.”
Learning to Represent Words in Context with Multilingual Supervision (2015, background): “The original task allow to make multiple predictions but we only predict only one substitution following (Melamud et al., 2015).”
Dependency Based Embeddings for Sentence Classification Tasks (2016, background): “to word similarity or lexical substitution in context (Melamud et al., 2015), and little is known about performance within other NLP tasks.”
Guiding Exploratory Behaviors for Multi-Modal Grounding of Linguistic Descriptions (2018, methodology): “The robot estimated the context reliability weights for the novel predicate using the lexical substitution-focused word embeddings (Melamud, Levy, and Dagan 2015) via Eq.”
Context Mover's Distance & Barycenters: Optimal transport of contexts for building representations (2018, background): “Based on which, many works (Levy & Goldberg, 2014a; Melamud et al., 2015; Bojanowski et al., 2016) have suggested specializing the embeddings to capture some particular information required for the task at hand.”