context2vec: Learning Generic Context Embedding with Bidirectional LSTM

By Oren Melamud, Jacob Goldberger, Ido Dagan. 2016

514 citations recorded by the source corpus.

Context representations are central to various NLP tasks, such as word sense disam-biguation, named entity recognition, co-reference resolution, and many more. In this work we present a neural model for efﬁciently learning a generic context embedding function from large corpora, us-ing bidirectional LSTM. With a very simple application of our context representations, we manage to surpass or nearly reach state-of-the-art results on sentence completion, lexical substitution and word sense disambiguation tasks, while substantially outperforming the popular context representation of averaged word embeddings. We release our code and pre-trained models, suggesting they could be useful in a wide variety of NLP tasks.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019, background): “4179 mixed results on the downstream task impact of increasing the pre-trained bi-LM size from two to four layers and Melamud et al. (2016) mentioned in passing that increasing hidden dimension size from 200 to 600 helped, but increasing further to 1,000 did not bring further improvements.”
Deep Contextualized Word Representations (2018, methodology): “Word sense disambiguation Given a sentence, we can use the biLM representations to predict the sense of a target word using a simple 1-nearest neighbor approach, similar to Melamud et al. (2016).”
Semi-supervised sequence tagging with bidirectional language models (2017, methodology): “Melamud et al. (2016) learned a context encoder from unlabeled data with an objective function similar to a bi-directional LM and applied it to several NLP tasks closely related to the unlabeled objective function: sentence completion, lexical substitution and word sense disambiguation.”
WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations (2018, background): “…and 1 Ambiguous words are important as they constitute the most frequent words in a natural language (Zipf, 1949). contextualized word embeddings (Melamud et al., 2016; Peters et al., 2018), which instead compute a single dynamic embedding for a given word which can adapt itself to arbitrary…”
SpanBERT: Improving Pre-training by Representing and Predicting Spans (2019, methodology): “Pre-trained contextualized word representations that can be trained from unlabeled text (Dai and Le, 2015; Melamud et al., 2016; Peters et al., 2018) have had immense impact on NLP lately, particularly as methods for initializing a large model before fine-tuning it for a specific task (Howard and…”
How Can We Know What Language Models Know? (2019, background): “Prompts where the hidden vectors learned through a language modeling objective are then used in downstream language understanding systems (Dai and Le, 2015; Melamud et al., 2016; Peters et al., 2018; Devlin et al., 2019).”
Knowledge Enhanced Contextual Word Representations (2019, methodology): “The ELMo and BERT baselines use a nearest neighbor approach trained on the SemCor dataset, similar to the evaluation in Melamud et al. (2016), which has previously been shown to be competitive with task-speciﬁc architectures (Raganato et al., 2017 match the performance of KnowBert-WordNet despite…”
Neural Approaches to Conversational AI (2018, positive): “(2013) and Mesnil et al. (2015) applied recurrent neural networks to slot tagging, where inputs are one-hot encoding of the words in the utterance, and obtained higher accuracy than statistical baselines such as conditional random fields and support vector machines.”
Pre-Trained Models: Past, Present and Future (2021, background): “This motivates pre-training RNNs to provide contextualized word embeddings (Melamud et al., 2016; Peters et al., 2018; Howard and Ruder, 2018), yet the performance of these models is still limited by their model size and depth.”
Adversarial Removal of Demographic Attributes from Text Data (2018, background): “The fact that intermediary vector representations that are trained for one task are predictive of another is not surprising: it is at the core of the success of NLP methods for deriving “generic” word and sentence representations (e.g. Word2vec (Mikolov et al., 2013), Skip-thought vectors (Kiros et ”
Toxicity Detection can be Sensitive to the Conversational Context (2021, background): “In text/word represen- tation, context has a central role (Mikolov et al., 2013; Pennington et al., 2014; Melamud et al., 2016; Peters et al., 2018; Devlin et al., 2019).”
Word Sense Disambiguation: A Unified Evaluation Framework and Empirical Comparison (2017, methodology): “In addition to these conventional approaches, the latest developments in neural language models have motivated some researchers to include them in their WSD architectures (K˚ageb¨ack and Salomonsson, 2016; Melamud et al., 2016; Yuan et al., 2016).”
FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow (2019, result): “This technique is similar to the “masked language model” in previous studies (Melamud et al., 2016; Devlin et al., 2018; Ma et al., 2018).”
SensEmBERT: Context-Enhanced Sense Embeddings for Multilingual Word Sense Disambiguation (2020, background): “In fact, the word representations provided by these models encode several pieces of linguistic information and, differently from static word embed-dings (Mikolov et al. 2013; Pennington, Socher, and Man-ning 2014), they depend directly on the context a word is surrounded by.”
WiC: 10, 000 Example Pairs for Evaluating Context-Sensitive Representations (2018, background): “distinct representations for individual meanings of words, and contextualized word embeddings (Melamud et al., 2016; Peters et al., 2018), which instead compute a single dynamic embedding for a given word which can adapt itself to arbitrary contexts for the word.”