Not All Contexts Are Created Equal: Better Word Representations with Variable Attention

By Wang Ling, Yulia Tsvetkov, Silvio Amir, Ramón Fernandez Astudillo, Chris Dyer, Alan W. Black, Isabel Trancoso, Chu-Cheng Lin. 2015

142 citations recorded by the source corpus.

We introduce an extension to the bag-ofwords model for learning words representations that take into account both syntactic and semantic properties within language. This is done by employing an attention model that finds within the contextual words, the words that are relevant for each prediction. The general intuition of our model is that some words are only relevant for predicting local context (e.g. function words), while other words are more suited for determining global context, such as the topic of the document. Experiments performed on both semantically and syntactically oriented tasks show gains using our model over the existing bag of words model. Furthermore, compared to other more sophisticated models, our model scales better as we increase the size of the context of the model.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

A Structured Self-attentive Sentence Embedding (2017, background): “Ling et al. (2015) proposed an attention based model for word embedding, which calculates an attention weight for each word at each possible position in the context window.”
context2vec: Learning Generic Context Embedding with Bidirectional LSTM (2016, background): “Ling et al. (2015b) proposed a continuous window , which is a simple linear projection of the context window embeddings into a low dimensional vector.”
Gated Self-Matching Networks for Reading Comprehension and Question Answering (2017, background): “Ling et al. (2015) propose considering window-based contextual words differently depending on the word and its relative position.”
Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering (2015, methodology): “3 We propose a new deep learning approach to VQA that incorporates explicit spatial attention, which we call the Spatial Memory Network VQA (SMemVQA).”
Neural Architectures for Named Entity Recognition (2016, methodology): “They have been found useful for morphologically rich languages and to handle the outof-vocabulary problem for tasks like part-of-speech tagging and language modeling (Ling et al., 2015b) or dependency parsing (Ballesteros et al., 2015).”
Directional Skip-Gram: Explicitly Distinguishing Left and Right Context for Word Embeddings (2018, methodology): “, 2015a), learning context words with different weights (Ling et al., 2015b), etc.”
The Role of Context Types and Dimensionality in Learning Word Embeddings (2016, background): “Ling et al. (2015b) and Ling et al. (2015a) propose extensions to the standard window-based context modeling.”
DeepIntent: Learning Attentions for Online Advertising with Recurrent Neural Networks (2016, methodology): “Methodology wise, our work extends the recently proposed attention mechanism [1, 14] from the NLP tasks such as machine translation and language modeling to the web search applications.”
Episodic CAMN: Contextual Attention-Based Memory Networks with Iterative Feedback for Scene Labeling (2017, background): “Attention-based models have been successfully applied on a broad range of Natural Language Processing (NLP) related tasks, which include machine translation [18] [43], speech recognition [13], image caption generation [65], reading comprehension [28], sentence summarization [48], part-of-speech (POS”
Gated Word-Character Recurrent Language Model (2016, methodology): “Ling et al. (2015a) introduce the compositional character-to-word (C2W) model that takes as input character-level representation of a word and generates vector representation of the word us-ing a bidirectional LSTM (Graves and Schmidhu-ber, 2005).”
TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-Rays (2018, background): “The attention mechanism attempts to ease the above problems by allowing the decoder to refer back to the input sequence [39, 23, 25].”
Evaluation of Word Vector Representations by Subspace Alignment (2015, methodology): “Ling et al. (2015b) propose a syntactic modiﬁca-tion to the WORD 2 VEC models that accounts for word order information, obtaining state-of-the-art performance in syntactic downstream tasks.”
Fast and Accurate Entity Recognition with Iterated Dilated Convolutions (2017, methodology): “In initial experiments, we found the 100-dimensional skip-n-gram (Ling et al., 2013) embeddings of Lample et al. (2016) to outperform the 50-dimensional word embeddings of Collobert et al. (2011), and so we use these 100-dimensional embeddings in all experiments.”
Fast and Accurate Sequence Labeling with Iterated Dilated Convolutions (2017, methodology): “In initial experiments, we found the 100-dimensional skip-n-gram (Ling et al., 2013) embeddings of Lample et al. (2016) to outperform the 50-dimensional word embeddings of Collobert et al. (2011), and so we use these 100-dimensional embeddings in all experiments.”
GAKE: Graph Aware Knowledge Embedding (2016, methodology): “In this work, we model representative powers of different subjects in graph context by an attention mechanism (Ling et al., 2015; Hermann et al., 2015).”