Tailoring Continuous Word Representations for Dependency Parsing

By Mohit Bansal, Kevin Gimpel, Karen Livescu. 2014

313 citations recorded by the source corpus.

Word representations have proven useful for many NLP tasks, e.g., Brown clusters as features in dependency parsing (Koo et al., 2008). In this paper, we investigate the use of continuous word representations as features for dependency parsing. We compare several popular embeddings to Brown clusters, via multiple types of features, in both news and web domains. We find that all embeddings yield significant parsing gains, including some recent ones that can be trained in a fraction of the time of others. Explicitly tailoring the representations for the task leads to further improvements. Moreover, an ensemble of all representations achieves the best results, suggesting their complementarity.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

context2vec: Learning Generic Context Embedding with Bidirectional LSTM (2016, background): “Syntactic dependency context embeddings have been proposed recently (Levy and Goldberg, 2014a; Bansal et al., 2014).”
SimLex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation (2014, background): “(Huang et al., 2012; Bansal et al., 2014) (5)This fact is also noted by the dataset authors.”
Transition-Based Dependency Parsing with Stack Long Short-Term Memory (2015, methodology): “However, for syntax modeling problems, embedding approaches which discard order perform less well (Bansal et al., 2014); therefore we used a variant of the skip n -gram model introduced by Ling et al. (2015), named “structured skip n -gram,” where a different set of parameters is used to predict…”
Machine Comprehension with Syntax, Frames, and Semantics (2015, methodology): “They have been used for many tasks, including semantic role labeling (Collobert et al., 2011), named entity recognition (Turian et al., 2010), parsing (Bansal et al., 2014), and for the Facebook QA tasks (Weston et al., 2015; Sukhbaatar et al., 2015).”
Commonsense Knowledge Base Completion (2016, methodology): “Similar approaches have been used to learn embeddings for particular downstream tasks, e.g., dependency parsing (Bansal et al., 2014).”
What’s in an Embedding? Analyzing Word Embeddings through Multilingual Evaluation (2015, methodology): “By using the continuous features directly instead of clustering them (as e.g. done by Bansal et al. (2014)), we ensure that no information is lost during preprocessing.”
Not All Contexts Are Created Equal: Better Word Representations with Variable Attention (2015, background): “While these models are adequate for learning semantic features, one of the problems of this model is the lack of sensitivity for word order, which limits their ability of learn syntactically motivated embeddings (Ling et al., 2015a; Bansal et al., 2014).”
Sparse Overcomplete Word Vector Representations (2015, background): “Distributed representations of words have been shown to beneﬁt NLP tasks like parsing (Lazari-dou et al., 2013; Bansal et al., 2014), named entity recognition (Guo et al., 2014), and sentiment analysis (Socher et al., 2013).”
From Paraphrase Database to Compositional Paraphrase Model and Back (2015, methodology): “These embeddings are now being used in new ways as they are being tailored to specific downstream tasks (Bansal et al., 2014).”
Neural CRF Parsing (2015, methodology): “We use the Brown clusters from Koo et al. (2008), which are trained on the same data as the vectors of Bansal et al. (2014).”
The Role of Context Types and Dimensionality in Learning Word Embeddings (2016, methodology): “…as a source of features in a broad range of NLP tasks with limited supervision (Turian et al., 2010; Collobert et al., 2011; Socher et al., 2013; Bansal et al., 2014). word2vec1 skip-gram (Mikolov et al., 2013a) and GloVe2 (Pennington et al., 2014) are among the most widely used word embedding…”
Beyond Bilingual: Multi-sense Word Embeddings using Multilingual Context (2017, background): “Use of embed-dings as features has been shown to beneﬁt several NLP tasks and serve as good initializations for deep architectures ranging from dependency parsing (Bansal et al., 2014) to named entity recognition (Guo et al., 2014b).”
Finnish resources for evaluating language model semantics (2017, methodology): “The word embeddings have been used in a vari-1 ety of semantics-incorporating downstream applications such as sentiment analysis and text generation (see e.g., Brigadir et al., 2015; Karpathy and Fei-Fei, 2015; Bansal et al., 2014).”
Learning Domain-Specific Word Embeddings from Sparse Cybersecurity Texts (2017, background): “…word embeddings in a variety of NLP tasks such as Named Entity Recognition (Lample et al. 2016), (Santos and Guimaraes 2015), syntactic parsing (Bansal, Gimpel, and Livescu 2014), semantic relation extraction (Nguyen and Grishman 2015), (Fu et al. 2014) and sentiment analysis (Maas et al.…”
Evaluation of Word Vector Representations by Subspace Alignment (2015, background): “Unsupervised word vectors have been shown to benefit parsing (Lazaridou et al., 2013; Bansal et al., 2014), chunking (Turian et al.”
Counter-fitting Word Vectors to Linguistic Constraints (2016, methodology): “The objective function used to counter-ﬁt the pre-trained word vectors V to the sets of linguistic constraints A and S contains three different terms: 1.”