Two/Too Simple Adaptations of Word2Vec for Syntax Problems
By Wang Ling, Chris Dyer, Alan W. Black, Isabel Trancoso. 2015
404 citations recorded by the source corpus.
We present two simple modifications to the models in the popular Word2Vec tool, in order to generate embeddings more suited to tasks involving syntax. The main issue with the original models is the fact that they are insensitive to word order. While order independence is useful for inducing semantic representations, this leads to suboptimal results when they are used to solve syntax-based problems. We show improvements in part-ofspeech tagging and dependency parsing using our proposed models.
Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Computational Linguistics & Semantics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Noah A. Smith
- Mohit Bansal
- Graham Neubig
- Yoav Goldberg
- Luke S. Zettlemoyer
- Percy Liang
- Yoshua Bengio
- Kyunghyun Cho
In Representation & Self-Supervised Learning
- Yoav Goldberg
- Chris Dyer
- Noah A. Smith
- Eduard H. Hovy
- Anders Søgaard
- Diego Marcheggiani
- Graham Neubig
- Yoshua Bengio
In Computational Linguistics & Semantics
- Yoav Goldberg
- Ivan Titov
- Diego Marcheggiani
- Carlos Gómez-Rodríguez
- Daniel Klein
- Daniel Fried
- Eduard H. Hovy
- Xuezhe Ma
Citations in the corpus
- Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling (2017, methodology): “For English, we used external embeddings of Dyer et al. (2015), learned using the structured skip n-gram approach of Ling et al. (2015).”
- Not All Contexts Are Created Equal: Better Word Representations with Variable Attention (2015, background): “While these models are adequate for learning semantic features, one of the problems of this model is the lack of sensitivity for word order, which limits their ability of learn syntactically motivated embeddings (Ling et al., 2015a; Bansal et al., 2014).”
- A Primer on Neural Network Models for Natural Language Processing (2015, methodology): “Ling et al (2015b) model the embedding of a word using the concatenation of the final states of two RNN (LSTM) encoders (Section 10), one reading the characters from left to right, and the other from right to left.”
- Easy-First Dependency Parsing with Hierarchical Tree LSTMs (2016, methodology): “When using pretrained word embeddings, we follow (Dyer et al., 2015) and use embedding vectors which are trained using positional context (Ling et al., 2015), as these were shown to work better than traditional skipgram vectors for syntactic tasks such as part-ofspeech tagging and parsing.”
- Neural Probabilistic Model for Non-projective MST Parsing (2017, methodology): “nese, Dutch, English, German and Spanish, we use the structured-skipgram (Ling et al., 2015) embeddings, and for other languages we use the Polyglot (Al-Rfou et al.”
- Improving Neural Parsing by Disentangling Model Combination and Reranking Effects (2017, methodology): “We train RNNG discriminative (RD) and generative (RG) models, following Dyer et al. (2016) by using the same hyperparameter settings, and using pre-trained word embeddings from Ling et al. (2015) for the discriminative model.”
- Distilling an Ensemble of Greedy Dependency Parsers into One MST Parser (2016, methodology): “All models were augmented with pretrained structured-skipgram (Ling et al., 2015) embeddings; for English we used the Gi-gaword corpus and 100 dimensions, for Chinese Gi-gaword and 80, and for German WMT 2010 mono-lingual data and 64.”
- Deep Active Learning for Named Entity Recognition (2017, methodology): “We use structured skip-gram model (Ling et al., 2015) trained on Gigawords-English corpus (Graff and Cieri, 2003), which showed a good boost over vanilla skip-gram model (Mikolov et al.”
- Stack-Pointer Networks for Dependency Parsing (2018, methodology): “For Chinese, Dutch, English, German and Spanish, we use the structured-skipgram (Ling et al., 2015) embeddings.”
- Parsing as Pretraining (2020, methodology): “• Structured skip-gram word2vec (Ling et al. 2015).”
- A Simple and Accurate Syntax-Agnostic Neural Model for Dependency-based Semantic Role Labeling (2017, methodology): “For the semantic role labeler, we used external embeddings of Dyer et al. (2015) learned using the structured skip n-gram approach of Ling et al. (2015).”
- Learning Cross-Context Entity Representations from Text (2020, methodology): “For FIGMENT, Yaghoobzadeh et al. (2018) is an ensemble of several standard representation learning techniques: word2vec skip-gram contexts (Mikolov et al., 2013), structured skip-gram contexts (Ling et al., 2015), and FastText representations of the entity names (Bojanowski et al., 2017).”
- Head-Driven Phrase Structure Grammar Parsing on Penn Treebank (2019, methodology): “Hyperparameters In our experiments, we use 100D GloVe (Pennington et al., 2014) and structured-skipgram (Ling et al., 2015) pre-train embeddings for English and Chinese respectively.”