Structured Training for Neural Network Transition-Based Parsing

By David Weiss, Chris Alberti, Michael Collins, Slav Petrov. 2015

236 citations recorded by the source corpus.

We present structured perceptron training for neural network transition-based dependency parsing. We learn the neural network representation using a gold corpus augmented by a large number of automatically parsed sentences. Given this fixed network representation, we learn a final layer using the structured perceptron with beam-search decoding. On the Penn Treebank, our parser reaches 94.26% unlabeled and 92.41% labeled attachment accuracy, which to our knowledge is the best accuracy on Stanford Dependencies to date. We also provide indepth ablative analysis to determine which aspects of our model provide the largest gains in accuracy.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Recent Trends in Deep Learning Based Natural Language Processing (2017, background): “6 (UAS/LAS) [134] Deep fully-connected NN with features including POS 94.”
Transition-Based Dependency Parsing with Stack Long Short-Term Memory (2015, background): “The resurgence of interest in neural networks has resulted in in several applications to transition-based dependency parsers (Weiss et al., 2015; Chen and Manning, 2014; Stenetorp, 2013).”
Globally Normalized Transition-Based Neural Networks (2016, methodology): “Even though we do not use tri-training, our model compares favorably to the 94.26% LAS and 92.41% UAS reported by Weiss et al. (2015) with tri-training.”
A Primer on Neural Network Models for Natural Language Processing (2015, background): “In particular, Weiss et al (Weiss et al., 2015) report strong results for transition-based dependency parsing in a two-stage model.”
Improved Transition-based Parsing by Modeling Characters instead of Words with LSTMs (2015, methodology): “For English, we report (Weiss et al., 2015) and for Chinese, we report (Dyer et al.”
Easy-First Dependency Parsing with Hierarchical Tree LSTMs (2016, methodology): “to a neural-network classiﬁer (Pei et al., 2015; Durrett and Klein, 2015); greedy transition based parsers that use local features that are fed into a neuralnetwork classiﬁer (Chen and Manning, 2014; Weiss et al., 2015), sometimes coupled with a node composition function (Dyer et al., 2015; Watanabe”
Stochastic Beams and Where to Find Them: The Gumbel-Top-k Trick for Sampling Sequences Without Replacement (2019, methodology): “Our experiments support the idea that it can be used as a drop-in replacement in places where sampling or beam search is used.”
Stack-propagation: Improved Representation Learning for Syntax (2016, methodology): “In recent years, transition-based dependency parsers powered by neural network scoring functions have dramatically increased the state-of-theart in terms of both speed and accuracy (Chen and Manning, 2014; Alberti et al., 2015; Weiss et al., 2015).”
Distilling an Ensemble of Greedy Dependency Parsers into One MST Parser (2016, background): “Neural network dependency parsers achieve state of the art performance (Dyer et al., 2015; Weiss et al., 2015; Andor et al., 2016), but training them involves gradient descent on non-convex objectives, which is unstable with respect to initial parameter values.”
Fast and Accurate Sequence Labeling with Iterated Dilated Convolutions (2017, background): “…models for sequence labeling include an inference step that searches the space of possible output sequences of a chain-structured graphical model, or approximates this search with a beam (Collobert et al., 2011; Weiss et al., 2015; Lample et al., 2016; Ma and Hovy, 2016; Chiu and Nichols, 2016).”
Learning Beam Search Policies via Imitation Learning (2018, methodology): “Beam search is the dominant method for approximate decoding in structured prediction tasks such as machine translation [1], speech recognition [2], image captioning [3], and syntactic parsing [4].”
A Fast, Compact, Accurate Model for Language Identification of Codemixed Text (2018, background): “Simple feed-forward networks have achieved near state-of-the-art performance in a wide range of NLP tasks (Botha et al., 2017; Weiss et al., 2015).”
A Survey of Deep Learning for Scientific Discovery (2020, background): “…might be of interest are structured prediction , where the predicted output has some kind of structure, from tree structures (in e.g. parsing) [28, 246] to short, executable computer program structure [271] and summarization , where passages of text are summarized by a neural network [130, 273].”
Recursive Neural Conditional Random Fields for Aspect-based Sentiment Analysis (2016, background): “Note that Weiss et al. (2015) proposed to combine deep learning and structured learning for language parsing which can be learned by structured perceptron.”
Human-in-the-Loop Parsing (2016, background): “Even relatively large datasets, such as the Penn Treebank, are much smaller than required—as demonstrated by improvements from semi-supervised learning (Søgaard and Rishøj, 2010; Weiss et al., 2015).”