Discriminative Training of a Neural Network Statistical Parser

By James Henderson. 2004

120 citations recorded by the source corpus.

Discriminative methods have shown significant improvements over traditional generative methods in many machine learning applications, but there has been difficulty in extending them to natural language parsing. One problem is that much of the work on discriminative methods conflates changes to the learning method with changes to the parameterization of the problem. We show how a parser can be trained with a discriminative learning method while still parameterizing the problem according to a generative probability model. We present three methods for training a neural network to estimate the probabilities for a statistical parser, one generative, one discriminative, and one where the probability model is generative but the training criteria is discriminative. The latter model outperforms the previous two, achieving state-of-the-art levels of performance (90.1% F-measure on constituents).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Recurrent Neural Network Grammars (2016, methodology): “To further reduce the computational complexity of modeling the generation of a word, we use a class-factored softmax (Baltescu and Blunsom, 2015; Goodman, 2001).”
Grammar as a Foreign Language (2014, background): “Early neural network approaches to parsing, for example by Henderson (2003; 2004) also relied on strong linguistic insights.”
A Fast Unified Model for Parsing and Sentence Understanding (2016, background): “There is a fairly long history of work on building neural network-based parsers that use the core operations and data structures from transition-based parsing, of which shift-reduce parsing is a variant (Henderson, 2004; Emami and Jelinek, 2005; Titov and Henderson, 2010; Chen and Manning, 2014; Buy”
A Fast and Accurate Dependency Parser using Neural Networks (2014, methodology): “(Henderson, 2004) was the ﬁrst to attempt to use neural networks in a broad-coverage Penn Tree-bank parser, using a simple synchrony network to predict parse decisions in a constituency parser.”
Globally Normalized Transition-Based Neural Networks (2016, background): “For early work on neural-networks for transition-based parsing, see Henderson (2003; 2004).”
LSTMs Can Learn Syntax-Sensitive Dependencies Well, But Modeling Structure Makes Them Better (2018, positive): “As RNNGs build phrase-structure trees through top-down operations, we propose extensions to the structure-building sequences and model architecture that enable left-corner (Henderson, 2003, 2004) and bottom-up (Chelba and Jelinek, 2000; Emami and Jelinek, 2005) generation orders (§4).”
Learning Accurate, Compact, and Interpretable Tree Annotation (2006, background): “2 Other techniques are also possible; Henderson (2004) uses neural networks to induce latent left-corner parser states. cursively: Although we show only the binary component here, of course there are both binary and unary productions that are included.”
A Minimal Span-Based Neural Constituency Parser (2017, background): “, 2014) and deep neural potentials (Henderson, 2004; Ballesteros et al., 2016).”
Measurements of the properties of the Higgs-like boson in the four lepton decay channel with the ATLAS detector using 25 fb$^{-1}$ of proton-proton collision data (2013, unknown): “[38] and Fisher Linear Discriminant [55].”
Feature Optimization for Constituent Parsing via Neural Networks (2015, background): “Later, he also proposed to rerank k-best parse trees with a neural network model which achieved state-of-the-art performance (Henderson, 2004).”
CCGbank: A Corpus of CCG Derivations and Dependency Structures Extracted from the Penn Treebank (2007, methodology): “Statistical parsing on the Penn Treebank has made great progress by focusing on the machine-learning or algorithmic aspects (Magerman 1994; Ratnaparkhi 1998; Collins 1999; Charniak 2000; Henderson 2004; McDonald, Crammer, and Pereira 2005).”
What Syntax Can Contribute in the Entailment Task (2005, background): “Various parsers providing constituent level analysis are now available to the research community, and state-of-the-art parsers have reported accuracy of between 89% and 90.1% F-measure (Collins and Duffy, 2002, Henderson 2004; see Ringger et al., 2004 for results with a non-treebank parser).”
Translation Modeling with Bidirectional Recurrent Neural Networks (2014, background): “They have been since applied to different tasks like parsing (Henderson, 2004) and spoken language understanding (Mesnil et al., 2013).”
Deep Learning for Efficient Discriminative Parsing (2011, methodology): “Henderson (2004) outperforms Collins parser only by using a generative model and performing re-ranking.”
The Unstoppable Rise of Computational Linguistics in Deep Learning (2020, unknown): “imate the parameters of an n-gram language model, and showed PTB Constituents model LP LR F1 Costa et al. (2001) PoS 57.8 64.9 61.1 Henderson (2003) PoS 83.3 84.3 83.8 Henderson (2003) 88.8 89.5 89.1 Henderson (2004) 89.8 90.4 90.1 Vinyals et al. (2015) seq2seq <70 Vinyals et al. (2015) attn 88.3”
Forest Reranking: Discriminative Parsing with Non-Local Features (2008, background): “Our ﬁnal result (91.7) is better than any previously reported system trained on the Treebank, although type system F 1 % Collins (2000) 89.7 Henderson (2004) 90.1 Charniak and Johnson (2005) 91.0 updated (Johnson, 2006) 2006) achieved an even higher ac-cuarcy (92.1) by leveraging on much larger…”
Incremental Sigmoid Belief Networks for Grammar Learning (2010, methodology): “Also, in many complex predication tasks, such as phrase structure parsing, many of the most accurate models make use of a joint model, either in reranking or model combinations (e.g. (Charniak and Johnson, 2005; Henderson, 2004)).”
Structured Training for Neural Network Transition-Based Parsing (2015, background): “Finally, we also note that neural network representations have a long history in syntactic parsing (Henderson, 2004; Titov and Henderson, 2007; Titov and Henderson, 2010); however, like Chen and Manning (2014), our network avoids any recurrent structure so as to keep inference fast and efficient and”