Neural CRF Parsing

By Greg Durrett, Daniel Klein. 2015

129 citations recorded by the source corpus.

This paper describes a parsing model that combines the exact dynamic programming of CRF parsing with the rich nonlinear featurization of neural net approaches. Our model is structurally a CRF that factors over anchored rule productions, but instead of linear potential functions based on sparse features, we use nonlinear potentials computed via a feedforward neural network. Because potentials are still local to anchored rules, structured inference (CKY) is unchanged from the sparse case. Computing gradients during learning involves backpropagating an error signal formed from standard CRF sufficient statistics (expected rule counts). Using only dense features, our neural CRF already exceeds a strong baseline CRF model (Hall et al., 2014). In combination with sparse features, our system achieves 91.1 F1 on section 23 of the Penn Treebank, and more generally outperforms the best prior single parser results on a range of languages.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Structured Attention Networks (2017, background): “…applications, most notably in seminal work on graph transformers (LeCun et al., 1998), key work on NLP (Collobert et al., 2011), and in many other areas (Peng et al., 2009; Do & Arti´eres, 2010; Jaderberg et al., 2014; Chen et al., 2015; Durrett & Klein, 2015; Lample et al., 2016, inter alia ).”
Globally Normalized Transition-Based Neural Networks (2016, neutral): “We purpose-fully abstain from complicated hand engineering of input features, which might improve performance further (Durrett and Klein, 2015).”
Constituency Parsing with a Self-Attentive Encoder (2018, methodology): “RNNs have largely replaced approaches such as the ﬁxed-window-size feed-forward networks of Durrett and Klein (2015) in part due to their ability to capture global context.”
code2vec: learning distributed representations of code (2018, methodology): “In general, CRFs can also use distributed representations [Artieres et al. 2010; Durrett and Klein 2015], but for the purpose of this discussion, “CRFs” refers to a CRFs with symbolic representations as used by Alon et al. [2018] and Raychev et al. [2015].”
A Primer on Neural Network Models for Natural Language Processing (2015, methodology): “A hinge based approached was used by Pei et al (2015) for arc-factored dependency parsing, and the probabilistic approach by Durrett and Klein (Durrett & Klein, 2015) for a CRF constituency parser.”
What Does it Mean for a Language Model to Preserve Privacy? (2022, background): “…is a short list of some common language model tasks, which are the foundation of many of LM applications: part of speech (POS) tagging and parsing [25], optical character recognition (OCR) [65], automatic speech recognition [74], natural language generation [11], sentiment analysis [97], and…”
Easy-First Dependency Parsing with Hierarchical Tree LSTMs (2016, background): “…there are four dominant approaches: search based parsers that use local features that are fed to a neural-network classifier (Pei et al., 2015; Durrett and Klein, 2015); greedy transition based parsers that use local features that are fed into a neuralnetwork classifier (Chen and Manning,…”
Approximation-Aware Dependency Parsing by Belief Propagation (2015, background): “Another natural extension of this work is to explore other types of factors: here we considered only log-linear potential functions (commonly used in CRFs), but any differentiable function would be appropriate, such as a neural network (Durrett and Klein, 2015; Gormley et al., 2015b).”
An Empirical Study of Building a Strong Baseline for Constituency Parsing (2018, background): “In constituency parsing, several systems also incorporate pre-trained word embed-dings, such as Vinyals et al. (2015); Durrett and Klein (2015).”
LSTM CCG Parsing (2016, background): “…shift-reduce parsing (Henderson et al., 2013; Chen and Manning, 2014; Weiss et al., 2015; Dyer et al., 2015), constituency parse re-ranking (Socher et al., 2013), and string-to-string transduction (Vinyals et al., 2015), or (2) avoid recursive computations entirely (Durrett and Klein, 2015).”
In-Order Transition-based Constituent Parsing (2017, unknown): “(2013) 90.4 Zhu et al. (2013) 90.4 Vinyals et al. (2015) 90.7 Watanabe and Sumita (2015) 90.7 Shindo et al. (2012) 91.1 Durrett and Klein (2015) 91 2016; Watanabe and Sumita, 2015) and the state-of-the-art hybrid parsers (Durrett and Klein, 2015; Liu and Zhang, 2017), achieving the state-of-the-art…”
Semantic Role Labeling with Neural Network Factors (2015, methodology): “A similar use of non-linear potential functions in a structured model was proposed by Do and Artières (2010) for speech recognition, and by Durrett and Klein (2015) for syntactic phrase-structure parsing.”
Compound Probabilistic Context-Free Grammars for Grammar Induction (2019, methodology): “tween the original model and the URNNG’s structured inference network, which is parameterized as a neural CRF constituency parser (Durrett and Klein, 2015; Liu et al., 2018).”
Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both? (2020, methodology): “Unlike Durrett and Klein (2015), we compute these scores based on contextualized representations from a BERT (Devlin et al., 2019) encoder.”
Unsupervised Recurrent Neural Network Grammars (2019, methodology): “C L ] 9 A pr 2 01 9 ditional random field (CRF) constituency parser (Finkel et al., 2008; Durrett and Klein, 2015), which makes significant independence assumptions, acts as a guide on the generative model to learn meaningful trees through regularizing the posterior (Ganchev et al., 2010).”
A Minimal Span-Based Neural Constituency Parser (2017, background): “An alternative line of work focuses on chart parsers, which use log-linear or neural scoring potentials to parameterize a tree-structured dynamic program for maximization or marginalization (Finkel et al., 2008; Durrett and Klein, 2015).”
Stack-propagation: Improved Representation Learning for Syntax (2016, methodology): “Similar approaches also achieve state-ofthe-art in other NLP tasks, such as constituency parsing (Durrett and Klein, 2015) or semantic role labeling (FitzGerald et al., 2015).”
Lifelong Domain Word Embedding via Meta-Learning (2018, background): “…et al. , 2013a; Mikolov et al. , 2013b; Pennington et al. , 2014] has received a great deal of attention due to its success in numerous NLP applications, e.g., named entity recognition [ Sien ˇ cnik, 2015 ] , sentiment analysis [ Maas et al. , 2011] and syntactic parsing [Durrett and Klein, 2015].”