Recurrent Neural Network Grammars

By Chris Dyer, A. Kuncoro, Miguel Ballesteros, Noah A. Smith. 2016

551 citations recorded by the source corpus.

We introduce recurrent neural network grammars, probabilistic models of sentences with explicit phrase structure. We explain efficient inference procedures that allow application to both parsing and language modeling. Experiments show that they provide better parsing in English than any single previously published supervised generative model and better language modeling than state-of-the-art sequential RNNs in English and Chinese.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

Attention is All you Need (2017, neutral): “(2014) [37] semi-supervised 92.1 Transformer (4 layers) semi-supervised 92.7 Luong et al. (2015) [23] multi-task 93.0 Dyer et al. (2016) [8] generative 93.3 increased the maximum output length to input length + 300 .”
Evidence Aggregation for Answer Re-Ranking in Open-Domain Question Answering (2017, methodology): “Re-ranking has been commonly used in NLP problems, such as in parsing and translation, in order to make use of high-order or global features that are too expensive for decoding algorithms (Collins & Koo, 2005; Shen et al., 2004; Huang, 2008; Dyer et al., 2016).”
Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies (2016, positive): “…1997) or gated recurrent units (GRU) (Cho et al., 2014), has led to signiﬁcant gains in language modeling (Mikolov et al., 2010; Sundermeyer et al., 2012), parsing (Vinyals et al., 2015; Kiperwasser and Goldberg, 2016; Dyer et al., 2016), machine translation (Bahdanau et al., 2015) and other tasks.”
Targeted Syntactic Evaluation of Language Models (2018, methodology): “This can be done by combining it with a supervised parser (Dyer et al., 2016) or other multi-task learning objectives (Enguehard et al.”
A Fast Unified Model for Parsing and Sentence Understanding (2016, background): “In addition, there has been recent work proposing models designed primarily for generative language modeling tasks that use this architecture as well (Zhang et al., 2016; Dyer et al., 2016).”
Direct Output Connection for a High-Rank Language Model (2018, methodology): “Neural language models also positively inﬂuence syntactic parsing (Dyer et al., 2016; Choe and Charniak, 2016).”
Junction Tree Variational Autoencoder for Molecular Graph Generation (2018, background): “On the decoding side, tree generation naturally arises in natural language parsing (Dyer et al., 2016; Kiperwasser & Goldberg, 2016).”
Deriving Neural Architectures from Sequence and Graph Kernels (2017, background): “Dyer et al. (2015; 2016) recently introduced a recursive neural model for transition-based language modeling and parsing.”
Graph Convolutional Encoders for Syntax-aware Neural Machine Translation (2017, background): “Eriguchi et al. (2017) multi-task with a target-side RNNG parser (Dyer et al., 2016) and improve on various language pairs with English on the target side.”
Deep Learning for Symbolic Mathematics (2019, methodology): “Alternative approaches have been proposed to generate trees, such as Tree-LSTM (Tai et al., 2015) or Recurrent Neural Network Grammars (RNNG) (Dyer et al., 2016; Eriguchi et al., 2017).”
Learning Deep Generative Models of Graphs (2018, background): “Dyer et al. (2016) proposed recurrent neural network models capable of modeling any top-down transition-based parsing process for generating parse trees.”
Assessing Composition in Sentence Vector Representations (2018, methodology): “We plan to test more substantially distinct models, like those with explicit incorporation of syntactic structure (Bowman et al., 2016; Dyer et al., 2016; Socher et al., 2013) in future work.”
Barack’s Wife Hillary: Using Knowledge Graphs for Fact-Aware Language Modeling (2019, methodology): “Given samples from a proposal distribution q ( E | x ) the marginal distribution is: This approach is used to evaluate models in Ji et al. (2017) and Dyer et al. (2016).”
Good-Enough Compositional Data Augmentation (2019, background): “A large body of work in natural language processing provides generalization to data like (2a) by adding structure to the learned predictor (Chelba and Jelinek, 1998; Chiang, 2005; Dyer et al., 2016).”
What Do Recurrent Neural Network Grammars Learn About Syntax? (2016, methodology): “In this paper, we focus on a recently proposed class of probability distributions, recurrent neural network grammars (RNNGs; Dyer et al., 2016), designed to model syntactic derivations of sentences.”
LSTMs Can Learn Syntax-Sensitive Dependencies Well, But Modeling Structure Makes Them Better (2018, methodology): “Dyer et al. (2016) instead used an importance sampling procedure to marginalize and obtain an estimate of p(x).”
Tree-to-tree Neural Networks for Program Translation (2018, background): “translation, where both the encoder and the decoder are variants of the RNNG model [12]; however, the performance of their model is slightly worse than the sequence-to-sequence model with attention, which is mainly due to the fact that their attention mechanism can not condition the future attention”