What Do Recurrent Neural Network Grammars Learn About Syntax?

By Noah A. Smith, Chris Dyer, Miguel Ballesteros, Graham Neubig, Lingpeng Kong, A. Kuncoro. 2016

152 citations recorded by the source corpus.

Recurrent neural network grammars (RNNG) are a recently proposed probablistic generative modeling family for natural language. They show state-of-the-art language modeling and parsing performance. We investigate what information they learn, from a linguistic perspective, through various ablations to the model and the data, and by augmenting the model with an attention mechanism (GA-RNNG) to enable closer inspection. We find that explicit modeling of composition is crucial for achieving the best performance. Through the attention mechanism, we find that headedness plays a central role in phrasal representation (with the model’s latent attention largely agreeing with predictions made by hand-crafted head rules, albeit with some important differences). By training grammars without nonterminal labels, we find that phrasal representations depend minimally on nonterminals, providing support for the endocentricity hypothesis.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Semi-Supervised Sequence Modeling with Cross-View Training (2018, methodology): “For dependency parsing, we omit results from Choe and Charniak (2016), Kuncoro et al. (2017), and Liu and Zhang (2017) because these train constituency parsers and convert the system outputs to dependency parses.”
Dissecting Contextual Word Embeddings: Architecture and Representation (2018, methodology): “Similarity, the layers most transferable to parsing are at or below the layers with maximum pronom-inal coreference accuracy in all models, as constituent structure tends to be more local than coreference (Kuncoro et al., 2017).”
How Can We Know What Language Models Know? (2019, background): “, 2017), or by ablations to the models to investigate how behavior varies (Li et al., 2016b; Smith et al., 2017).”
Linguistic Knowledge and Transferability of Contextual Representations (2019, background): “, 2016a, inter alia), ablating components of the model and dataset (Kuncoro et al., 2017; Gaddy et al., 2018; Khandelwal et al., 2018), or interpreting attention mechanisms (Bahdanau et al.”
Deep Biaffine Attention for Neural Dependency Parsing (2016, methodology): “This makes it the highest-performing graph-based parser on this benchmark— outperforming Kiperwasser & Goldberg (2016) by 1.8% and 2.2%—and comparable to the highest performing transition-based parser (Kuncoro et al., 2016), which achieves 95.8% UAS and 94.6% LAS.”
An Analysis of Encoder Representations in Transformer-Based Machine Translation (2018, background): “Nowadays, given its popularity, recurrent neural networks are the most evaluated networks, mainly investigated on the structures and linguistic properties they are encoding (Linzen et al., 2016; Enguehard et al., 2017; Kuncoro et al., 2017; Gulordava et al., 2018).”
LSTMs Can Learn Syntax-Sensitive Dependencies Well, But Modeling Structure Makes Them Better (2018, methodology): “Our choice of RNNGs is motivated by the findings of Kuncoro et al. (2017), who find evidence for syntactic headedness in RNNG phrasal representations.”
Learning Language Representations for Typology Prediction (2017, background): “…typology from bilingual data ( ¨Ostling, 2015) and English as Second Language texts (Berzak et al., 2014), as well as work in NLP (Shi et al., 2016; Kuncoro et al., 2017; Belinkov et al., 2017) showing that syntactic knowledge can be extracted from neural nets on the word-by-word or…”
Structural Supervision Improves Learning of Non-Local Grammatical Dependencies (2019, methodology): “The model was trained using embedding size 256, dropout 0.3, and was able to achieve a parsing F1 score of 92.81 on the PTB, which is only marginally better than the performance of the original architecture on the same test set, as reported in Kuncoro et al. (2016).”
An Empirical Study of Building a Strong Baseline for Constituency Parsing (2018, background): “In fact, RNNG and its extensions (Kuncoro et al., 2017; Fried et al., 2017) provide the current state-of-the-art performance.”
Semantic Parsing for Task Oriented Dialog using Hierarchical Representations (2018, methodology): “We also experimented with more minimal representations of the RNNG model (Kuncoro et al., 2017).”
In-Order Transition-based Constituent Parsing (2017, unknown): “With the reranking setting, the in-order parser outperforms the best discrete parser (Huang, 2008) and have the same performance of Kuncoro et al. (2017), which extend the work of Dyer et al. (2016) by adding gated attention mechanism on composition functions.”
Cross-Domain Generalization of Neural Constituency Parsers (2019, methodology): “…model of Liu and Zhang (2017) uses a simpler LSTM-based encoding of the input sentence but a decoder that explicitly conditions on previously constructed structure of the output tree, obtaining the best performance among similarly structured models (Dyer et al., 2016; Kuncoro et al., 2017).”
Scheduled Multi-Task Learning: From Syntax to Translation (2018, background): “This is below the best results achieved by state-of-the-art parsers, that are already around 95 for English (Dozat and Manning, 2016; Kuncoro et al., 2017), and around 90 for the same German dataset (Andor et al., 2016; Kuncoro et al., 2016; Bohnet and Nivre, 2012).”
Interpreting Word-Level Hidden State Behaviour of Character-Level LSTM Language Models (2018, methodology): “More recently, Li et al. (2016) directly visualize representations over time using heatmaps, and Strobelt et al. (2018) develop interactive tools for visualizing LSTM hidden states and testing hypotheses about distributed representations.”