Deep RNNs Encode Soft Hierarchical Syntax

By Terra Blevins, Omer Levy, Luke S. Zettlemoyer. 2018

118 citations recorded by the source corpus.

We present a set of experiments to demonstrate that deep recurrent neural networks (RNNs) learn internal representations that capture soft hierarchical notions of syntax from highly varied supervision. We consider four syntax tasks at different depths of the parse tree; for each word, we predict its part of speech as well as the first (parent), second (grandparent) and third level (great-grandparent) constituent labels that appear above it. These predictions are made from representations produced at different depths in networks that are pretrained with one of four objectives: dependency parsing, semantic role labeling, machine translation, or language modeling. In every case, we find a correspondence between network depth and syntactic depth, suggesting that a soft syntactic hierarchy emerges. This effect is robust across all conditions, indicating that the models encode significant amounts of syntax even in the absence of an explicit syntactic training supervision.

Subfields: Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Model Architectures

Citations in the corpus

Dissecting Contextual Word Embeddings: Architecture and Representation (2018, background): “…et al. (2018) studied the role of context in inﬂuencing language model predictions, Gaddy et al. (2018) analyzed neural constituency parsers, Blevins et al. (2018) explored whether RNNs trained with several different objectives can learn hierarchical syntax, and Con-neau et al. (2018)…”
The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives (2019, background): “This observation relates also to the ﬁndings by Blevins et al. (2018).”
Compositionality Decomposed: How do Neural Networks Generalise? (2019, background): “While depth – and, as shown by Blevins et al. (2018), thus hierarchy – can be created by stacking neural layers, the number of layers in popular recurrent sequence-to-sequence setups tends to be limited.”
Analysis Methods in Neural Language Processing: A Survey (2018, background): “Hierarchical representations of syntax were also reported to emerge in other RNN models (Blevins et al., 2018).”
BERT Rediscovers the Classical NLP Pipeline (2019, background): “…evidence that deep language models can encode a range of syntactic and semantic information (e.g. Shi et al., 2016; Belinkov, 2018; Tenney et al., 2019), and that more complex structures are developed hierarchically in the higher layers of the model (Peters et al., 2018b; Blevins et al., 2018).”
What Does BERT Look at? An Analysis of BERT’s Attention (2019, neutral): “Like our work, some of these studies have also demonstrated models capturing aspects of syntax (Shi et al., 2016; Blevins et al., 2018) or coreference (Tenney et al., 2018, 2019; Liu et al., 2019) without explicitly being trained for the tasks.”
What do you learn from context? Probing for sentence structure in contextualized word representations (2019, background): “Recent work has explored many token-level properties of these representations, such as their ability to capture part-of-speech tags (Blevins et al., 2018; Belinkov et al., 2017b; Shi et al., 2016), morphology (Belinkov et al., 2017a;b), or word-sense disambiguation (Peters et al., 2018a).”
A Structural Probe for Finding Syntax in Word Representations (2019, background): “Our work extends the literature on linguistic probes, found at least in (Peters et al., 2018b; Belinkov et al., 2017; Blevins et al., 2018; Hupkes et al., 2018).”
Linguistic Knowledge and Transferability of Contextual Representations (2019, background): “, knowledge of hierarchical syntax (Blevins et al., 2018) or morphology (Belinkov et al.”
Analyzing the Structure of Attention in a Transformer Language Model (2019, background): “This is consistent with previous ﬁndings showing that deeper layers focus on higher-level properties (Blevins et al., 2018; Belinkov, 2018).”
The Importance of Being Recurrent for Modeling Hierarchical Structure (2018, background): “Recent work has shown that recurrent neural networks (RNNs) can implicitly capture and exploit hierarchical information when trained to solve common natural language processing tasks (Blevins et al., 2018) such as language modeling (Linzen et al.”
Visualizing and Measuring the Geometry of BERT (2019, background): “, [1, 6, 10, 20, 24]) point to the existence of similarly rich internal representations of linguistic structure.”
Emergent linguistic structure in artificial neural networks trained by self-supervision (2020, methodology): “This method has been extended (46, 47) to predicting the label of the dependency edge governing a word (e.”
Analyzing Individual Neurons in Pre-trained Language Models (2020, unknown): “…such as machine translation and language modeling, these deep models capture fundamental linguistic phenomena such as word morphology, syntax and various other relevant properties of interest (Shi et al., 2016; Adi et al., 2016; Belinkov et al., 2017a,b; Dalvi et al., 2017; Blevins et al., 2018).”
Discovering the Compositional Structure of Vector Representations with Role Learning Networks (2019, methodology): “In the probing approach, an auxiliary classifier is trained to classify the model’s internal representations based on some linguistically-relevant distinction (Adi et al., 2017; Conneau et al., 2018; Conneau & Kiela, 2018; Belinkov et al., 2017; Blevins et al., 2018; Peters et al., 2018; Tenney et a”
What Happens To BERT Embeddings During Fine-tuning? (2020, background): “Adi et al. (2016); Blevins et al. (2018) tested recurrent networks for sentence length, word context, and syntactic properties, while Conneau et al. (2018); Jawahar et al. (2019) use another 10 probing tasks, including sensitivity to bigram shift.”
RNNs as psycholinguistic subjects: Syntactic state and grammatical dependency (2018, background): “…al., 2016; Gulordava et al., 2018; Wilcox et al., 2018) and complements a variety of other approaches currently practiced (Shi et al., 2016; Belinkov et al., 2018; Blevins et al., 2018; K´ad´ar et al., 2017; Williams et al., 2018; Et-tinger et al., 2017; Lake and Baroni, 2017; Weber et al., 2018).”
Visually Grounded Compound PCFGs (2020, background): “While the produced representation does not reﬂect the structural decisions made by the parser, it can be sensitive to word order and may be affected by its syntactic structure (Blevins et al., 2018).”
Towards a Universal Continuous Knowledge Base (2020, background): “A number of researchers have developed probing methods to evaluate the extent to which continuous representations encode knowledge of interest (Linzen et al., 2016; Belinkov et al., 2017; Blevins et al., 2018; Hewitt and Manning, 2019).”
From Balustrades to Pierre Vinken: Looking for Syntax in Transformer Self-Attentions (2019, background): “Blevins et al. (2018) measure the amount of syntax in RNNs by predicting part-of-speech tags and constituent labels.”