The Importance of Being Recurrent for Modeling Hierarchical Structure

By Ke M. Tran, Arianna Bisazza, Christof Monz. 2018

160 citations recorded by the source corpus.

Recent work has shown that recurrent neural networks (RNNs) can implicitly capture and exploit hierarchical information when trained to solve common natural language processing tasks (Blevins et al., 2018) such as language modeling (Linzen et al., 2016; Gulordava et al., 2018) and neural machine translation (Shi et al., 2016). In contrast, the ability to model structured data with non-recurrent neural networks has received little attention despite their success in many NLP tasks (Gehring et al., 2017; Vaswani et al., 2017). In this work, we compare the two architectures—recurrent versus non-recurrent—with respect to their ability to model hierarchical structure and find that recurrency is indeed important for this purpose. The code and data used in our experiments is available at https://github.com/ ketranm/fan_vs_rnn

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Universal Transformers (2018, methodology): “copying strings or even simple logical inference when the string or formula lengths exceed those observed at training time [28]).”
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned (2019, methodology): “An alternative method to study the ability of language models and machine translation models to capture hierarchical information is to test their sensitivity to speciﬁc grammatical errors (Linzen et al., 2016; Gulordava et al., 2018; Tran et al., 2018; Sennrich, 2017; Tang et al., 2018).”
Analysis Methods in Neural Language Processing: A Survey (2018, methodology): “(Bisazza and Tump, 2018) NMT states Morphology Classification (Tran et al., 2018) RNN / self-attention states Subject-verb agreement Likelihood comparison, direct classification (Wang et al.”
Assessing BERT's Syntactic Abilities (2019, background): “Indeed, Tran et al. (2018) ﬁnds that transformer-based models perform worse than LSTM models on the Linzen et al. (2016) agreement prediction dataset.”
An Analysis of Encoder Representations in Transformer-Based Machine Translation (2018, background): “, 2017), while another line of analysis probes the representation learned by a neural network as input to a classifier of another task (Shi et al., 2016; Adi et al., 2016; Belinkov et al., 2017a; Tran et al., 2018).”
Information-Theoretic Probing with Minimum Description Length (2020, methodology): “…predictions using manually deﬁned templates, either evaluating sensitivity to speciﬁc grammatical errors (Linzen et al., 2016; Gulordava et al., 2018; Tran et al., 2018; Marvin and Linzen, 2018) or understanding what language models know when applying them as knowledge bases or in question…”
Are Multilingual Neural Machine Translation Models Better at Capturing Linguistic Features? (2020, result): “This is consistent with some earlier observations, eg., (Tran et al., 2018) who show that RNN-based models tend to outperform the Transformer in subject-verb agreement.”
Refining Targeted Syntactic Evaluation of Language Models (2021, background): “Table 2: MW, EW, and TSE evaluations on various models and syntactic constructions (See Warstadt et al. (2020); Marvin and Linzen (2018) for descriptions).”
Hierarchical Reinforcement Learning for Open-Domain Dialog (2019, background): “Recurrent models have been shown to be more adept at capturing hierarchical information in language [48].”
Augmenting Neural Machine Translation with Knowledge Graphs (2019, background): “Recently, Tran et al. (2018) concluded that RNN performs better than Transformer on a subject-verb agreement task, while Tang et al. (2018a) found that Transformer models surpass RNN models only in high-resource conditions.”
Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation (2020, methodology): “The subjectverb agreement task is commonly used to evaluate long-range dependencies (Linzen et al., 2016; Tran et al., 2018; Tang et al., 2018), while the WSD task addresses lexical ambiguity phenomena, i.”
One Size Does Not Fit All: Comparing NMT Representations of Different Granularities (2019, methodology): “…Wang et al., 2017), and NMT representations (Shi et al., 2016; Belinkov et al., 2017a), as applied to morphological (Vy-lomova et al., 2017; Dalvi et al., 2017), semantic (Qian et al., 2016b; Belinkov et al., 2017b) and syntactic (Linzen et al., 2016; Tran et al., 2018; Conneau et al., 2018) tasks.”