Memory Architectures in Recurrent Neural Network Language Models
By Dani Yogatama, Yishu Miao, Gábor Melis, Wang Ling, A. Kuncoro, Chris Dyer, Phil Blunsom. 2018
66 citations recorded by the source corpus.
Subfields: Natural Language Processing, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Natural Language Processing
- Graham Neubig
- Mohit Bansal
- Richard Socher
- Kyunghyun Cho
- Jianfeng Gao
- Caiming Xiong
- Luke S. Zettlemoyer
- Christopher D. Manning
In Model Architectures
- Richard Socher
- Caiming Xiong
- Kyunghyun Cho
- Yoshua Bengio
- Chris Dyer
- Graham Neubig
- Quoc V. Le
- Oriol Vinyals
Citations in the corpus
- Character-Level Language Modeling with Deeper Self-Attention (2018, background): “Giving RNNs access to external memory has also shown improvements in the language modeling domain [17, 47].”
- Universal Transformers (2018, methodology): “Model Number of attractors 0 1 2 3 4 5 Total Previous best results [33]: Best Stack-RNN 0.”
- Colorless Green Recurrent Networks Dream Hierarchically (2018, neutral): “Linzen’s dataset was recently reported by Yogatama et al. (2018).”
- LSTMs Can Learn Syntax-Sensitive Dependencies Well, But Modeling Structure Makes Them Better (2018, positive): “Our finding is consistent with the recent work of Yogatama et al. (2018), who find that introducing elements of hierarchical modeling through a stackstructured memory is beneficial for number agreement, outperforming LSTM language models and attention-augmented variants by increasing margins as the ”
- AdaFrame: Adaptive Frame Selection for Fast Video Recognition (2018, background): “However, learning where to look next is difficult due to the huge search space and limited capacity of hidden states [1, 37] to remember input history.”
- O N M EMORY IN H UMAN AND A RTIFICIAL L ANGUAGE P ROCESSING S YSTEMS (2020, background): “…processing, e.g. , Memory Networks (Weston et al., 2015; Sukhbaatar et al., 2015), stack-augmented neural networks (Joulin & Mikolov, 2015; Grefenstette et al., 2015; Yogatama et al., 2018), and cache-based sequence models (Graves et al., 2014; Merity et al., 2017; Khandelwal et al., 2020).”
- Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks (2018, background): “More recently, Yogatama et al. (2018) implicitly learned structure by using a stack-like memory.”
- Can Recurrent Neural Networks Learn Nested Recursion? (2018, background): “Yogatama et al. (2018) have investigated the effect of various memory architectures, including stacks, on the long-distance dependency task, for up to 5 attractors.”