Recurrent Memory Networks for Language Modeling

By Ke M. Tran, Arianna Bisazza, Christof Monz. 2016

92 citations recorded by the source corpus.

Recurrent Neural Networks (RNN) have obtained excellent result in many natural language processing (NLP) tasks. However, understanding and interpreting the source of this success remains a challenge. In this paper, we propose Recurrent Memory Network (RMN), a novel RNN architecture, that not only amplifies the power of RNN but also facilitates our understanding of its internal functioning and allows us to discover underlying patterns in data. We demonstrate the power of RMN on language modeling and sentence completion tasks. On language modeling, RMN outperforms Long Short-Term Memory (LSTM) network on three large German, Italian, and English dataset. Additionally we perform in-depth analysis of various linguistic dimensions that RMN captures. On Sentence Completion Challenge, for which it is essential to capture sentence coherence, our RMN obtains 69.2% accuracy, surpassing the previous state-of-the-art by a large margin.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

Long Short-Term Memory-Networks for Machine Reading (2016, background): “Tran et al. (2016) combine the LSTM with an external memory block component which interacts with its hidden state.”
The Importance of Being Recurrent for Modeling Hierarchical Structure (2018, background): “Our interest here is the ability of capturing hierarchical structure without being equipped with explicit structural representations (Bowman et al., 2015b; Tran et al., 2016; Linzen et al., 2016).”
Dynamic Entity Representations in Neural Language Models (2017, background): “Unlike the work by Tran et al. (2016), where memory blocks are used to store general contextual information for language modeling, E NTITY NLM assigns each memory block speciﬁcally to only one entity.”
Improving Neural Language Models by Segmenting, Attending, and Predicting the Future (2019, background): “LSTMs with self-attention can improve the performance of language modeling (Tran et al., 2016; Cheng et al., 2016).”
Modeling Recurrence for Transformer (2019, background): “Modeling recurrence is crucial for capturing several essential properties of input sequence, such as structural representations (Tran et al., 2016) and positional encoding (Shaw et al., 2018), which are exactly the weaknesses of SAN (Tran et al., 2018).”
Memory Architectures in Recurrent Neural Network Language Models (2018, background): “These include (i) a soft attention mechanism (Daniluk et al., 2017) and (ii) an explicit memory block or cache model (Tran et al., 2016; Grave et al., 2017).”
Log2Intent: Towards Interpretable User Modeling via Recurrent Semantics Memory Unit (2019, methodology): “First, Ref [31] utilizes memory network for language modeling, while Log2Intent adopts a sequence-to-sequence model for the user modeling task.”
Learning to Remember Translation History with a Continuous Cache (2017, result): “, 2017) and language modeling (Tran et al., 2016; Grave et al., 2017; Daniluk et al., 2017).”
Frustratingly Short Attention Spans in Neural Language Modeling (2017, methodology): “This is in line with observations by Tran et al. (2016).”
Code Completion with Neural Attention and Pointer Networks (2017, background): “The soft attention or memory mechanisms [Bahdanau et al., 2014; Cheng et al., 2016; Tran et al., 2016] have been proposed to ease the gradient vanishing problem in standard RNNs.”