Relational recurrent neural networks

By Adam Santoro, David Raposo, Jack W. Rae, Mike Chrzanowski, Theophane Weber, Daan Wierstra, Oriol Vinyals, Razvan Pascanu, Timothy P. Lillicrap. 2018

222 citations recorded by the source corpus.

Memory-based neural networks model temporal data by leveraging an ability to remember information for long periods. It is unclear, however, whether they also have an ability to perform complex relational reasoning with the information they remember. Here, we first confirm our intuitions that standard memory architectures may struggle at tasks that heavily involve an understanding of the ways in which entities are connected -- i.e., tasks involving relational reasoning. We then improve upon these deficits by using a new memory module -- a \textit{Relational Memory Core} (RMC) -- which employs multi-head dot product attention to allow memories to interact. Finally, we test the RMC on a suite of tasks that may profit from more capable relational reasoning across sequential information, and show large gains in RL domains (e.g. Mini PacMan), program evaluation, and language modeling, achieving state-of-the-art results on the WikiText-103, Project Gutenberg, and GigaWord datasets.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

Compressive Transformers for Long-Range Sequence Modelling (2019, neutral): “(2016) 32 33 RMC (Santoro et al., 2018) 30.”
Object-Centric Learning with Slot Attention (2020, background): “Notably, RMC [63] and RIM [67] introduce an attention mechanism to aggregate information from inputs to slots.”
Perceiver IO: A General Architecture for Structured Inputs & Outputs (2021, background): “Cross-attention itself is in widespread use as a mechanism for allowing information from a source to influence processing of a target domain of a different size or structure [18, 20, 53, 88, 64, 68, 34, 51].”
Perceiver: General Perception with Iterative Attention (2021, background): “…architectures with attention to the longer-horizon past (Rae et al., 2020; Dai et al., 2019) and to produce architectures that write to and/or read from ﬁxed-size arrays or memories (Santoro et al., 2018; Goyal et al., 2021), all while keeping the cost of each operation linear in the input size.”
Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solving (2019, methodology): “Previous work used Multi-Head Attention also in recurrent neural networks in order to perform complex relational reasoning (Santoro et al., 2018).”
Analysing Mathematical Reasoning Abilities of Neural Models (2019, methodology): “Santoro et al. (2018b) use Raven’s progressive matrix puzzles to measure abstract reasoning of networks.”
On the Binding Problem in Artificial Neural Networks (2020, methodology): “…and has been successfully applied for relational reasoning about objects (Zambaldi et al., 2019; van Steenkiste et al., 2020; Goyal et al., 2019; Santoro et al., 2018a), citation networks (Veličković et al., 2018), question answering (Dehghani et al., 2019), and language modeling (Devlin et…”
Object Files and Schemata: Factorizing Declarative and Procedural Knowledge in Dynamical Systems (2020, methodology): “re details please refer to the appendix section A. Baselines: We compare the proposed method with RMC, a memory based relational recurrent model with attention between saved memory and hidden states [Santoro et al., 2018]. We also compare the proposed method to Recurrent Independent Mechanisms (RIMs”
Recurrent Independent Mechanisms (2019, methodology): “ated. In our use of key-value attention, the queries come from the RIMs, while the keys and values come from the current input. The mechanics of this attention mechanism follow (Vaswani et al., 2017; Santoro et al., 2018), with the modiﬁcation that the parameters of the attention mechanism itself ar”
Fitted Q-Learning for Relational Domains (2020, background): “While this work captures the relation existing in structured domain, they are implicitly captured by the attention aware Neural network architecture (Santoro et al. 2018) and therefore relations are not speciﬁed before learning.”
Stabilizing Transformers for Reinforcement Learning (2019, background): “Despite progress at designing more expressive memory architectures (Graves et al., 2016; Wayne et al., 2018; Santoro et al., 2018) that perform better than LSTMs in memory-based tasks and partially-observable environments, they have not seen widespread adoption in RL agents perhaps due to their…”
Deep Equilibrium Models (2019, neutral): “1GB Relational Memory Core [40] 195M 60M 31.”
Coordination Among Neural Modules Through a Shared Global Workspace (2021, methodology): “9, we study the performance of the proposed model compared with LSTM, RIMs and RMC.”
Inductive biases for deep learning of higher-level cognition (2020, background): “te-of-the art NLP systems (Devlin et al., 2018; Brown et al., 2020) and is the standard tool for memory-augmented neural networks (Graves et al., 2014; Sukhbaatar et al., 2015; Gulcehre et al., 2016; Santoro et al., 2018). Attention and memory can also help address the problem of credit assignment t”
Adaptive Posterior Learning: few-shot learning with a surprise-based memory module (2019, methodology): “• Relational working memory decoder (Santoro et al., 2018) The relational working memory module (figure 2, right) takes in the concatenated neighbor embeddings and corresponding label embeddings as its initial memory state.”
Natural language processing: state of the art, current trends and challenges (2017, background): “Later the use of CNNs can be observed in tackling problems associated with NLP tasks like Sentence Classification [127], Sentiment Analysis [135], Text Classification [118], Text Summarization [158], Machine Translation [70] and Answer Relations [150].”
A Review of Speaker Diarization: Recent Advances with Deep Learning (2021, methodology): “However, traditional DL architectures may fail when the problem involves relational information between observations [155].”
CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Text (2019, methodology): “Relational Recurrent Network (RMC) (Santoro et al., 2018): We also implemented RMC, a recently proposed model for relational reasoning.”
Learning Distributed Representations of Symbolic Structure Using Binding and Unbinding Operations (2018, background): “ global geometrical constraint on the representation space, such as hyperbolic embedding [21, 42], and introducing inductive biases into the architecture of networks, including Relational Memory Core [30] and neural-symbolic computing methods [3]. The latter category subdivides into models that inse”