MEMO: A Deep Network for Flexible Combination of Episodic Memories

By Andrea Banino, Adrià Puigdomènech Badia, Raphael Köster, Martin Chadwick, Vinícius Flores Zambaldi, Demis Hassabis, C. Barry, Matthew Botvinick, Dharshan Kumaran, Charles Blundell. 2020

40 citations recorded by the source corpus.

Recent research developing neural network architectures with external memory have often used the benchmark bAbI question and answering dataset which provides a challenging number of tasks requiring reasoning. Here we employed a classic associative inference task from the human neuroscience literature in order to more carefully probe the reasoning capacity of existing memory-augmented architectures. This task is thought to capture the essence of reasoning -- the appreciation of distant relationships among elements distributed across multiple facts or memories. Surprisingly, we found that current architectures struggle to reason over long distance associations. Similar results were obtained on a more complex task involving finding the shortest path between nodes in a path. We therefore developed a novel architecture, MEMO, endowed with the capacity to reason over longer distances. This was accomplished with the addition of two novel components. First, it introduces a separation between memories/facts stored in external memory and the items that comprise these facts in external memory. Second, it makes use of an adaptive retrieval mechanism, allowing a variable number of ‘memory hops’ before the answer is produced. MEMO is capable of solving our novel reasoning tasks, as well as all 20 tasks in bAbI.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence (2020, background): “…all great engineering accomplishments of humankind have rested on some sort of symbolic reasoning, and the evidence that humans make use of 6 DeepMind's interesting new MEMO architecture (Banino et al., 2020) comes close to representing a database of records. them in everyday cognition is immense.”
Towards mental time travel: a hierarchical memory for reinforcement learning agents (2021, methodology): “Figure 2: We evaluated our model across a wide variety of task domains, some of which are modified or used from other recent memory papers [19, 21, 2, 48].”
PonderNet: Learning to Ponder (2021, methodology): “For MEMO and for Universal transformer the results were taken from Banino et al. (2020).”
Hopfield Networks is All You Need (2020, methodology): “MEMO uses the transformer attention mechanism for reasoning over longer distances [8].”