The Kanerva Machine: A Generative Distributed Memory

By Yan Wu, Greg Wayne, Alex Graves, Timothy P. Lillicrap. 2018

41 citations recorded by the source corpus.

We present an end-to-end trained memory system that quickly adapts to new data and generates samples like them. Inspired by Kanerva's sparse distributed memory, it has a robust distributed reading and writing mechanism. The memory is analytically tractable, which enables optimal on-line compression via a Bayesian update-rule. We formulate it as a hierarchical conditional generative model, where memory provides a rich data-dependent prior distribution. Consequently, the top-down memory and bottom-up perception are combined to produce the code representing an observation. Empirically, we demonstrate that the adaptive memory significantly improves generative models trained on both the Omniglot and CIFAR datasets. Compared with the Differentiable Neural Computer (DNC) and its variants, our memory model has greater capacity and is significantly easier to train.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sergey Levine
2. Yoshua Bengio
3. Ruslan Salakhutdinov
4. Pieter Abbeel
5. Quoc V. Le
6. Mohit Bansal
7. J. Urgen Schmidhuber
8. Kyunghyun Cho

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Trevor Darrell
4. Bernt Schiele
5. Xiaogang Wang
6. Ming-Hsuan Yang
7. Jitendra Malik
8. Andrew Zisserman

In Generative Models

1. Aäron van den Oord
2. Timothy P. Lillicrap
3. David R Ha
4. J. Urgen Schmidhuber
5. Danilo Jimenez Rezende
6. Karol Gregor
7. Fei Ye
8. Adrian G. Bors

In Signal Processing

1. J. Urgen Schmidhuber
2. David R Ha
3. Karol Gregor
4. Yan Wu
5. Danilo Jimenez Rezende
6. Aäron van den Oord
7. Frederic Besse
8. Michael C. Mozer

Citations in the corpus

• Recurrent World Models Facilitate Policy Evolution (2018, methodology): “Future work will explore replacing the VAE and MDN-RNN with higher capacity models [27, 89, 93, 97, 98], or incorporating an external memory module [19, 107], if we want our agent to learn to explore more complicated worlds.”
• Shaping Belief States with Generative Environment Models for RL (2019, methodology): “The main contributions of this paper are: 1) We demonstrate that an expressive generative model of rich 3D environments can be learned from merely first-person views and actions to capture long-term consistency; 2) we provide an analysis of three different belief-state architectures (LSTM [16], LSTM ”
• Lifelong Generative Modeling (2017, background): “such as the Kanerva Machine [133] and its extensions [134] provide a principled way to do this in the VAE setting.”
• Convolutional Bipartite Attractor Networks (2019, background): “First, in both early [12, 13] and recent [5, 24, 41, 42] work, ANs are characterized as content-addressable memories: activation vectors are stored and can later be retrieved with only partial information.”
• Learning Attractor Dynamics for Generative Memory (2018, methodology): “Our model shares the same essential structure as the Kanerva Machine (figure 1, left) [32], which views memory as a global latent variable in a generative model.”
• Continual Learning of Large Language Models: A Comprehensive Survey (2024, unknown): “While regularization-based and architectural-based methods require additional parameter storage and GPU memory, together with replay-based methods they remain for CIT due to the simplicity and effectiveness [243].”