Learning Attractor Dynamics for Generative Memory

By Yan Wu, Greg Wayne, Karol Gregor, Timothy P. Lillicrap. 2018

18 citations recorded by the source corpus.

A central challenge faced by memory systems is the robust retrieval of a stored pattern in the presence of interference due to other stored patterns and noise. A theoretically well-founded solution to robust retrieval is given by attractor dynamics, which iteratively cleans up patterns during recall. However, incorporating attractor dynamics into modern deep learning systems poses difficulties: attractor basins are characterised by vanishing gradients, which are known to make training neural networks difficult. In this work, we exploit recent advances in variational inference and avoid the vanishing gradient problem by training a generative distributed memory with a variational lower-bound-based Lyapunov function. The model is minimalistic with surprisingly few parameters. Experiments shows it converges to correct patterns upon iterative retrieval and achieves competitive performance as both a memory model and a generative model.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Shaping Belief States with Generative Environment Models for RL (2019, methodology): “For such mechanism we use Kanerva machines [17], speciﬁcally the version in [65].”
Lifelong Generative Modeling (2017, background): “such as the Kanerva Machine [58] and its extensions [59] provide a principled way to do this in the VAE setting.”
Convolutional Bipartite Attractor Networks (2019, background): “First, in both early [12, 13] and recent [5, 24, 41, 42] work, ANs are characterized as content-addressable memories: activation vectors are stored and can later be retrieved with only partial information.”