Variational Memory Addressing in Generative Models

By Jörg Bornschein, Andriy Mnih, Daniel Zoran, Danilo Jimenez Rezende. 2017

63 citations recorded by the source corpus.

Aiming to augment generative models with external memory, we interpret the output of a memory module with stochastic addressing as a conditional mixture distribution, where a read operation corresponds to sampling a discrete memory address and retrieving the corresponding content from memory. This perspective allows us to apply variational inference to memory addressing, which enables effective training of the memory module by using the target information to guide memory lookups. Stochastic addressing is particularly well-suited for generative models as it naturally encourages multimodality which is a prominent aspect of most high-dimensional datasets. Treating the chosen address as a latent variable also allows us to quantify the amount of information gained with a memory lookup and measure the contribution of the memory module to the generative process. To illustrate the advantages of this approach we incorporate it into a variational autoencoder and apply the resulting model to the task of generative few-shot learning. The intuition behind this architecture is that the memory module can pick a relevant template from memory and the continuous part of the model can concentrate on modeling remaining variations. We demonstrate empirically that our model is able to identify and access the relevant memory contents even with hundreds of unseen Omniglot characters in memory

Subfields: Artificial Intelligence, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Generative Models

Citations in the corpus

Few-shot Autoregressive Density Estimation: Towards Learning to Learn Distributions (2017, methodology): “Bornschein et al. (2017) refers to Variational Memory Addressing and Gregor et al.”
Conditional Neural Processes (2018, methodology): “, 2017) or including a memory unit in a VAE model (Bornschein et al., 2017).”
Sample Efficient Adaptive Text-to-Speech (2018, background): “In generative modeling, few-shot learning has been addressed from several perspectives, including matching networks (Bartunov and Vetrov, 2017) and variable inference for memory addressing (Bornschein et al., 2017).”
Neural Processes (2018, background): “Related are generative meta-learning approaches that carry out few-shot estimation of the data densities (van den Oord et al., 2016; Reed et al., 2017; Bornschein et al., 2017; Rezende et al., 2016).”
VAE with a VampPrior (2017, background): “Concurrently to our work, in [4] a variational VAE with a memory was proposed.”
The Kanerva Machine: A Generative Distributed Memory (2018, methodology): “We generalise Kanerva’s original model through learnable addresses and re-parametrised latent variables (Rezende et al., 2014; Kingma & Welling, 2013; Bornschein et al., 2017).”
Attentive Neural Processes (2019, methodology): “Few-shot density estimation using attention has also been explored extensively in numerous works (Rezende et al., 2016; Reed et al., 2017; Bornschein et al., 2017; Bartunov & Vetrov, 2018).”
Meta-Amortized Variational Inference and Learning (2019, background): “(2016)), memory-augmented models (Bornschein et al. (2017)), weight-updates for conditional generative models (Reed et al. (2017)), and hierarchical models (Edwards & Storkey (2016), Hewitt et al.”
DVAE#: Discrete Variational Autoencoders with Relaxed Boltzmann Priors (2018, methodology): “Further, models such as clustering, semi-supervised learning, and variational memory addressing [11] all require discrete variables, which makes the training of discrete models an important challenge.”
Latent Alignment and Variational Attention (2018, background): “The term "variational attention" has been used to refer to a different component the output from attention (commonly called the context vector) as a latent variable [7], or to model both the memory and the alignment as a latent variable [9].”