Few-shot Autoregressive Density Estimation: Towards Learning to Learn Distributions

By Scott E. Reed, Yutian Chen, T. Paine, Aäron van den Oord, S. M. Ali Eslami, Danilo Jimenez Rezende, Oriol Vinyals, Nando de Freitas. 2017

90 citations recorded by the source corpus.

Deep autoregressive models have shown state-of-the-art performance in density estimation for natural images on large-scale datasets such as ImageNet. However, such models require many thousands of gradient-based weight updates and unique image examples for training. Ideally, the models would rapidly learn visual concepts from only a handful of examples, similar to the manner in which humans learns across many vision tasks. In this paper, we show how 1) neural attention and 2) meta learning techniques can be used in combination with autoregressive models to enable effective few-shot density estimation. Our proposed modifications to PixelCNN result in state-of-the art few-shot density estimation on the Omniglot dataset. Furthermore, we visualize the learned attention policy and find that it learns intuitive algorithms for simple tasks such as image mirroring on ImageNet and handwriting on Omniglot without supervision. Finally, we extend the model to natural images and demonstrate few-shot image generation on the Stanford Online Products dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Representation & Self-Supervised Learning

Citations in the corpus

Conditional Neural Processes (2018, methodology): “Another way of carrying out few-shot density estimation is by updating existing models like Pix-elCNN (van den Oord et al., 2016) and augmenting them with attention mechanisms (Reed et al., 2017) or including a memory unit in a VAE model (Bornschein et al., 2017).”
Sample Efficient Adaptive Text-to-Speech (2018, methodology): “Rezende et al. (2016) developed a sequential generative model that extended the Deep Recurrent Attention Writer (DRAW) model (Gregor et al., 2015), and Reed et al. (2018) extended PixelCNN (Van Oord et al., 2016) with neural attention for few-shot auto-regressive density modeling.”
Neural Voice Cloning with a Few Samples (2018, methodology): “Recently, deep neural networks achieve great successes in few-shot density estimation and conditional image generation [e.g., Rezende et al., 2016, Reed et al., 2017, Azadi et al., 2017], because of the great potential for composition in their learned representation.”
Neural Processes (2018, background): “Related are generative meta-learning approaches that carry out few-shot estimation of the data densities (van den Oord et al., 2016; Reed et al., 2017; Bornschein et al., 2017; Rezende et al., 2016).”
Meta-Learning: A Survey (2018, neutral): “[127] propose a few-shot approach for density estimation (DE).”
Meta-Learning in Neural Networks: A Survey (2020, background): “Gradient-based meta-learning of PixelCNN generators was shown to enable their few-shot learning [217].”
Learning Adaptive Classifiers Synthesis for Generalized Few-Shot Learning (2019, background): “L has achieved promising results in various domains such as visual recognition [13, 38, 57], domain adaptation [14, 24], neural machine translation [19], data compression [64], and density estimation [48]. Empirical studies of FSL could be found in [11,58]. FSL emphasizes on building models of the u”
Generalizing from a Few Examples (2019, methodology): “] a hierarchy of θ as one of Dtrain c ’s object recognition HDP-DBM [117] a hierarchy of θ as one of Dtrain c ’s object recognition latent variables SeqGen[96] θ as input generation Attention PixelCNN[94] θ as input image flipping, generation Neural Statistician [31] θ as input classification, gener”
Meta-Amortized Variational Inference and Learning (2019, unknown): “The focus has been on few-shot density estimation, with approaches ranging from the use of conditioning [24] to nested optimization [25].”
Semantics-Driven Generative Replay for Few-Shot Class Incremental Learning (2022, result): “Apart from GANs, several studies utilized other models like VAE [7], auto-regressive models [29], and recurrent generators [30], respectively.”
Compositional Visual Generation with Energy Based Models (2020, background): “This view of few-shot concept learning and generation is similar to work of [23], with the distinction that instead of learning to generate holistic images from few examples, we learn factors from examples, which can be composed with other factors.”
How to Train Your MAML to Excel in Few-Shot Classification (2021, background): “Recent years have witnessed an abundance of new approaches on meta-learning [29, 30, 48, 61, 63, 64, 68, 77, 79, 80, 94, 98], developed in various areas including few-shot learning [70, 85, 89, 105, 107, 110, 111, 115], optimization [3, 9, 53, 108], reinforcement and imitation learning [23, 24, 31, ”
Adaptive Posterior Learning: few-shot learning with a surprise-based memory module (2019, background): “Some meta learning models generate predictions in an autoregressive fashion by predicting the next target point from the entire prior sequence of consecutive observations (Reed et al., 2018; Mishra et al., 2018).”
The Variational Homoencoder: Learning to learn high capacity generative models from few examples (2018, unknown): “…from a set D as input, and produce a distribution over new elements p ( x | D ) by some learning procedure, expressed either as a neural network (Rezende et al., 2016; Bartunov & Vetrov, 2016; Reed et al., 2017) or by adapting the parameters of an unconditional model (Reed et al., 2017).”
Encoding Musical Style with Transformer Autoencoders (2019, background): “For future work, it would be interesting to explore other training procedures such as variational techniques or few-shot learning approaches (Finn et al., 2017; Reed et al., 2017) to account for situations in which the input signals are from slightly different data distributions than the training se”
SketchEmbedNet: Learning Novel Concepts by Imitating Drawings (2020, methodology): “One-shot generations are sampled from Omniglot (Lake et al., 2015) and are visually compared with other few-and one-shot generation methods (Rezende et al., 2016; Reed et al., 2018) (Figure 8).”