Towards Conceptual Compression

By Karol Gregor, Frederic Besse, Danilo Jimenez Rezende, Ivo Danihelka, Daan Wierstra. 2016

258 citations recorded by the source corpus.

We introduce a simple recurrent variational auto-encoder architecture that significantly improves image modeling. The system represents the state-of-the-art in latent variable models for both the ImageNet and Omniglot datasets. We show that it naturally separates global conceptual information from lower level details, thus addressing one of the fundamentally desired properties of unsupervised learning. Furthermore, the possibility of restricting ourselves to storing only global information about an image allows us to achieve high quality 'conceptual compression'.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Denoising Diffusion Probabilistic Models (2020, background): “Perhaps these are hints of conceptual compression [16].”
Few-shot Autoregressive Density Estimation: Towards Learning to Learn Distributions (2017, background): “Rezende et al. (2016) develop a sequential generative model for few-shot learning, generalizing the Deep Recurrent Attention Writer (DRAW) model (Gregor et al., 2015).”
Neural Discrete Representation Learning (2017, methodology): “Our numbers for the continuous VAE are comparable to those reported for a Deep convolutional VAE: 4.54 bits/dim [13] on this dataset.”
VQ-DRAW: A Sequential Discrete VAE (2020, methodology): “While DRAW itself uses attention, convolutional DRAW [9] maintains the same sequential architecture but opts for an attention-free approach.”
BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling (2019, neutral): “With autoregressive components C ONV DRAW [12] 3 .”
PixelSNAIL: An Improved Autoregressive Generative Model (2017, neutral): “Method CIFAR-10 ImageNet 32 × 32 Conv DRAW [4] 3.5 4.40 Real NVP [2] 3.49 4.28 VAE with IAF [5] 3.11 – PixelRNN [8] 3.00 3.86 PixelCNN [12] 3.03 3.83 Image Transformer [1] 2.98 3.81 PixelCNN++ [11] 2.92 – Block Sparse PixelCNN++ [9] 2.90 – PixelSNAIL (ours) 2.85 3.80”
PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications (2017, methodology): “5.40 NICE (Dinh et al., 2014) 4.48 DRAW (Gregor et al., 2015) 4.13 Deep GMMs (van den Oord & Dambre, 2015) 4.00 Conv DRAW (Gregor et al., 2016) 3.58 Real NVP (Dinh et al., 2016) 3.49 PixelCNN (van den Oord et al., 2016b) 3.14 VAE with IAF (Kingma et al., 2016) 3.11 Gated PixelCNN (van den Oord et…”
Conditional Image Generation with PixelCNN Decoders (2016, background): “64x64 Model NLL Test (Train) Conv Draw: [8] 4.”
Improved Variational Inference with Inverse Autoregressive Flow (2016, background): “(van den Oord et al., 2016) 4.70 NICE (Dinh et al., 2014) 4.48 Deep GMMs (van den Oord and Schrauwen, 2014) 4.00 Real NVP (Dinh et al., 2016) 3.49 Pixel CNN van den Oord et al. (2016) 3.14 Pixel RNN (Row LSTM) van den Oord et al. (2016) 3.07 Pixel RNN (Diagonal BiLSTM) van den Oord et al. (2016) 3.0”
Density estimation using Real NVP (2016, background): “Dataset PixelRNN [46] Real NVP Conv DRAW [22] IAF-VAE [34] CIFAR-10 3.”
NVAE: A Deep Hierarchical Variational Autoencoder (2020, background): “38 Conv Draw [42] - 3.”
Comparison of Maximum Likelihood and GAN-based training of Real NVPs (2017, background): “Gregor et al. (2016) and Dinh et al. (2016) visualized ‘conceptual compression’ by inferring the latent variables and then resampling a part of the latent variables from the normal N (0 , 1) prior.”
Variational Lossy Autoencoder (2016, neutral): “Kingma et al. (2016), Kaae Sønderby et al. (2016), Gregor et al. (2016) and Salimans (2016) explored VAE architecture with an explicitly deep autoregressive prior for continuous latent variables, but the autoregressive data likelihood is intractable in those architectures and needs to inferred…”
PixelVAE: A Latent Variable Model for Natural Images (2016, neutral): “Model NLL Validation (Train) FLOPs Convolutional DRAW (Gregor et al., 2016) ≤ 4.”
End-to-end Optimized Image Compression (2016, background): “Gregor, Besse, et al. (2016) introduce an interesting hierarchical representation of images, in which degradations are more natural looking than those of linear representations.”
Towards a Neural Statistician (2016, methodology): “Our approach can easily beneﬁt from advances in deep generative models by upgrading our base generative model, and so in the future we could for instance use a recurrent generative model as in [8], [28], [9].”
Shaping Belief States with Generative Environment Models for RL (2019, methodology): “We found that a modiﬁed ConvDRAW in combination with GECO [28] works well in practice, which allows us to learn stable belief-states while maintaining good sample quality.”
Variational Autoencoders Pursue PCA Directions (by Accident) (2018, background): “The generative model is predominantely exploited in computer vision [25, 15, 22, 16] with notable exceptions such as generating combinatorial graphs [26].”