Learning Representations by Maximizing Compression

By Karol Gregor, Yann LeCun. 2011

17 citations recorded by the source corpus.

We give an algorithm that learns a representation of data through compression. The algorithm 1) predicts bits sequentially from those previously seen and 2) has a structure and a number of computations similar to an autoencoder. The likelihood under the model can be calculated exactly, and arithmetic coding can be used directly for compression. When training on digits the algorithm learns filters similar to those of restricted boltzman machines and denoising autoencoders. Independent samples can be drawn from the model by a single sweep through the pixels. The algorithm has a good compression performance when compared to other methods that work under random ordering of pixels.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Signal Processing

Citations in the corpus

Deep AutoRegressive Networks (2013, background): “In NADE (Larochelle & Murray, 2011; Gregor & LeCun, 2011), autoregression is present in the visibles, but only deterministic hidden units are used.”
MADE: Masked Autoencoder for Distribution Estimation (2015, background): “…algorithms for a variety of neural network models have been proposed (Bengio & Bengio, 2000; Larochelle & Murray, 2011; Gregor & LeCun, 2011; Uria et al., 2013; 2014; Kingma & Welling, 2014; Rezende et al., 2014; Bengio et al., 2014; Gregor et al., 2014; Goodfellow et al., 2014; Dinh et al., 2014).”
A Deep and Tractable Density Estimator (2013, background): “Gregor & LeCun (2011) proposed training a variant of the NADE architecture under stochastically generated random orderings.”
Learned Video Compression (2018, background): “In the last two years, we have seen a great surge of ML-based image compression approaches [13, 39, 40, 5, 4, 12, 20, 38, 33, 18, 2, 22, 6, 3, 8, 27, 28].”
Connectionist multivariate density-estimation and its application to speech synthesis (2016, methodology): “In Chapter 3 we introduce the real-valued neural autoregressive density estimator (RNADE), an extension of the neural autoregressive distribution estimator (NADE) (Larochelle and Murray, 2011) , that can model real-valued data.”