Autoencoders, Minimum Description Length and Helmholtz Free Energy

By Geoffrey E. Hinton, Richard S. Zemel. 1993

Subfields: Generative Models, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Generative Models

In Probabilistic & Bayesian Methods

Citations in the corpus

Neural Variational Inference and Learning in Belief Networks (2014, background): “It goes back at least to Hinton & Zemel (1994), who derived the variational objective from the Minimum Description Length (MDL) perspective and used it to train linear au-toencoders.”
Better Mixing via Deep Representations (2012, methodology): “An auto-encoder (LeCun, 1987; Hinton and Zemel, 1994) is parametrized through an encoder function f mapping input-space vector x to representationspace vector h, and a decoder function g mapping representation-space vector h to input-space reconstruction r.”
Masked Autoencoders Are Scalable Vision Learners (2021, methodology): “For example, PCA and k-means are autoencoders [25].”
Minimal Random Code Learning: Getting Bits Back from Compressed Model Parameters (2018, background): “In (Hinton & Zemel, 1994; Frey & Hinton, 1997) coding schemes were proposed which practically exploited the bits-back argument for the purpose of coding data .”
Perceiver IO: A General Architecture for Structured Inputs & Outputs (2021, background): “Autoencoding [32], especially for images, was among the first attempts to provide a representation which can encode and reproduce its inputs.”
Dual Contradistinctive Generative Autoencoder (2020, neutral): “Autoencoder (AE) [42, 26] aims to compactly represent and faithfully reproduce the original input signal by concatenating an encoder and a decoder in an end-to-end learning framework.”
Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning (2020, methodology): “Therefore, we can interpret d as the minimal description length of the task within the framework dictated by the pre-trained representations (Hinton & Zemel, 1993).”
Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning (2020, methodology): “To elaborate, the adapter block consists of 1) a layer normalization (Ba et al., 2016) for an efﬁcient adaptation and 2) a following au-toencoder (Hinton and Zemel, 1994), with a resid-1 ual connection.”
Representation Learning: A Review and New Perspectives (2012, methodology): “In the autoencoder framework [129], [37], [93], one starts by explicitly defining a feature-extracting function in a specific parameterized closed form.”
UvA-DARE (Digital Academic Repository) Self-Supervised Video Representation Learning With Odd-One-Out Networks (neutral): “The most common techniques studied include autoencoders [6, 20], restricted Boltzmann machines [19], convolutional deep belief networks [28], LSTMs and recurrent neural networks [36].”
Tutorial on Variational Autoencoders (2016, background): “Hence, we need to measure the amount of extra information that we get about X when z comes from Q(z|X) instead of from P(z) (for more details, see the “bits back” argument of [19, 20]).”
Isolating Sources of Disentanglement in VAEs (2018, background): “It acts as a complexity penalty on the aggregate posterior which reasonably follows from the minimum description length [21] formulation of the ELBO.”
Continual Learning in Task-Oriented Dialogue Systems (2020, background): “, 2016), followed by a two linear layer (Hinton and Zemel, 1994) with a residual connection.”
The Variational Gaussian Process (2015, methodology): “An inference network provide a ﬂexible parameterization of approximating distributions as used in Helmholtz machines (Hinton & Zemel, 1994), deep Boltzmann machines (Salakhutdinov & Larochelle, 2010), and variational auto-encoders (Kingma & Welling, 2014; Rezende et al., 2014).”