Auto-encoders: reconstruction versus compression
By Yann Ollivier. 2014
23 citations recorded by the source corpus.
We discuss the similarities and differences between training an auto-encoder to minimize the reconstruction error, and training the same auto-encoder to compress the data via a generative model. Minimizing a codelength for the data using an auto-encoder is equivalent to minimizing the reconstruction error plus some correcting terms which have an interpretation as either a denoising or contractive property of the decoding function. These terms are related but not identical to those used in denoising or contractive auto-encoders [Vincent et al. 2010, Rifai et al. 2011]. In particular, the codelength viewpoint fully determines an optimal noise level for the denoising criterion.
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
- Yoshua Bengio
- Bernhard Schölkopf
- David Lopez-Paz
- Luc Van Gool
- Sergey Levine
- Dacheng Tao
- Trevor Darrell
- Bernt Schiele
Citations in the corpus
- Adversarial Vulnerability of Neural Networks Increases With Input Dimension (2018, background): “In the same vein, a gradient-regularization of the loss of generative models also appears in Proposition 6 of Ollivier (2014), where it stems from a code-length bound on the data (minimum description length).”
- Lossy Image Compression with Compressive Autoencoders (2017, positive): “in x and the normalization constant does not depend on y, or otherwise Cwill not be constant. An direct empirical comparison of our approach with variational autoencoders is provided in Appendix A.5. Ollivier (2015) discusses variational autoencoders for lossless compression as well as connections t”
- The Description Length of Deep Learning models (2018, background): “In unsupervised learning, autoen-coders and especially variational autoencoders (Kingma and Welling, 2013) are compression meth-ods of the data (Ollivier, 2014).”
- First-Order Adversarial Vulnerability of Neural Networks and Input Dimension (2018, background): “A gradient regularization of the loss of generative models also appears in Proposition 6 of Ollivier (2014), where it stems from a code-length bound on the data (minimum description length).”