Denoising Criterion for Variational Auto-Encoding Framework

By Daniel Jiwoong Im, Sungjin Ahn, Roland Memisevic, Yoshua Bengio. 2015

224 citations recorded by the source corpus.

Denoising autoencoders (DAE) are trained to reconstruct their clean inputs with noise injected at the input level, while variational autoencoders (VAE) are trained with noise injected in their stochastic hidden layer, with a regularizer that encourages this noise injection. In this paper, we show that injecting noise both in input and in the stochastic hidden layer can be advantageous and we propose a modified variational lower bound as an improved objective function in this setup. When input is corrupted, then the standard VAE lower bound involves marginalizing the encoder conditional distribution over the input noise, which makes the training criterion intractable. Instead, we propose a modified training criterion which corresponds to a tractable bound when input is corrupted. Experimentally, we find that the proposed denoising variational autoencoder (DVAE) yields better average log-likelihood than the VAE and the importance weighted autoencoder on the MNIST and Frey Face datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Stefano Ermon
2. Sergey Levine
3. Yoshua Bengio
4. Jianfei Cai
5. Quoc V. Le
6. Kyunghyun Cho
7. Mohit Bansal
8. Percy Liang

In Computer Vision and Pattern Recognition

1. Jianfei Cai
2. G. Wang
3. Luc Van Gool
4. Ming-Hsuan Yang
5. Dacheng Tao
6. Alan L. Yuille
7. Yi Yang
8. Trevor Darrell

In Statistical and Nonlinear Physics

1. Jordi Bonada
2. Merlijn Blaauw
3. Shengjia Zhao
4. Daniel Jiwoong Im
5. Roland Memisevic
6. Graham W. Taylor
7. Hung Hai Bui
8. Jianfei Cai

In Generative Models

1. Shengjia Zhao
2. Graham W. Taylor
3. He Ma
4. Stefano Ermon
5. Kristin Branson
6. Jordi Bonada
7. Merlijn Blaauw
8. Daniel Jiwoong Im

Citations in the corpus

• Generating images with recurrent adversarial networks (2016, methodology): “One interesting aspect of GRAN is that the procedure of GRAN starts with a decoder instead of an encoder.”
• Recent advances in convolutional neural networks (2015, methodology): “In this section, we introduce some fast processing methods of CNNs.”
• Amortized Inference Regularization (2018, background): “However, our treatment of DVAE differs from [17]’s in both theoretical analysis and underlying motivation.”
• Quantitatively Evaluating GANs With Divergences Proposed for Training (2018, methodology): “Alternatively, we can also try to evaluate other log-likelihood-based models like NVIL (Mnih & Gregor, 2014), VAE (Kingma & Welling, 2014), DVAE (Im et al., 2015), DRAW (Gregor et al., 2015), RBMs (Hinton et al., 2006; Salakhutdinov & Hinton, 2009), NICE (Dinh et al., 2014), etc.”
• AAVAE: Augmentation-Augmented Variational Autoencoders (2021, background): “[36] studied this combination with a simple corruption distribution that is agnostic to the input domain in the context of density estimation.”
• Adversarial and Contrastive Variational Autoencoder for Sequential Recommendation (2021, background): “There are also some variants of VAE, such as β-VAE [7] which learns disentangled representations and DVAE [10] which is similar to denoising autoencoder.”
• Modeling and Transforming Speech Using Variational Autoencoders (2016, methodology): “Additionally, we can make reconstruction more difficult for the network by using the denoising VAE objective [25],”
• A Multimodal Anomaly Detector for Robot-Assisted Feeding Using an LSTM-Based Variational Autoencoder (2017, background): “in (1) with a denoising variational lower bound Ldvae [30],”