Learning to Generate Samples from Noise through Infusion Training

By Florian Bordes, Sina Honari, Pascal Vincent. 2017

44 citations recorded by the source corpus.

In this work, we investigate a novel training procedure to learn a generative model as the transition operator of a Markov chain, such that, when applied repeatedly on an unstructured random noise sample, it will denoise it into a sample that matches the target distribution from the training set. The novel training procedure to learn this progressive denoising operation involves sampling from a slightly different chain than the model chain used for generation in the absence of a denoising target. In the training chain we infuse information from the training target example that we would like the chains to reach with a high probability. The thus learned transition operator is able to produce quality and varied samples in a small number of steps. Experiments show competitive results compared to the samples generated with a basic Generative Adversarial Net

Subfields: Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Generative Models

Citations in the corpus

Denoising Diffusion Probabilistic Models (2020, methodology): “Other methods for learning transition operators of Markov chains include infusion training [2], variational walkback [13], generative stochastic networks [1], and others [47, 51, 33, 39].”
Generative Modeling by Estimating Gradients of the Data Distribution (2019, methodology): “Previous approaches such as Infusion Training (IT [5]) and Variational Walkback (VW [16]) also employed different noise levels/temperatures for training transition operators of a Markov chain.”
Variational Walkback: Learning a Transition Operator as a Stochastic Recurrent Net (2017, background): “There’s also another related recent approach to learning a transition operator with a denoising cost, developed in parallel, called Infusion training (Bordes et al., 2017), which tries to reconstruct the target data in the chain, instead of the previous step in the destructive chain.”
A-NICE-MC: Adversarial Training for MCMC (2017, methodology): “This is drastically different from other iterative generative models trained with maximum likelihood, such as Generative Stochastic Networks (GSN, [18]) and Infusion Training (IF, [19]), because when we train T✓(xt+1|xt) we are not specifying a particular target for xt+1.”
Score-Based Generative Modeling through Stochastic Differential Equations (2020, neutral): “Score-based generative models, and related techniques (Bordes et al., 2017; Goyal et al., 2017; Du & Mordatch, 2019), have proven effective at generation of images (Song & Ermon, 2019; 2020; Ho et al., 2020), audio (Chen et al., 2020; Kong et al., 2020), graphs (Niu et al., 2020), and shapes (Cai ˚…”
Stacked Generative Adversarial Networks (2016, background): “Serge Belongie is supported in part by a Google Focused Research Award.”
Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement (2018, background): “Training For each training pair ( X, Y ∗ ) , we ﬁrst approximate the marginal log-probability.”
Towards Deeper Understanding of Variational Autoencoding Models (2017, methodology): “Existing techniques can model fairly complex datasets, including natural images and speech (Radford et al., 2015; Arjovsky et al., 2017; van den Oord et al., 2016b;a; Bordes et al., 2016; Kingma et al., 2016; Gulrajani et al., 2016; Bowman et al., 2015; Chung et al., 2015).”
Optimizing the Latent Space of Generative Networks (2017, methodology): “Finally, the idea of mapping between images and noise to learn generative models is a well known technique (Chen & Gopinath, 2000; Laparra et al., 2011; Sohl-Dickstein et al., 2015; Bordes et al., 2017).”
Tackling the Generative Learning Trilemma with Denoising Diffusion GANs (2021, background): “Diffusion-based models (Sohl-Dickstein et al., 2015; Ho et al., 2020) learn the finite-time reversal of a diffusion process, sharing the idea of learning transition operators of Markov chains with Goyal et al. (2017); Alain et al. (2016); Bordes et al. (2017).”
Diffusion Normalizing Flow (2021, methodology): “When the forward process is modeled by a stochastic differential equation (SDE), the optimal backward SDE [1] can be retrieved by learning the score function [39, 40, 17, 2].”
Variational Autoencoder with Arbitrary Conditioning (2018, background): “Rezende et al. (2014) [Appendix F], Sohl-Dickstein et al. (2015), Goyal et al. (2017), and Bordes et al. (2017) propose to ﬁll missing data with noise and run Markov chain with a learned transition operator.”
Score-Based Generative Modeling with Critically-Damped Langevin Diffusion (2021, background): “Generative modeling by learning stochastic processes has a long history (Movellan, 2008; Lyu, 2009; Sohl-Dickstein et al., 2011; Bengio et al., 2014; Alain et al., 2016; Goyal et al., 2017; Bordes et al., 2017; Song & Ermon, 2019; Ho et al., 2020).”