Cooperative Learning of Energy-Based Model and Latent Variable Model via MCMC Teaching

By Jianwen Xie, Yang Lu, Ruiqi Gao, Ying Nian Wu. 2018

90 citations recorded by the source corpus.

This paper proposes a cooperative learning algorithm to train both the undirected energy-based model and the directed latent variable model jointly. The learning algorithm interweaves the maximum likelihood algorithms for learning the two models, and each iteration consists of the following two steps: (1) Modified contrastive divergence for energy-based model: The learning of the energy-based model is based on the contrastive divergence, but the finite-step MCMC sampling of the model is initialized from the synthesized examples generated by the latent variable model instead of being initialized from the observed examples. (2) MCMC teaching of the latent variable model: The learning of the latent variable model is based on how the MCMC in (1) changes the initial synthesized examples generated by the latent variable model, where the latent variables that generate the initial synthesized examples are known so that the learning is essentially supervised. Our experiments show that the cooperative learning algorithm can learn realistic models of images.

Subfields: Computer Vision and Pattern Recognition, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Probabilistic & Bayesian Methods

Citations in the corpus

Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model (2019, methodology): “For GAN, we perform Langevin inference of latent variables [16, 46].”
On the Anatomy of MCMC-based Maximum Likelihood Learning of Energy-Based Models (2019, methodology): “Several works studying the energy (2) recruit Langevin Dynamics to obtain the negative samples (Xie et al. 2016; Lu, Zhu, and Wu 2016; Xie et al. 2018a; Gao et al. 2018; Lee et al. 2018).”
Wasserstein Introspective Neural Networks (2017, background): “Recently, there has also been a cluster of algorithms developed in [50, 51, 15] where Langevin dynamics are adopted in generator CNNs.”
Cooperative Training of Descriptor and Generator Networks (2016, background): “This paper is an expansion of our conference paper [39].”
Generalized Energy Based Models (2020, methodology): “In (Xie et al., 2018b;a), two independent models, a full support EBM and a generator network, are trained cooperatively using MCMC.”
Improved Contrastive Divergence Training of Energy Based Models (2020, methodology): “…issue, a separate line of work utilizes an auxiliary network to amortize the negative portions of the sampling procedure (Kim & Bengio, 2016; Kumar et al., 2019; Han et al., 2019; Xie et al., 2018a; Song & Ou, 2018; Dai et al., 2019; Che et al., 2020; Grathwohl et al., 2020a; Arbel et al., 2020).”
SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations (2021, background): “, 2021) and energy-based models (Ackley et al., 1985; Gao et al., 2017; Du & Mordatch, 2019; Xie et al., 2018; 2016; Song & Kingma, 2021), have achieved comparable image sample quality as GANs.”
Learning Latent Space Energy-Based Prior Model (2020, unknown): “See also [80] for a temporal difference MCMC teaching scheme for amortizing MCMC.”
Building a Telescope to Look Into High-Dimensional Image Spaces (2018, methodology): “The ideas of the previous section can be taken a step further by deﬁning an energy function over the latent generator variable Z ∈ R n , where E ( ·| W 1 ) and g ( ·| W 2 ) are learned according to the standard Co-Op Net Algorithm.”
SDEdit: Image Synthesis and Editing with Stochastic Differential Equations (2021, background): “Recent advances in training non-normalized probabilistic models, such as score-based generative models [52, 53, 55, 23, 51, 28] and energy-based models [3, 15, 14, 62, 63, 54], have achieved comparable image sample quality as GANs.”