Cooperative Training of Descriptor and Generator Networks

By Jianwen Xie, Yang Lu, Ruiqi Gao, Song-Chun Zhu, Ying Nian Wu. 2016

151 citations recorded by the source corpus.

This paper studies the cooperative training of two generative models for image modeling and synthesis. Both models are parametrized by convolutional neural networks (ConvNets). The first model is a deep energy-based model, whose energy function is defined by a bottom-up ConvNet, which maps the observed image to the energy. We call it the descriptor network. The second model is a generator network, which is a non-linear version of factor analysis. It is defined by a top-down ConvNet, which maps the latent factors to the observed image. The maximum likelihood learning algorithms of both models involve MCMC sampling such as Langevin dynamics. We observe that the two learning algorithms can be seamlessly interwoven into a cooperative learning algorithm that can train both models simultaneously. Specifically, within each iteration of the cooperative learning algorithm, the generator model generates initial synthesized examples to initialize a finite-step MCMC that samples and trains the energy-based descriptor model. After that, the generator model learns from how the MCMC changes its synthesized examples. That is, the descriptor model teaches the generator model by MCMC, so that the generator model accumulates the MCMC transitions and reproduces them by direct ancestral sampling. We call this scheme MCMC teaching. We show that the cooperative algorithm can learn highly realistic generative models.

Subfields: Biophysics, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biophysics

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Learning Energy-Based Spatial-Temporal Generative ConvNets for Dynamic Patterns (2019, background): “[37] proposed to recruit a top-down generator [38] serving as an approximate sampler of the energybased generative ConvNetmodel.”
Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model (2019, methodology): “The maximum likelihood learning of the energy-based model (EBM) [29, 51, 19, 40, 30, 34, 6, 32, 48, 49, 22, 7, 47] follows what Grenander [15] called “analysis by synthesis” scheme.”
Learning Descriptor Networks for 3D Shape Synthesis and Analysis (2018, methodology): “The 3D generator model can be trained simultaneously with the 3D DescriptorNet in a cooperative training scheme [31].”
Learning Generative ConvNets via Multi-grid Modeling and Sampling (2017, background): “[44] also studied the problem of joint learning of the energy-based model and the generator model.”
Learning Energy-Based Models by Diffusion Recovery Likelihood (2020, background): “The log-likelihood function is In MLE, we seek to maximize the log-likelihood function, where the gradient approximately follows (Younes, 1999; Xie et al., 2016b) The expectations can be approximated by averaging w.r.t. the observed samples and the synthesized samples drawn from the model…”
Wasserstein Introspective Neural Networks (2017, background): “Recently, there has also been a cluster of algorithms developed in [48, 47, 13] where Langevin dynamics are adopted in generator CNNs.”
Divergence Triangle for Joint Training of Generator Model, Energy-Based Model, and Inferential Model (2018, methodology): “Table 4 shows the IS and FID scores of our model compared with VAE [15], DCGAN [20], WGAN [22], CoopNet [25], CEGAN [24], ALI [26], ALICE [28].”
Energy-based Out-of-distribution Detection (2020, methodology): “[41, 43–45] explored using EBMs for video generation and 3D shape pattern generation.”
Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models (2021, background): “✓ _ 37.30 3:27 VAEBM [246] ?????”
Configurable 3D Scene Synthesis and 2D Image Rendering with Per-pixel Ground Truth Using Stochastic Grammars (2017, methodology): “Image synthesis has been attempted using various deep neural network architectures, including recurrent neural networks (RNN) [23], generative adversarial networks (GAN) [80, 65], inverse graphics networks [42], and generative convolutional networks [53, 86, 85].”
VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generation (2019, background): “…changes, such as predicting high-level structure (Villegas et al., 2017b), energy-based models (Xie et al., 2017), generative cooperative nets (Xie et al., 2020), ABPTT (Xie et al., 2019), incorporating pixel transformations (Finn et al., 2016; De Brabandere et al., 2016; Liu et al., 2017)…”
Stochastic Image-to-Video Synthesis using cINNs (2021, methodology): “[84] Jianwen Xie, Yang Lu, Ruiqi Gao, Song-Chun Zhu, and”
Energy-Based Models for Continual Learning (2020, methodology): “Xie et al. (2018) cooperatively train the descriptor and generator using MCMC sampling.”
Unpaired Image-to-Image Translation via Latent Energy Transport (2020, background): “Energy-based Model Recently, energy-based generative models have drawn significant attention [40, 39, 9, 29, 7, 11, 43, 41].”
Variational (Gradient) Estimate of the Score Function in Energy-based Latent Variable Models (2020, background): “, 2020) and CoopNets (Xie et al., 2018) report FID results on a subset of CelebA and on a different resolution of CelebA respectively.”
Joint Training of Variational Auto-Encoder and Latent Energy-Based Model (2020, methodology): “The generator and energy-based model can be trained separately using maximum likelihood criteria as in [13, 33], and they can also be trained jointly as recently explored by [20, 38, 14, 24].”