Learning Generative ConvNets via Multi-grid Modeling and Sampling

By Ruiqi Gao, Yang Lu, Junpei Zhou, Song-Chun Zhu, Ying Nian Wu. 2017

82 citations recorded by the source corpus.

This paper proposes a multi-grid method for learning energy-based generative ConvNet models of images. For each grid, we learn an energy-based probabilistic model where the energy function is defined by a bottom-up convolutional neural network (ConvNet or CNN). Learning such a model requires generating synthesized examples from the model. Within each iteration of our learning algorithm, for each observed training image, we generate synthesized images at multiple grids by initializing the finite-step MCMC sampling from a minimal 1 × 1 version of the training image. The synthesized image at each subsequent grid is obtained by a finite-step MCMC initialized from the synthesized image generated at the previous coarser grid. After obtaining the synthesized examples, the parameters of the models at multiple grids are updated separately and simultaneously based on the differences between synthesized and observed examples. We show that this multi-grid method can learn realistic energy-based generative ConvNet models, and it outperforms the original contrastive divergence (CD) and persistent CD.

Subfields: Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Generative Models

1. Ying Nian Wu
2. Song-Chun Zhu
3. Yilun Du
4. Ruiqi Gao
5. Erik Nijkamp
6. Arash Vahdat
7. Igor Mordatch
8. Yang Song

Citations in the corpus

• Denoising Diffusion Probabilistic Models (2020, neutral): “By the known connection between score matching and energy-based modeling, our work could have implications for other recent work on energy-based models [67–69, 12, 70, 13, 11, 41, 17, 8].”
• Residual Energy-Based Models for Text Generation (2020, methodology): “, 2003) and Langevyn dynamics (Ranzato et al., 2007; Du & Mordatch, 2019; Xie et al., 2016; 2017; 2019; 2018; Gao et al., 2018; Nijkamp et al., 2019) are not applicable when the input is discrete like in text applications.”
• Flow Contrastive Estimation of Energy-Based Models (2019, background): “CD has been generalized to persistent CD [67], and has more recently been generalized to modified CD [13], adversarial CD [29, 5, 22] with modern CNN structure.”
• Energy-Inspired Models: Learning with Sampler-Induced Distributions (2019, neutral): “, [70, 71, 72, 25, 22]).”
• Learning Energy-Based Spatial-Temporal Generative ConvNets for Dynamic Patterns (2019, methodology): “[36] developed a multi-grid sampling and learning method for the energy-based generative ConvNet, and Xie et al.”
• Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model (2019, background): “Recent work that parametrizes the energy function by modern convolutional neural networks (ConvNets) [28, 26] suggests that the “analysis by synthesis” process can indeed generate highly realistic images [48, 11, 21, 10].”
• Learning Energy-Based Models by Diffusion Recovery Likelihood (2020, background): “EBMs [27, 33, 21, 47, 10, 43, 6, 7, 25, 35, 5, 11, 4, 8, 2, 12] are an appealing class of probabilistic models, which can be viewed as generative versions of discriminators [18, 26, 28, 12], yet can be learned from unlabeled data.”
• Implicit Generation and Generalization in Energy-Based Models (2019, methodology): “A multi-grid sampling and training strategy for the energy-based generative ConvNet is also studied in [Gao et al., 2018].”
• How to Train Your Energy-Based Model for Regression (2020, methodology): “EBMs have therefore become increasingly popular within computer vision in recent years, commonly being applied for various generative image modeling tasks [9, 11, 12, 13, 39, 40, 55].”
• Learning Energy-based Model with Flow-based Backbone by Neural Transport MCMC (2020, methodology): “Taking advantage of the approximation capacity of modern deep networks such as convolutional network (ConvNet) [39, 34], recent papers [57, 13, 36, 47, 11] parametrize the energy function by ConvNet.”
• Energy-Based Models for Deep Probabilistic Regression (2020, methodology): “Recently, energy-based models have been used primarily for unsupervised learning problems within computer vision [58, 14, 11, 29, 40], where DNNs are directly used to predict Eθ(x).”
• Improved Contrastive Divergence Training of Energy Based Models (2020, methodology): “(Gao et al., 2018) also proposes a multiscale approach towards generating images from EBMs, but different from our work, uses each sub-scale EBM to initialize the generation of the next EBM, while we jointly sample across each resolution.”
• Residual Energy-Based Models for Text (2021, methodology): “, 2003) and Langevyn dynamics (Ranzato et al., 2007; Du and Mordatch, 2019; Xie et al., 2016, 2017, 2019, 2018; Gao et al., 2018; Nijkamp et al., 2019) are not applicable when the input is discrete like in text applications.”
• Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, background): “Energy-based Models (EBMs) [26, 48, 58, 64, 67, 68, 75, 78, 79, 120, 129, 132, 165, 170, 182, 196, 254, 281] can be viewed as one generative version of discriminators [79, 104, 131, 134], while can be learned from unlabeled input data.”
• On Learning Non-Convergent Short-Run MCMC Toward Energy-Based Model (2019, methodology): “Alternatively, [11] devises a non-persistent multi-grid short-run MCMC that is always initialized from the histogram extracted from 1×1 versions of the observed images.”