A Theory of Generative ConvNet

By Jianwen Xie, Yang Lu, Song-Chun Zhu, Ying Nian Wu. 2016

344 citations recorded by the source corpus.

We show that a generative random field model, which we call generative ConvNet, can be derived from the commonly used discriminative ConvNet, by assuming a ConvNet for multicategory classification and assuming one of the categories is a base category generated by a reference distribution. If we further assume that the non-linearity in the ConvNet is Rectified Linear Unit (ReLU) and the reference distribution is Gaussian white noise, then we obtain a generative ConvNet model that is unique among energy-based models: The model is piecewise Gaussian, and the means of the Gaussian pieces are defined by an auto-encoder, where the filters in the bottom-up encoding become the basis functions in the top-down decoding, and the binary activation variables detected by the filters in the bottom-up convolution process become the coefficients of the basis functions in the top-down deconvolution process. The Langevin dynamics for sampling the generative ConvNet is driven by the reconstruction error of this autoencoder. The contrastive divergence learning of the generative ConvNet reconstructs the training images by the auto-encoder. The maximum likelihood learning algorithm can synthesize realistic natural image patterns.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Molecular Biology, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Molecular Biology

In Generative Models

Citations in the corpus

Denoising Diffusion Probabilistic Models (2020, neutral): “By the known connection between score matching and energy-based modeling, our work could have implications for other recent work on energy-based models [67–69, 12, 70, 13, 11, 41, 17, 8].”
Residual Energy-Based Models for Text Generation (2020, methodology): “, 2003) and Langevyn dynamics (Ranzato et al., 2007; Du & Mordatch, 2019; Xie et al., 2016; 2017; 2019; 2018; Gao et al., 2018; Nijkamp et al., 2019) are not applicable when the input is discrete like in text applications.”
Flow Contrastive Estimation of Energy-Based Models (2019, background): “On the other end of the spectrum of deep generative models lies the family of energy-based models (EBMs) [42, 54, 29, 77, 73, 13, 38, 55, 9, 11, 16, 18, 6].”
Energy-Inspired Models: Learning with Sampler-Induced Distributions (2019, neutral): “, [70, 71, 72, 25, 22]).”
Learning Energy-Based Spatial-Temporal Generative ConvNets for Dynamic Patterns (2019, methodology): “Xie et al. [39] is a generalization of [22] for dynamic patterns by adopting spatial-temporal ConvNets [3] to capture spatial and temporal features of the video sequences.”
Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model (2019, methodology): “Recent work that parametrizes the energy function by modern convolutional neural networks (ConvNets) [28, 26] suggests that the “analysis by synthesis” process can indeed generate highly realistic images [48, 11, 21, 10].”
On the Anatomy of MCMC-based Maximum Likelihood Learning of Energy-Based Models (2019, methodology): “Several works studying the energy (2) recruit Langevin Dynamics to obtain the negative samples [38, 23, 37, 8, 22].”
Learning Descriptor Networks for 3D Shape Synthesis and Analysis (2018, methodology): “(1) We propose a 3D deep convolutional energy-based model that we call 3D DescriptorNet for modeling 3D object patterns by combining the volumetric ConvNets [16] and the generative ConvNets [33].”
Learning Generative ConvNets via Multi-grid Modeling and Sampling (2017, background): “The energy-based generative ConvNet model is as follows [45]: where p 0 ( Y ) is the reference distribution such as Gaussian white noise p 0 ( Y ) ∝ exp (cid:0) −(cid:107) Y (cid:107) 2 / 2 σ 2 (or a uniform distribution within a bounded range). f θ ( Y ) is deﬁned by a bottom-up ConvNet whose…”
Diffusion Models Beat GANs on Image Synthesis (2021, background): “[76] demonstrate that Langevin dynamics can be used to sample coherent images from these models.”
Dual Contradistinctive Generative Autoencoder (2020, neutral): “Energy-based generative models [56, 66, 31, 46] — which aim to directly model data density — are making a steady improvement for a simultaneously generative and discriminative single model.”
VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models (2020, methodology): “Among likelihood-based models, EBMs model the unnormalized data density by assigning low energy to high-probability regions in the data space (Xie et al., 2016; Du & Mordatch, 2019).”
Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks (2016, background): “The input textures are “ropenet” from the project link of [19], [34]’s “Ivy”, and Pablo Picasso’s “self portrait 1907”. a rather simple MRF, so the diﬀerence between 4 channel and 64 channel are marginal, unlike in the other two cases.”
Energy-based Latent Aligner for Incremental Learning (2022, methodology): “[54] represents EBM using a CNN and utilizes Langevin dynamics for MCMC sampling to generate realistic images.”
Image Generation from Scene Graphs (2018, background): “have investigated text to 3D scene generation [4, 5]; other approaches to image synthesis include stochastic grammars [20], probabalistic programming [27], inverse graphics [28], neural de-rendering [55], and generative ConvNets [56].”
Learning Energy-Based Models by Diffusion Recovery Likelihood (2020, background): “EBMs [27, 33, 21, 47, 10, 43, 6, 7, 25, 35, 5, 11, 4, 8, 2, 12] are an appealing class of probabilistic models, which can be viewed as generative versions of discriminators [18, 26, 28, 12], yet can be learned from unlabeled data.”
Learning to Draw Samples: With Application to Amortized MLE for Generative Adversarial Learning (2016, methodology): “Other more traditional approaches for training energy-based models (e.g., Ngiam et al., 2011; Xie et al., 2016) are often based on variants of MCMC-MLE or contrastive divergence (Geyer, 1991; Hinton, 2002; Tieleman, 2008), and have difﬁculty generating realistic-looking images from scratch.”
Wasserstein Introspective Neural Networks (2017, background): “Recently, there has also been a cluster of algorithms developed in [50, 51, 15] where Langevin dynamics are adopted in generator CNNs.”
Divergence Triangle for Joint Training of Generator Model, Energy-Based Model, and Inferential Model (2018, methodology): “[3], [4] generalized the model to deep versions.”