Generative Modeling of Convolutional Neural Networks

By Jifeng Dai, Ying Nian Wu. 2014

75 citations recorded by the source corpus.

The convolutional neural networks (CNNs) have proven to be a powerful tool for discriminative learning. Recently researchers have also started to show interest in the generative aspects of CNNs in order to gain a deeper understanding of what they have learned and how to further improve them. This paper investigates generative modeling of CNNs. The main contributions include: (1) We construct a generative model for the CNN in the form of exponential tilting of a reference distribution. (2) We propose a generative gradient for pre-training CNNs by a non-parametric importance sampling scheme, which is fundamentally different from the commonly used discriminative gradient, and yet has the same computational architecture and cost as the latter. (3) We propose a generative visualization method for the CNNs by sampling from an explicit parametric image distribution. The proposed visualization method can directly draw synthetic samples for any given node in a trained CNN by the Hamiltonian Monte Carlo (HMC) algorithm, without resorting to any extra hold-out images. Experiments on the challenging ImageNet benchmark show that the proposed generative gradient pre-training consistently helps improve the performances of CNNs, and the proposed generative visualization method generates meaningful and varied samples of synthetic images from a large-scale deep CNN.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Flow Contrastive Estimation of Energy-Based Models (2019, methodology): “Moreover, the model has a direct correspondence with the commonly used discriminative network, as a simple consequence of the Bayes rule [2, 13, 25], so that the learned network can be turned into a classifier, or the features of the learned network can be used for downstream tasks such as classific”
Learning Energy-Based Spatial-Temporal Generative ConvNets for Dynamic Patterns (2019, methodology): “Our model in (2) directly corresponds to a classifier, specifically a spatial-temporal discriminative ConvNet, in the following sense [22], [51], [52], [53], [54], [55].”
Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model (2019, methodology): “The maximum likelihood learning of the energy-based model (EBM) [29, 51, 19, 40, 30, 34, 6, 32, 48, 49, 22, 7, 47] follows what Grenander [15] called “analysis by synthesis” scheme.”
On the Anatomy of MCMC-based Maximum Likelihood Learning of Energy-Based Models (2019, methodology): “The energy-based model in the form of (2) is introduced in [4].”
Learning Generative ConvNets via Multi-grid Modeling and Sampling (2017, methodology): “It can be derived from the commonly used discriminative ConvNet [23, 21] as a direct consequence of the Bayes rule [4], but unlike the discriminative ConvNet, the generative ConvNet is endowed with the gift of imagination in that it can generate images by sampling from the probability distribution…”
A Theory of Generative ConvNet (2016, methodology): “The model was first proposed by (Dai et al., 2015).”
Representation Learning by Learning to Count (2017, methodology): “There are methods to count humans in crowds [4, 42, 8], cars [28], and penguins [2].”
Understanding Neural Networks Through Deep Visualization (2015, neutral): “Work by Dai et al. (2015) shows some interesting results in this direction.”
Learning FRAME Models Using CNN Filters (2015, background): “See also (Dai, Lu, and Wu 2015) for generative gradient of CNN. Finally, we can also learn a FRAME model based on the features at the top fully connected layer, p(I;W ) = 1 Z(W ) exp [ N∑ i=1 Wi[Ψi ∗ I] ] q(I), (18) where Ψi is the i-th feature at the top fully connected layer, N is the total…”
Divergence Triangle for Joint Training of Generator Model, Energy-Based Model, and Inferential Model (2018, background): “A complementary model is the energy-based model [3], [4], [33], [34], where −fα(x) defines the energy of x, and a low energy x is assigned a high probability.”
Cooperative Training of Descriptor and Generator Networks (2016, background): “(a) In the energybased model, the energy function can be defined by a bottom-up ConvNet that maps the image to the energy [12], [13], [14], [15], [16], and the energy function is usually the sum or a linear combination of the features at the top layer.”
Alternating Back-Propagation for Generator Network (2016, methodology): “al., 2011; Zeiler & Fergus, 2014), the generator network (in the context of a generator-discriminator adversarial pair) (Goodfellow et al., 2014; Radford et al., 2015), or the generative ConvNet (Dai et al., 2015; Lu et al., 2016). In this paper , we consider a model that has become increasingly”
Super-Resolution with Deep Convolutional Sufficient Statistics (2015, background): “Learning non-linear convolutional features that capture natural image statistics has also been considered in the context of unsupervised learning (H´enaff et al. (2014)), as well as Gibbs models whose sufﬁcient statistics are given by deep convolutional features Dai et al. (2015).”
Learning Latent Space Energy-Based Prior Model (2020, methodology): “The energy-based model can be translated into a classifier and vice versa via the Bayes rule [25, 73, 12, 81, 36, 42, 19, 23, 59].”
What Happened to My Dog in That Network: Unraveling Top-down Generators in Convolutional Neural Networks (2015, background): “Another recent work is (Dai & Wu, 2014), modeling random fields in convolutional layers; however, they do not perform image synthesis, nor do they study explicit top-down transformations.”
Learning Neural Trans-Dimensional Random Field Language Models with Noise-Contrastive Estimation (2017, background): “The idea of defining a model in the form of exponential tilting of a reference distribution has been studied in natural image generative modeling [13, 14] where the reference distribution is set to the Gaussian white noise distribution.”