Learning FRAME Models Using CNN Filters

By Yang Lu, Song-Chun Zhu, Ying Nian Wu. 2015

67 citations recorded by the source corpus.

The convolutional neural network (ConvNet or CNN) has proven to be very successful in many tasks such as those in computer vision. In this conceptual paper, we study the generative perspective of the discriminative CNN. In particular, we propose to learn the generative FRAME (Filters, Random field, And Maximum Entropy) model using the highly expressive filters pre-learned by the CNN at the convolutional layers. We show that the learning algorithm can generate realistic and rich object and texture patterns in natural scenes. We explain that each learned model corresponds to a new CNN unit at a layer above the layer of filters employed by the model. We further show that it is possible to learn a new layer of CNN units using a generative CNN model, which is a product of experts model, and the learning algorithm admits an EM interpretation with binary latent variables.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning Energy-Based Spatial-Temporal Generative ConvNets for Dynamic Patterns (2019, methodology): “In terms of synthesis, various approaches based on the ConvNet have been proposed for synthesizing realistic static images [13], [14], [15], [16], [17].”
Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model (2019, methodology): “The maximum likelihood learning of the energy-based model (EBM) [29, 51, 19, 40, 30, 34, 6, 32, 48, 49, 22, 7, 47] follows what Grenander [15] called “analysis by synthesis” scheme.”
On the Anatomy of MCMC-based Maximum Likelihood Learning of Energy-Based Models (2019, methodology): “Several works studying the energy (2) recruit Langevin Dynamics to obtain the negative samples [38, 23, 37, 8, 22].”
Learning Descriptor Networks for 3D Shape Synthesis and Analysis (2018, methodology): “Inspired by the successes of deep convolutional neural networks (CNNs or ConvNets), [15] proposes a deep FRAME model, where the linear filters used in the original FRAME model are replaced by the non-linear filters at a certain convolutional layer of a pre-trained deep ConvNet.”
Learning Generative ConvNets via Multi-grid Modeling and Sampling (2017, background): “This paper studies the problem of learning energy-based generative ConvNet models [24, 10, 12, 11, 39, 25, 35, 28, 45, 46, 15] of images.”
A Theory of Generative ConvNet (2016, methodology): “(Lu et al., 2016) proposed to learn the FRAME (Filters, Random field, And Maximum Entropy) models (Zhu et al.”
Divergence Triangle for Joint Training of Generator Model, Energy-Based Model, and Inferential Model (2018, methodology): “[3], [4] generalized the model to deep versions.”
Alternating Back-Propagation for Generator Network (2016, neutral): “One can also obtain a probabilistic model based on a bottomup ConvNet that defines descriptive features (Xie et al., 2016; Lu, Zhu, and Wu, 2016).”
Cooperative Learning of Energy-Based Model and Latent Variable Model via MCMC Teaching (2018, methodology): “Powering the MCMC sampling of the descriptor model in (Lu, Zhu, and Wu, 2016; Xie et al., 2016) is a main motivation of this paper, with the bonus of turning the unsupervised learning of the generator (Han et al. 2017) into supervised learning.”
Interpretable R-CNN (2017, methodology): “There are a variety of methods including identifying high-scoring image patches [23, 52] or over-segmented atomic regions [62] directly, visualizing the layers of convolutional networks using deconvolutional networks to understand what contents are emphasized in the high-scoring input image patches ”
Configurable 3D Scene Synthesis and 2D Image Rendering with Per-pixel Ground Truth Using Stochastic Grammars (2017, methodology): “Image synthesis has been attempted using various deep neural network architectures, including recurrent neural networks (RNN) [23], generative adversarial networks (GAN) [80, 65], inverse graphics networks [42], and generative convolutional networks [53, 86, 85].”
Towards Interpretable R-CNN by Unfolding Latent Structures (2017, background): “More recently, Spatial attention-like mechanism has been widely studied in deep neural network based systems, including the seminal spatial transform network (Jaderberg et al. 2015) which warps the feature map via a global parametric transformation such as affine transformation, the exploration of g”
Towards Interpretable Object Detection by Unfolding Latent Structures (2019, background): “…images by maximizing the response of a given node in the network [12, 36, 55] or by developing a top-down generative convolutional networks [45, 62], and analyzing and visualizing state activation in recurrent networks [26, 29, 38, 14] to link word vectors to semantic lexicons or word…”
Synthesizing Dynamic Patterns by Spatial-Temporal Generative ConvNet (2016, background): “In terms of synthesis, various approaches based on the ConvNet have been proposed to synthesize realistic static images [3, 7, 1, 13, 16].”
On Learning Non-Convergent Short-Run MCMC Toward Energy-Based Model (2019, methodology): “The maximum likelihood learning of the energy-based model (EBM) [28, 56, 19, 41, 29, 35, 4, 32, 49, 54, 21, 5, 48], or the Gibbs distribution, follows what Grenander [15] called “analysis by synthesis” scheme.”
StyleBank: An Explicit Representation for Neural Image Style Transfer (2017, background): “It is very analogous to the concept of ”texton” [30, 41, 24] and filter bank in [42, 29], but StyleBank is defined in feature embedding space produced by auto-encoder [17] rather than image space.”
Building a Telescope to Look Into High-Dimensional Image Spaces (2018, methodology): “The DeepFRAME density incorporates aspects of both families, because the filters serve as both implicit features and sparse basis functions for image synthesis [31, 48].”