Learning Deep Energy Models

By Jiquan Ngiam, Zhenghao Chen, Pang Wei Koh, Andrew Y. Ng. 2011

201 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Generative Models

Citations in the corpus

Neural Autoregressive Distribution Estimation (2016, background): “rgy function (Tieleman and Hinton, 2009; Cho et al., 2013; Montavon and Muller, 2012) as well as improved approximate inference of the hidden layers (Salakhutdinov and Larochelle, 2010). The work of Ngiam et al. (2011) also proposed an undirected model that distinguishes itself from the DBM by havin”
Flow Contrastive Estimation of Energy-Based Models (2019, background): “On the other end of the spectrum of deep generative models lies the family of energy-based models (EBMs) [42, 54, 29, 77, 73, 13, 38, 55, 9, 11, 16, 18, 6].”
Learning Energy-Based Spatial-Temporal Generative ConvNets for Dynamic Patterns (2019, methodology): “field model, a Gibbs distribution, or an energy-based model [23], [24] that is in the form of exponential tilting of a reference distribution such as the Gaussian white noise distribution or the uniform distribution.”
Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model (2019, methodology): “The maximum likelihood learning of the energy-based model (EBM) [29, 51, 19, 40, 30, 34, 6, 32, 48, 49, 22, 7, 47] follows what Grenander [15] called “analysis by synthesis” scheme.”
On the Anatomy of MCMC-based Maximum Likelihood Learning of Energy-Based Models (2019, background): “[28] is an important early work proposing feedforward neural networks to model energy functions.”
Learning Generative ConvNets via Multi-grid Modeling and Sampling (2017, background): “This paper studies the problem of learning energy-based generative ConvNet models [24, 10, 12, 11, 39, 25, 35, 28, 45, 46, 15] of images.”
A Theory of Generative ConvNet (2016, methodology): “The hierarchical energy-based models (LeCun et al., 2006) were studied by the pioneering work of (Hinton et al., 2006b) and (Ngiam et al., 2011).”
Representation Learning: A Review and New Perspectives (2012, methodology): “Ngiam et al. (2011) have used Hybrid Monte Carlo (Neal, 1993), but other options include contrastive divergence (Hinton, 1999; Hinton et al., 2006), score matching (Hyvärinen, 2005; Hyvärinen, 2008), denoising score matching (Kingma and LeCun, 2010; Vincent, 2011), ratio-matching (Hyvärinen,…”
Generative Image Modeling Using Spatial LSTMs (2015, methodology): “When extended by labels, the Bayesian framework can be used to perform semi-supervised learning in the generative model [19, 28] while it is less clear how to combine other unsupervised approaches with discriminative learning.”
Deep Directed Generative Models with Energy-Based Probability Estimation (2016, methodology): “We use the form of a product of experts that is analogous to the free energy of an RBM with Gaussian visible units [13]: The ﬁrst two terms capture the mean and global variance, and the last term is a set of experts ˜ E θ i ( f ϕ ( x )) = − log (cid:0) 1 + e W Ti f ϕ ( x )+ b i over the feature…”
Learning Energy-Based Models by Diffusion Recovery Likelihood (2020, background): “EBMs [27, 33, 21, 47, 10, 43, 6, 7, 25, 35, 5, 11, 4, 8, 2, 12] are an appealing class of probabilistic models, which can be viewed as generative versions of discriminators [18, 26, 28, 12], yet can be learned from unlabeled data.”
Learning FRAME Models Using CNN Filters (2015, background): “See (Hinton et al. 2006; Ngiam et al. 2011) for earlier work along this direction.”
Learning to Draw Samples: With Application to Amortized MLE for Generative Adversarial Learning (2016, methodology): “Other more traditional approaches for training energy-based models (e.g., Ngiam et al., 2011; Xie et al., 2016) are often based on variants of MCMC-MLE or contrastive divergence (Geyer, 1991; Hinton, 2002; Tieleman, 2008), and have difﬁculty generating realistic-looking images from scratch.”
Wasserstein Introspective Neural Networks (2017, neutral): “DEM [35] extends the standard density estimation by using multi-layer neural networks (MLNN) with a rather complex training procedure.”
Divergence Triangle for Joint Training of Generator Model, Energy-Based Model, and Inferential Model (2018, background): “A complementary model is the energy-based model [3], [4], [33], [34], where −fα(x) defines the energy of x, and a low energy x is assigned a high probability.”
Cooperative Training of Descriptor and Generator Networks (2016, background): “(a) In the energybased model, the energy function can be defined by a bottom-up ConvNet that maps the image to the energy [12], [13], [14], [15], [16], and the energy function is usually the sum or a linear combination of the features at the top layer.”
Distributed optimization of deeply nested systems (2012, background): “Recent works in deep net learning, such as pretraining (Hinton and Salakhutdinov, 2006) or greedy layerwise training (Bengio et al., 2007; Ngiam et al., 2011), do a single pass over the net from the input to the output layer, fixing the weights of each layer sequentially, but without optimizing a jo”
Alternating Back-Propagation for Generator Network (2016, methodology): “tive ConvNet with no explicit latent factors Xie et al. (2016) recently proposed a generative ConvNet model that did not involve explicit latent factors. Their energy based model (LeCun et al., 2006; Ngiam et al., 2011; Zhu et al., 1997; Roth & Black, 2005), which can be derived from the commonl”
Cooperative Learning of Energy-Based Model and Latent Variable Model via MCMC Teaching (2018, methodology): “In the energy-based model, the energy function is parametrized by a bottom-up ConvNet that maps the signal to the energy (Ngiam et al. 2011), (Xie et al.”