Using fast weights to improve persistent contrastive divergence

By Tijmen Tieleman, Geoffrey E. Hinton. 2009

320 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistics and Probability, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistics and Probability

In Optimization & Training

Citations in the corpus

Neural Autoregressive Distribution Estimation (2016, background): “d models (Salakhutdinov, 2009, 2010; Desjardins et al., 2010; Cho et al., 2010). Work has also been directed towards proposing improved update rules or parametrization of the model’s energy function (Tieleman and Hinton, 2009; Cho et al., 2013; Montavon and Muller, 2012) as well as improved approxim”
Super-Samples from Kernel Herding (2010, neutral): “This behavior is reminiscent of Quasi Monte Carlo integration and Bayesian quadrature methods [Rasmussen and Ghahramani, 2002], and is also related to the idea of fast weights for persistent contrastive divergence [Tieleman and Hinton, 2009].”
Statistical inference using weak chaos and infinite memory (2010, methodology): “In this context the work of (Tieleman & Hinton, 2009) should also be mentioned where fast weights fulfil a similar role.”
Short-term plasticity as cause–effect hypothesis testing in distal reward learning (2014, background): “Such as distinction is not new in connectionist models (Hinton and Plaut, 1987; Schmidhuber, 1992; Levy and Bairaktaris, 1995; Tieleman and Hinton, 2009; Bullinaria, 2009), however, in the proposed study the idea is extended to model hypothesis testing and memory consolidation with distal rewards.”
Online Contrastive Divergence with Generative Replay: Experience Replay without Storing Data (2016, background): “Examples of these are: persistent contrastive divergence (Tieleman, 2008), fast persistent contrastive divergence (Tieleman and Hinton, 2009), parallel tempering (Desjardins et al, 2010), and the replace of the Gibbs sampling with a transition operator to obtain a faster mixing rate and an improved…”
Practical Recommendations for Gradient-Based Training of Deep Architectures (2012, background): “An interesting alternative is rates-FPCD (Tieleman and Hinton, 2009; Breuleux et al. , 2011) which appears to be more robust to this problem and generally mixes faster, but at the cost of losing theoretical guarantees.”
Herding Dynamic Weights for Partially Observed Random Field Models (2009, background): “The above analysis has important implications for any attempts to combine herding of “fast weights” with learning of “slow weights” [9].”
Learning Generative Models with Visual Attention (2013, methodology): “For the top layer we use Fast Persistent Contrastive Divergence [32], which substantially improved generative performance of GDBNs (see supplementary material for more details).”
Deep Mixtures of Factor Analysers (2012, methodology): “For both the GRBM and SSU, we used Fast Persistent Contrastive Divergence (Tieleman & Hinton, 2009) for learning and AIS (Salakhutdinov & Murray, 2008) to estimate their log-partition functions.”
Robust Boltzmann Machines for recognition and denoising (2012, methodology): “We first pretrained a GRBM model with 2,000 hidden nodes using Fast PCD [24] on the 60,000 training images for 500 epochs.”
A Two-Stage Pretraining Algorithm for Deep Boltzmann Machines (2013, methodology): “We used contrastive divergence (CD) to train RBMs in the first stage, and the persistent CD [15] with coupled adaptive simulated annealing (CAST) was used in the second stage.”
3D ShapeNets: A deep representation for volumetric shapes (2014, methodology): “During pretraining , the first four layers are trained using standard Contrastive Divergence [14], while the top layer is trained more carefully using Fast Persistent Contrastive Divergence (FPCD) [32].”
Real-time classification and sensor fusion with a spiking deep belief network (2013, methodology): “This modiﬁcation is inspired by the idea of using auxiliary “fast-weights” for learning (Tieleman and Hinton, 2009), which transiently raise the energy of any state that the network is currently in, thereby slightly pushing it out of that state.”
Restoring Vision in Adverse Weather Conditions With Patch-Based Denoising Diffusion Models (2022, background): “Notably, energy-based models differ in its training approach which relies on contrastive divergence methods [30], [31], whereas diffusion- and score-based models exploit the sequential forward noising (diffusion) scheme to cover a smoother density across isolated modes of the training data distribut”
Building a better probabilistic model of images by factorization (2011, methodology): “Training was performed using several techniques: CD1, CD-5, Persistent CD, Fast Persistent CD [34], and Persistent MPF [32].”
Generating more realistic images using gated MRF's (2010, methodology): “Note that the use of FPCD as opposed to other estimation methods (like Persistent Contrastive Divergence [27]) turns out to be crucial to achieve good mixing of the sampler even after training.”