The Convergence of Contrastive Divergences

By Alan L. Yuille. 2004

113 citations recorded by the source corpus.

Subfields: Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Probabilistic & Bayesian Methods

Citations in the corpus

On the Anatomy of MCMC-based Maximum Likelihood Learning of Energy-Based Models (2019, background): “Although the validity short-run MCMC with informative initialization can be justified under restrictive assumptions [41], in general it is not consistent with theoretically correct ML.”
Representation Learning: A Review and New Perspectives (2012, background): “Much has been written about the properties and alternative interpretations of CD and its similarity to autoencoder training, for example, [42], [225], [14], [197].”
Herding Dynamic Weights for Partially Observed Random Field Models (2009, background): “The ideas presented in this paper are related to recent developments in learning Markov random fields using insights from stochastic approximation theory [13, 14].”
Minimum Probability Flow Learning (2009, background): “Although CD is not guaranteed to converge to the right answer, or even to a fixed point, it has proven to be an effective and fast heuristic for parameter estimation (MacKay, 2001; Yuille, 2005).”
Learning with Hierarchical-Deep Models (2013, background): “Together with the condition on the learning rate, this ensures almost sure convergence of the stochastic approximation algorithm to an asymptotically stable point.”
Stochastic Spectral Descent for Restricted Boltzmann Machines (2015, background): “[Yuille, 2004] explored when the CD based descent would converge.”
Learning and Evaluating Boltzmann Machines (2008, methodology): “Precise sufficient conditions that ensure almost sure conve rgence to an asymptotically stable point of u θ = S(θ) are given in [ 32 , 33].”
Deep Boltzmann Machines (2009, background): “One necessary condition requires the learning rate to decrease with time, i.e. ∑∞ t=0 αt = ∞ and ∑∞ t=0 α 2 t ∞.”
Learning generative models of mid-level structure in natural images (2012, background): “Its properties have been studied by various authors (Yuille, 2004; Carreira-Perpiñ and Hinton, 2005; Bengio and Delalleau, 2009; Sutskever and Tieleman, 2010), and it has been found to often work reasonably well in practice although its effectiveness can depend strongly on the chosen T as well as on”
Herding dynamical weights to learn (2009, background): “Learning MRF models can also be formulated as a problem in stochastic approximation theory (Younes, 1999; Yuille, 2004).”
Multimodal learning with deep Boltzmann machines (2012, background): “Together with the condition on the learning rate, this ensures almost sure convergence of the stochastic approximation algorithm to an asymptotically stable point (Younes, 1998; Yuille, 2004).”
Structured Learning and Prediction in Computer Vision (2011, background): “Despite the many simplifications compared to a complete sampling approach, there are theoretic results that guarantee convergence to the global minimum under certain conditions [173], and that characterize the role of the sampling scheme used in the contrastive divergence procedure [7].”
On Contrastive Divergence Learning (2005, methodology): “Assuming we can prove convergence for the exact case, the right tools to use to prove it in the noisy case are probably those of stochastic approximation (Benveniste et al., 1990; Yuille, 2004).”
Learning Deep Boltzmann Machines using Adaptive MCMC (2010, background): “Almost sure convergence guarantees of SAP to an asymptotically stable point are given in (Younes, 2000; Yuille, 2004).”
Unsupervised Feature Learning and Deep Learning: A Review and New Perspectives (2012, background): “Much has been written about the properties and alternative interpretations of CD, e.g. Carreira-Perpi˜nan and Hinton (2005); Yuille (2005); Bengio and Delalleau (2009); Sutskever and Tieleman (2010).”