Herding Dynamic Weights for Partially Observed Random Field Models

By Max Welling. 2009

38 citations recorded by the source corpus.

Learning the parameters of a (potentially partially observable) random field model is intractable in general. Instead of focussing on a single optimal parameter value we propose to treat parameters as dynamical quantities. We introduce an algorithm to generate complex dynamics for parameters and (both visible and hidden) state vectors. We show that under certain conditions averages computed over trajectories of the proposed dynamical system converge to averages computed over the data. Our "herding dynamics" does not require expensive operations such as exponentiation and is fully deterministic.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Probabilistic & Bayesian Methods

Citations in the corpus

On the Equivalence between Herding and Conditional Gradient Algorithms (2012, methodology): “Welling (2009b) proposed originally an algorithmic interpretation of herding as performing subgradient ascent with constant step size on a function obtained as the zero temperature limit of the log-likelihood of an exponential model that he called the “tipi function”.”
Representation Learning: A Review and New Perspectives (2012, background): “…2012) and it has been transformed into a sampling algorithm (Breuleux et al., 2011) that also shares this fast mixing property with herding (Welling, 2009), for the same reason, i.e., introducing negative correlations between consecutive samples of the chain in order to promote faster…”
Kernel Mean Embedding of Distributions: A Review and Beyonds (2016, methodology): “Following the result of Welling (2009a), kernel herding is shown to decrease the error of expectations of functions in the RKHS at a rate of O(1/T ) as opposed to the random samples whose rate is O(1/ √ T ).”
Super-Samples from Kernel Herding (2010, methodology): “We directly describe the herding algorithm in terms of Reproducing Kernel Hilbert Spaces and (potentially) continuous index spaces (note that previous work by [Welling, 2009b,a] cast it in terms of finite-dimensional spaces and discrete domains).”
Statistical inference using weak chaos and infinite memory (2010, background): “(For more details see (Welling, 2009a).)”
On Generalizing Beyond Domains in Cross-Domain Continual Learning (2022, background): “A classical approach is known as herding [36] by picking the nearest neighbors from the mean of exemplars in each class.”
On Learning the Geodesic Path for Incremental Learning (2021, methodology): “A method, so-called herding [34, 40], is used to pick potential exemplars for future use in training models.”
Parametric Herding (2010, methodology): “The nonlinear dynamical system in question turns out to be on the boundary between order and chaos, formally to be classified as “weakly chaotic” with polynomial sensitivity to initial conditions.”
On the Expressive Power of Deep Architectures (2011, background): “This principle may also explain some of the fast mixing observed in a related approach called Herding (Welling, 2009; Breuleux et al., 2011).”
Quickly Generating Representative Samples from an RBM-Derived Process (2011, methodology): “The herding algorithm (Welling, 2009a, 2009b) outright gives up on getting a point estimate of the parameters and instead simply aims to produce samples like those of the training set through a dynamical system.”
Unsupervised Feature Learning and Deep Learning: A Review and New Perspectives (2012, background): “…been applied in a variety of applications (Tieleman and Hinton, 2009; Ranzato et al. , 2011; Kivinen and Williams, 2012) and it has been transformed into a pure sampling algorithm (Breuleux et al. , 2011) that also shares this fast mixing property with herding (Welling, 2009), for the same reason.”
Bounding the Test Log-Likelihood of Generative Models (2013, background): “That includes Herding Welling (2009), the non-zero temperature version of Herding (Breuleux et al., 2011), and the recently proposed generative procedures for contractive auto-encoders (CAE, Rifai et al., 2012a), denoising auto-encoders (DAE, Bengio et al., 2013b), and generative stochastic networks”