Statistical inference using weak chaos and infinite memory

By Max Welling, Yutian Chen. 2010

24 citations recorded by the source corpus.

We describe a class of deterministic weakly chaotic dynamical systems with infinite memory. These "herding systems" combine learning and inference into one algorithm, where moments or data-items are converted directly into an arbitrarily long sequence of pseudo-samples. This sequence has infinite range correlations and as such is highly structured. We show that its information content, as measured by sub-extensive entropy, can grow as fast as K log T, which is faster than the usual ½K log T for exchangeable sequences generated by random posterior sampling from a Bayesian model. In one dimension we prove that herding sequences are equivalent to Sturmian sequences which have complexity exactly log(T + 1). More generally, we advocate the application of the rich theoretical framework around nonlinear dynamical systems, chaos theory and fractal geometry to statistical learning.

Subfields: Artificial Intelligence, Mathematical Physics, Molecular Biology, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Mathematical Physics

In Molecular Biology

In Statistics and Probability

Citations in the corpus

On the Equivalence between Herding and Conditional Gradient Algorithms (2012, background): “Copyright 2012 by the author(s)/owner(s). where X is the observation space; Φ is a feature map from X to F , which could be viewed as the vector of suﬃcient statistics for some exponential family, and µ is a mean vector to match (the empirical moment vector of the same family).”
Kernel Mean Embedding of Distributions: A Review and Beyonds (2016, methodology): “34), Zhao and Meng (2015) proposes an efficient test called FastMMD which employs the random Fourier feature to transform the MMD test with translation invariant kernel.”
Super-Samples from Kernel Herding (2010, background): “We also note that herding is an “infinite memory process” on xt (as opposed to a Markov process) because new samples depend on the entire history of samples generated thus far [Welling and Chen, 2010].”
Advanced Structured Prediction (2014, background): “In fact, it can be shown that the dynamical system is weakly chaotic (Welling and Chen, 2010) meaning that the sequence over ŷt is not periodic and is insensitive to the initial setting of w0.”