Predictability, Complexity, and Learning

By William Bialek, Ilya Nemenman, Naftali Tishby. 2000

219 citations recorded by the source corpus.

We define predictive information Ipred(T) as the mutual information between the past and the future of a time series. Three qualitatively different behaviors are found in the limit of large observation times T: Ipred(T) can remain finite, grow logarithmically, or grow as a fractional power law. If the time series allows us to learn a model with a finite number of parameters, then Ipred(T) grows logarithmically with a coefficient that counts the dimensionality of the model space. In contrast, power-law growth is associated, for example, with the learning of infinite parameter (or non-parametric) models such as continuous functions with smoothness constraints. There are connections between the predictive information and measures of complexity that have been defined both in learning theory and the analysis of physical systems through statistical mechanics and dynamical systems theory. Furthermore, in the same way that entropy provides the unique measure of available information consistent with some simple and plausible conditions, we argue that the divergent part of Ipred(T) provides the unique measure for the complexity of dynamics underlying a time series. Finally, we discuss how these ideas may be useful in problems in physics, statistics, and biology.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Molecular Biology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Molecular Biology

Citations in the corpus

Statistical inference using weak chaos and infinite memory (2010, background): “6 It is interesting to note that the authors of (Bialek et al., 2001) comment on the equivalence of information”
Generative Predictive Codes by Multiplexed Hippocampal Neuronal Tuplets. (2018, unknown): “A possible framework for optimizing speed and capacity of encoding is provided by the predictive coding theory, which proposes that the brain is a Bayesian interpreter that largely computes the difference between external stimuli and a prior, unconscious, internal model of the world (Bialek et al., ”
An information-theoretic approach to curiosity-driven reinforcement learning (2012, background): “Predictive information, defined as the mutual information between the past and the future within a time series, measures temporal correlations and is related to other measures of complexity [4, 7].”
Active inference and epistemic value (2015, background): “This is one instance of the Infomax principle (Linsker, 1990), which is closely related to the principle of maximum mutual information, or minimum redundancy (Barlow, 1961, 1974; Bialek et al., 2001; Najemnik & Geisler, 2005; Oja, 1989; Olshausen & Field, 1996; Optican & Richmond, 1987).”