Predictive Coding

By Yanping Huang, Rajesh P. N. Rao. 2019

A general predictive coder utilizes the signal samples up to N steps back in time to make a prediction (guess) of what the signal sample at the present time should be and then codes the difference between the prediction and the real value. We will concentrate on linear predictors, where the prediction is a linear combination of old sample values. This correpsonds to having an AR model of the source.

Subfields: Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Probabilistic & Bayesian Methods

Citations in the corpus

Representation Learning with Contrastive Predictive Coding (2018, methodology): “This idea of predictive coding [5, 6] is one of the oldest techniques in signal processing for data compression.”
Shaping Belief States with Generative Environment Models for RL (2019, methodology): “999] Lo Overshoot Length {1, 2, 3, 6, 12} Lu Unroll Length [24-100] Nt Number of points used to evaluate the generative loss per trajectory [6] Ng Number of points used to evaluate the generative loss per overshoot [2] Ns Number of ConvDRAW Steps [8] Nh Number of units in LSTM [512-1024] Table 2: Hy”
Lifelong learning of human actions with deep neural network self-organization (2017, background): “This principle roughly resembles schemes of predictive coding (Huang & Rao, 2011; Rao & Ballard, 1999) proposing that the visual cortex can be modeled as a hierarchical network with reciprocal connections where top-down feedback connections from higher-order cortical areas convey predictions of…”
A deep learning framework for neuroscience (2019, unknown): “We will return to some of the specific objective functions proposed for the brain below 50–53 .”
Training Spiking Neural Networks Using Lessons From Deep Learning (2021, background): “Predictive coding models often aim to minimize the information entropy of sensory encodings such that the brain can actively predict incoming signals and inhibit what it already expects [123].”
The Rhetoric and Reality of Anthropomorphism in Artificial Intelligence (2019, methodology): “In addition to its popularity as a model of information processing in the visual cortex (Huang and Rao 2011), predictive coding has also been extended to sensorimotor functions (Körding and Wolp-ert 2007) and mirror neuronal systems (Kilner et al. 2007).”
A review of predictive coding algorithms. (2017, background): “Predictive coding suggests one way in which the brain might apply constraints in order to solve the inverse problem of perception (Bubic et al., 2010; Clark, 2013; Huang and Rao, 2011; Rao and Ballard, 1999; Spratling, 2014a).”
The free energy principle for action and perception: A mathematical review (2017, methodology): “The main advantage of adopting Gaussian assumptions is that they vastly simplify the implementation of the FEP, and make it formally equivalent to the more widely known predictive coding framework (Clark, 2013; Elias, 1955; Friston & Kiebel, 2009c), see the Introduction.”
Wasserstein Dependency Measure for Representation Learning (2019, background): “[45] which has connections to predictive coding in speech [4, 19], predictive processing in the brain [10, 41, 46] and the free energy principle [20].”
Surprisal-Triggered Conditional Computation with Neural Networks (2020, background): “According to the predictive coding hypothesis in neuroscience, human brains communicate information in a similar way: not as the raw signals themselves but rather in the form of prediction error [63, 64].”