How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation

By Yoshua Bengio. 2014

216 citations recorded by the source corpus.

We propose to exploit {\em reconstruction} as a layer-local training signal for deep learning. Reconstructions can be propagated in a form of target propagation playing a role similar to back-propagation but helping to reduce the reliance on derivatives in order to perform credit assignment across many levels of possibly strong non-linearities (which is difficult for back-propagation). A regularized auto-encoder tends produce a reconstruction that is a more likely version of its input, i.e., a small move in the direction of higher likelihood. By generalizing gradients, target propagation may also allow to train deep networks with discrete hidden units. If the auto-encoder takes both a representation of input and target (or of any side information) in input, then its reconstruction of input representation provides a target towards a representation that is more likely, conditioned on all the side information. A deep auto-encoder decoding path generalizes gradient propagation in a learned way that can could thus handle not just infinitesimal changes but larger, discrete changes, hopefully allowing credit assignment through a long chain of non-linear operations. In addition to each layer being a good auto-encoder, the encoder also learns to please the upper layers by transforming the data into a space where it is easier to model by them, flattening manifolds and disentangling factors. The motivations and theoretical justifications for this approach are laid down in this paper, along with conjectures that will have to be verified either mathematically or experimentally, including a hypothesis stating that such auto-encoder mediated target propagation could play in brains the role of credit assignment through many non-linear, noisy and discrete transformations.

Subfields: Accounting, Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Accounting

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Optimization & Training

Citations in the corpus

NICE: Non-linear Independent Components Estimation (2014, background): “Another interesting comparison is with the work of (Ozair and Bengio, 2014), which also considers a deterministic encoder, but with discrete input and latent variables, which is not always invertible.”
Semi-supervised Learning with Ladder Networks (2015, background): “ the stacked what-where autoencoder (SWWAE). Their network is trained simultaneously to minimize a combination of the supervised cost and reconstruction errors on each level, just like ours. Recently Bengio (2014) proposed target propagation as an alternative to backpropagation. The idea is to base ”
Decoupled Neural Interfaces using Synthetic Gradients (2016, background): “E.g. target propagation (Lee et al., 2015; Bengio, 2014) removes the reliance on passing gradients between layers, by instead generating target activations which should be fitted to.”
Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning (2016, background): “Thus, the targets are designed to compute higher level features as the error propagates up the network [4].”
Deep Directed Generative Autoencoders (2014, methodology): “…(RWS) The DGA can be seen as a special case of the Variational Autoencoder (VAE), with various versions introduced by Kingma and Welling (2014); Gregor et al. (2014); Mnih and Gregor (2014); Rezende et al. (2014), and of the Reweighted Wake-Sleep (RWS) algorithm (Bornschein and Bengio, 2014).”
Toward an Integration of Deep Learning and Neuroscience (2016, background): “Bengio and colleagues have proposed (Bengio, 2014; Bengio and Fischer, 2015; Scellier and Bengio, 2016) another context in which the connection between STDP and plasticity rules that depend on the temporal derivative of the post-synaptic ﬁring rate can be exploited for biologically plausible…”
Towards Biologically Plausible Deep Learning (2015, methodology): “The proposal made here also owes a lot to the idea of target propagation introduced in Bengio (2014); Lee et al. (2014), to which it adds the idea that in order to ﬁnd a target that is consistent with both the input and the ﬁnal output target, it makes sense to perform iterative inference,…”
Dendritic cortical microcircuits approximate the backpropagation algorithm (2018, methodology): “Such learning could play a role in perceptual denoising, pattern completion and disambiguation, and boost alignment beyond that achieved by pure feedback alignment (Bengio, 2014).”
Training Spiking Neural Networks Using Lessons From Deep Learning (2021, background): “Target propagation is similarly motivated by assigning a reconstruction criterion to each layer [115].”
ReSeg: A Recurrent Neural Network-Based Model for Semantic Segmentation (2015, methodology): “ents we introduced an intermediate classiﬁcation pathway after some of the ReSeg sublayers to provide a stronger learning signal to the network (see Fig. 3). This is justiﬁed by previous work such as Bengio (2014) and (Simonyan & Zisserman, 2015). It is important to highlight that the prediction”
Associated Learning: Decomposing End-to-End Backpropagation Based on Autoencoders and Target Propagation (2019, background): “This concept is the same as in target propagation [15, 16, 17, 13]: our goal is to minimize the distance between si and yi for every local subnetwork i.”
Online Semi-Supervised Learning with Deep Hybrid Boltzmann Machines and Denoising Autoencoders (2015, methodology): “Below, we present the design of two candidates for building uniﬁed, deep hybrid architectures, namely, the Deep Hybrid Boltzmann Machine (DHBM) and the Deep Hybrid Denoising Auto-encoder (DHDA).”
Training Neural Networks with Local Error Signals (2019, background): “…for error-driven learning have so far focused on how to transport the error back to hidden layers (Lillicrap et al., 2014; Nøkland, 2016), or how to transport a target back to hidden layers (Xie & Seung, 2003; Bengio, 2014; Lee et al., 2015b; Scellier & Bengio, 2017; Whittington & Bogacz, 2017).”
Difference Target Propagation (2014, methodology): “Moreover, as gi approaches to f −1 i , this approaches to vanilla target propagation formula in (Bengio, 2014). gi(hi) ∼ hi−1 → ĥi−1 = hi−1 − gi(hi) + gi(ĥi) ∼ gi(ĥi) (27)”
An Approximation of the Error Backpropagation Algorithm in a Predictive Coding Network with Local Hebbian Synaptic Plasticity (2017, methodology): “There is a discussion of a possible plasticity rule at the end of Bengio (2014) that has a similar form as equation 4.2 of the GeneRec model.”