Difference Target Propagation

By Dong-Hyun Lee, Saizheng Zhang, Asja Fischer, Yoshua Bengio. 2014

422 citations recorded by the source corpus.

Back-propagation has been the workhorse of recent successes of deep learning but it relies on infinitesimal effects (partial derivatives) in order to perform credit assignment. This could become a serious issue as one considers deeper and more non-linear functions, e.g., consider the extreme case of non-linearity where the relation between parameters and cost is actually discrete. Inspired by the biological implausibility of back-propagation, a few approaches have been proposed in the past that could play a similar credit assignment role. In this spirit, we explore a novel approach to credit assignment in deep networks that we call target propagation. The main idea is to compute targets rather than gradients, at each layer. Like gradients, they are propagated backwards. In a way that is related but different from previously proposed proxies for back-propagation which rely on a backwards network with symmetric weights, target propagation relies on auto-encoders at each layer. Unlike back-propagation, it can be applied even when units exchange stochastic bits rather than real numbers. We show that a linear correction for the imperfectness of the auto-encoders, called difference target propagation, is very effective to make target propagation actually work, leading to results comparable to back-propagation for deep networks with discrete and continuous units and denoising auto-encoders and achieving state of the art for stochastic networks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Why Should We Add Early Exits to Neural Networks? (2020, background): “Based on the literature on target propagation [ 39 ] , the authors in [ 30 ] propose a further extension to decouple the different updates, in which the targets for the different early exits are computed adaptively from the original output y i by an additional set of autoencoder networks.”
Decoupled Neural Interfaces using Synthetic Gradients (2016, background): “E.g. target propagation (Lee et al., 2015; Bengio, 2014) removes the reliance on passing gradients between layers, by instead generating target activations which should be fitted to.”
MuProp: Unbiased Backpropagation for Stochastic Neural Networks (2015, methodology): “First, we might consider using other functions for the baseline rather than just the Taylor expansion, which could be learned in a manner that resembles Q-learning (Watkins & Dayan, 1992) and target propagation (Lee et al., 2015).”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “Other approaches using combinatorial optimization [65], target propagation [138], or Gumbel-softmax [115] have also been proposed.”
Direct Feedback Alignment Provides Learning in Deep Neural Networks (2016, background): “Recently, target-propagation (TP) was introduced as an biologically plausible training method, where each layer is trained to reconstruct the layer below [7].”
Greedy InfoMax for Biologically Plausible Self-Supervised Representation Learning (2019, background): “There are a number of optimization algorithms that eliminate the need for backpropagation altogether [Balduzzi et al., 2015, Kohan et al., 2018, Lee et al., 2015, Lillicrap et al., 2016, Scellier and Bengio, 2017].”
Online Semi-Supervised Learning with Deep Hybrid Boltzmann Machines and Denoising Autoencoders (2015, methodology): “In fact, any model that is capable of learning either discriminatively or generatively may be employed, such as Sum-Product Networks (Poon & Domingos, 2011) or recently proposed back-propagation-free models like the Difference-Target Propagation network (Lee et al., 2015).”
Presynaptic stochasticity improves energy efficiency and helps alleviate the stability-plasticity dilemma (2021, result): “Whether credit assignment in the brain relies on backpropagation – or more generally gradients – remains an active area of research, but several alternatives aiming to increase biological plausibility exist and are compatible with our model (Sacramento et al., 2018; Lillicrap et al., 2016; Lee et al”
Manifold Mixup: Better Representations by Interpolating Hidden States (2018, background): “ high level, we can imagine systems in the brain which compute predictions from a stream of changing inputs, and pass these 9 predictions onto other modules which return some kind of feedback signal (Lee et al., 2015; Scellier & Bengio, 2017; Whittington & Bogacz, 2017; Bartunov et al., 2018”
Towards Biologically Plausible Deep Learning (2015, background): “ork and Conclusion We consider this paper as an exploratory step towards explaining a central aspect of the brain’s learning algorithm: credit assignment through many layers. As argued byBengio(2014);Lee et al. (2015), departing from backpropagation could be useful not just for biological plausibili”
Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures (2018, methodology): “Unlike backpropagation, where backwards communication passes on gradients without inducing or altering neural activity, the backward pass in target propagation [19, 20, 3, 21] takes place in the same space as the forward-pass neural activity.”
Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets (2019, background): “In addition, Friesen & Domingos (2017) proposed the feasible target propagation algorithm for learning hard-threshold (or binary activated) networks (Lee et al., 2015) via convex combinatorial optimization.”
Training Neural Networks with Local Error Signals (2019, methodology): “…for error-driven learning have so far focused on how to transport the error back to hidden layers (Lillicrap et al., 2014; Nøkland, 2016), or how to transport a target back to hidden layers (Xie & Seung, 2003; Bengio, 2014; Lee et al., 2015b; Scellier & Bengio, 2017; Whittington & Bogacz, 2017).”
Theories of Error Back-Propagation in the Brain (2019, background): “There is a diversity of ideas on how the back-propagation algorithm may be approximated in the brain [31 – 36]; however, we review the principles behind a set of related models [11,13,14,37] that have substantial connections with biological data while closely paralleling the back-propagation…”