Equivalence of Backpropagation and Contrastive Hebbian Learning in a Layered Network

By Xiaohui Xie, H. Sebastian Seung. 2003

228 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Electrical and Electronic Engineering

Citations in the corpus

Deep Learning: Methods and Applications (2014, background): “How can computational neuroscience models about hierarchical brain structure help improve engineering deep learning architectures? How may the biologically feasible learning styles in the brain [158, 395] help design more effective and more robust deep learning algorithms? All these issues and those”
Direct Feedback Alignment Provides Learning in Deep Neural Networks (2016, background): “In the case of weak symmetric feedback-connections it resembles BP [16].”
Toward an Integration of Deep Learning and Neuroscience (2016, methodology): “These include Generalized Recirculation (O’Reilly, 1996), Contrastive Hebbian Learning (Xie and Seung, 2003), random feedback weights together with synaptic homeostasis (Lillicrap et al., 2014; Liao et al., 2015), spike timing dependent plasticity (STDP) with iterative inference and target propagati”
Dendritic cortical microcircuits approximate the backpropagation algorithm (2018, methodology): “This link between the two algorithms was first established for an unsupervised learning task (Hinton and McClelland, 1988) and later analyzed (Xie and Seung, 2003) and generalized to broader classes of models (O’Reilly, 1996; Scellier and Bengio, 2017).”
Self-refreshing memory in artificial neural networks: learning temporal sequences without catastrophic forgetting (2004, background): “However, it has been shown that the deterministic version of the contrastive Hebbian learning algorithm, or CHL (Peterson and Anderson 1987, Hinton 1989b), is formally equivalent to backpropagation (Xie and Seung 2003) and that CHL could be directly derived from backpropagation (O’Reilly 1996).”
Training Neural Networks with Local Error Signals (2019, background): “…for error-driven learning have so far focused on how to transport the error back to hidden layers (Lillicrap et al., 2014; Nøkland, 2016), or how to transport a target back to hidden layers (Xie & Seung, 2003; Bengio, 2014; Lee et al., 2015b; Scellier & Bengio, 2017; Whittington & Bogacz, 2017).”
Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures (2018, background): “Researchers have long sought biologically plausible and empirically powerful learning algorithms that avoid these flaws [2, 30, 31, 1, 26, 39, 14, 16, 12, 5, 23].”
Switching between internal and external modes: A multiscale learning principle (2017, background): “Contrastive Hebbian learning is a supervised learning approach that Supervised learning: A learning process in which input data are paired with explicit labels or feedback. can approximate backpropagation in multilayer settings (O’Reilly, 1996; Xie & Seung, 2003; see also Scellier & Bengio, 2017).”
How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation (2014, background): “…the idea of biologically plausible implementations of back-propagation somewhat difﬁcult, although something clear come out of previous attempts (Xie and Seung, 2003; Hinton, 2007): it is likely that feedback connections, especially those coming from downstream (moving away from the sensory…”
Towards a Biologically Plausible Backprop (2016, methodology): “…1987) on recurrent back-propagation (for recurrent networks that converge to a fixed point to obtain their prediction) as well as with the work of Xie and Seung (2003) on the back-propagation interpretation of contrastive Hebbian learning may suggest an apparent contradiction: these earlier work…”
Random feedback weights support learning in deep neural networks (2014, methodology): “More-plausible algorithms have been devised 9,10,13,14,22,26,27,28,29,30 , but these either fall far short of backprop’s speed or call for a lot of additional processing 10,11,12,13,22 .”
Equilibrium Propagation: Bridging the Gap between Energy-Based Models and Backpropagation (2016, methodology): “Previous work on the back-propagation interpretation of contrastive Hebbian learning was done by Xie and Seung (2003).”