Towards deep learning with segregated dendrites

By Jordan Guerguiev, Timothy P. Lillicrap, Blake Aaron Richards. eLife, 2016

423 citations recorded by the source corpus.

Deep learning has led to significant advances in artificial intelligence, in part, by adopting strategies motivated by neurophysiology. However, it is unclear whether deep learning could occur in the real brain. Here, we show that a deep learning algorithm that utilizes multi-compartment neurons might help us to understand how the neocortex optimizes cost functions. Like neocortical pyramidal neurons, neurons in our model receive sensory information and higher-order feedback in electrotonically segregated compartments. Thanks to this segregation, neurons in different layers of the network can coordinate synaptic weight updates. As a result, the network learns to categorize images better than a single layer network. Furthermore, we show that our algorithm takes advantage of multilayer architectures to identify useful higher-order representations—the hallmark of deep learning. This work demonstrates that deep learning can be achieved using segregated dendritic compartments, which may help to explain the morphology of neocortical pyramidal neurons.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/d6c850b41500be8cfadb5aa710934d9c5c7bc100

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Electrical and Electronic Engineering

Citations in the corpus

A review of learning in biologically plausible spiking neural networks (2019, neutral): “There are a considerable number of DSNNs that use rate-based approach (Diehl & Cook, 2015; Eliasmith et al., 2012; Guerguiev, Lillicrap, & Richards, 2017; Mesnard, Gerstner, & Brea, 2016; Neftci, Emre, Augustine, Paul, & Detorakis, 2016).”
The Forward-Forward Algorithm: Some Preliminary Investigations (2022, background): “As a model of how cortex learns, backpropagation remains implausible despite considerable effort to invent ways in which it could be implemented by real neurons (Lillicrap et al., 2020; Richards and Lillicrap, 2019; Guerguiev et al., 2017; Scellier and Bengio, 2017).”
Theories of Error Back-Propagation in the Brain (2019, methodology): “These models perform similarly to arti ﬁ cial neural networks on standard benchmark tasks (e.g., handwritten digit classi ﬁ cation) [12 – 14, The criticism of weight symmetry has been addressed by demonstrating that even if the errors in arti ﬁ cial neural networks are back-propagated by random…”
Dendritic cortical microcircuits approximate the backpropagation algorithm (2018, background): “…developments have started to bridge the gap between neuroscience and artiﬁcial intelligence (Marblestone et al., 2016; Lillicrap et al., 2016; Scellier and Bengio, 2017; Costa et al., 2017; Guerguiev et al., 2017), how the brain could implement a backprop-like algorithm remains an open question.”
Eligibility Traces and Plasticity on Behavioral Time Scales: Experimental Support of NeoHebbian Three-Factor Learning Rules (2018, methodology): “Interestingly, learning similar, but not identical, to backpropagation is still possible with feedback circuits, where the direct feedback from the output is replaced with fixed random weights (Lillicrap et al., 2016; Guerguiev et al., 2017), or in networks with a single hidden layer and winner-take”
Synaptic Plasticity Forms and Functions. (2020, background): “…spine neck resistance is relatively high (approaching 1 G (cid:4) ) and, as a result, average-sized unpotentiated excitatory postsynaptic potentials reach ∼ 25 mV in spines (Bloodgood et al. 2009, Gulledge et al. 2012, Harnett et al. 2012, Jayant et al. 2017, Beaulieu-Laroche & Harnett 2018).”
Training Spiking Neural Networks Using Lessons From Deep Learning (2021, background): “Backpropagation solves spatial credit assignment by applying a distinct backward pass after a forward pass during the learning process [126].”
Artificial Neural Networks for Neuroscientists: A Primer (2020, methodology): “Efforts have been devoted to approximating gradient descent with algorithms more compatible with the brain’s hardware [Lillicrap et al., 2016, Guerguiev et al., 2017, Roelfsema and Holtmaat, 2018, Lillicrap et al., 2020].”
Surrogate Gradient Learning in Spiking Neural Networks: Bringing the Power of Gradient-based optimization to spiking neural networks (2019, methodology): “[26] used probabilistic neurons to study biologically plausible ways of propagating error or target signals using segregated dendrites (see the “Feedback Alignment and Random Error BP” section).”
Deep Learning With Spiking Neurons: Opportunities and Challenges (2018, background): “This concept has been recently demonstrated for deep SNNs (Neftci et al., 2017), and for networks with spiking multi-compartment neurons (Guerguiev et al., 2017).”