Towards Biologically Plausible Deep Learning

By Yoshua Bengio, Dong-Hyun Lee, Jörg Bornschein, Zhouhan Lin. 2015

383 citations recorded by the source corpus.

Neuroscientists have long criticised deep learning algorithms as incompatible with current knowledge of neurobiology. We explore more biologically plausible versions of deep representation learning, focusing here mostly on unsupervised learning but developing a learning mechanism that could account for supervised, unsupervised and reinforcement learning. The starting point is that the basic learning rule believed to govern synaptic weight updates (Spike-TimingDependent Plasticity) can be interpreted as gradient descent on some objective function so long as the neuronal dynamics push firing rates towards better values of the objective function (be it supervised, unsupervised, or reward-driven). The second main idea is that this corresponds to a form of the variational EM algorithm, i.e., with approximate rather than exact posteriors, implemented by neural dynamics. Another contribution of this paper is that the gradients required for updating the hidden states in the above variational interpretation can be estimated using an approximation that only requires propagating activations forward and backward, with pairs of layers learning to form a denoising auto-encoder. Finally, we extend the theory about the probabilistic interpretation of auto-encoders to justify improved sampling schemes based on the generative interpretation of denoising auto-encoders, and we validate all these ideas on generative learning tasks.

Subfields: Artificial Intelligence, Biophysics, Cognitive Neuroscience

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biophysics

In Cognitive Neuroscience

Citations in the corpus

Learning Rapid-Temporal Adaptations (2017, methodology): “Direct feedback meta information Direct feedback (DF) meta information is inspired by feed-back alignment methods [21, 27] and on biologically plausible deep learning [4].”
Direct Feedback Alignment Provides Learning in Deep Neural Networks (2016, background): “The local update rule can be linked to Spike-Timing-Dependent Plasticity (STDP) believed to govern synaptic weight updates in the brain, see [1].”
Toward an Integration of Deep Learning and Neuroscience (2016, background): “…homeostasis (Lillicrap et al., 2014; Liao et al., 2015), spike timing dependent plasticity (STDP) with iterative inference and target propagation (Bengio et al., 2015a; Scellier and Bengio, 2016), complex neurons with backpropagating action-potentials ( Körding and König, 2000), and others…”
Towards deep learning with segregated dendrites (2016, methodology): “This sort of top-down modulation could be used by the brain to improve sensory processing in different contexts and engage in inference (Bengio et al., 2015).”
Understanding Synthetic Gradients and Decoupled Neural Interfaces (2017, background): “However, this scheme has many potential drawbacks, as well as lacking biological plausibility (Marblestone et al., 2016; Bengio et al., 2015).”
Entropy-based Sample Selection for Online Continual Learning (2021, background): “Although DNNs have achieved remarkable success and even reached or surpassed human performance in many areas of machine learning [1], they are trained in a way that differs signiﬁcantly from how humans learn [2].”
Training Excitatory-Inhibitory Recurrent Neural Networks for Cognitive Tasks: A Simple and Flexible Framework (2016, background): “Although none of the learning methods discussed here can at present be considered biological, recent work also suggests that spike-timing dependent plasticity (STDP) [38], which is believed to be a basic rule governing synaptic weight changes in the brain, may correspond to a form of SGD [39, 40].”
Backpropagation and Biological Plausibility (2018, background): “Since then, a number of papers have contributed with many insights on the overcoming of this limitation, including recent contributions [1],[5].”
Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures (2018, background): “Researchers have long sought biologically plausible and empirically powerful learning algorithms that avoid these flaws [2, 30, 31, 1, 26, 39, 14, 16, 12, 5, 23].”
Training Neural Networks with Local Error Signals (2019, background): “Back-propagation of global errors is not biologically plausible for a number of reasons (Bengio et al., 2015).”
Deep Learning With Spiking Neurons: Opportunities and Challenges (2018, background): “…realistic networks with backpropagation, despite deviating from the requirements of classical backpropagation, namely precise calculation of real-valued gradients, a separation and synchronization of forward-and backward-passes, and symmetry of weights in both directions (Bengio et al., 2015a).”
Switching between internal and external modes: A multiscale learning principle (2017, background): “Finally, Hinton (2007) and Bengio et al. (2015) have argued that spike-timing-dependent plasticity can approximate a weight update similar to backpropagation.”
On Adversarial Mixup Resynthesis (2019, background): “…concepts from evolution and biology in deep learning, for instance meta-learning (Bengio et al., 1991), dropout (as previously mentioned), biologically plausible deep learning (Bengio et al., 2015) and evolutionary strategies for reinforcement learning (Such et al., 2017; Salimans et al., 2017).”
Synaptic Plasticity Forms and Functions. (2020, background): “…an active research area, and recent work suggests that there may be more compatibilities NE43CH05_Magee ARjats.cls February 6, 2020 10:7 between the biology and theory than were originally appreciated (Bengio et al. 2016, Lillicrap et al. 2016, Roelfsema & Holtmatt 2018, Whittington & Bogacz 2019).”
Towards a Biologically Plausible Backprop (2016, methodology): “In this paper, we change the STDP learning rule from Bengio et al. (2015a) into dWij dt ∝ ρ(si) dρ(sj) dt .”
An Introduction to Deep Reinforcement Learning (2018, neutral): “However, subsequent developments are such that deep learning has become partly incompatible with current knowledge of neurobiology (Bengio et al., 2015).”
Training Spiking Neural Networks Using Lessons From Deep Learning (2021, neutral): “[211] previously made the case that STDP resembles stochastic gradient descent, provided that STDP is supplemented with gradient feedback [212].”