Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures

By Sergey Bartunov, Adam Santoro, Blake Aaron Richards, Geoffrey E. Hinton, Timothy P. Lillicrap. 2018

281 citations recorded by the source corpus.

The backpropagation of error algorithm (BP) is impossible to implement in a real brain. The recent success of deep networks in machine learning and AI, however, has inspired proposals for understanding how the brain might learn across multiple layers, and hence how it might approximate BP. As of yet, none of these proposals have been rigorously evaluated on tasks where BP-guided deep learning has proved critical, or in architectures more structured than simple fully-connected networks. Here we present results on scaling up biologically motivated models of deep learning on datasets which need deep networks with appropriate architectures to achieve good performance. We present results on the MNIST, CIFAR-10, and ImageNet datasets and explore variants of target-propagation (TP) and feedback alignment (FA) algorithms, and explore performance in both fully- and locally-connected architectures. We also introduce weight-transport-free variants of difference target propagation (DTP) modified to remove backpropagation from the penultimate layer. Many of these algorithms perform well for MNIST, but for CIFAR and ImageNet we find that TP and FA variants perform significantly worse than BP, especially for networks composed of locally connected units, opening questions about whether new architectures and algorithms are required to scale these approaches. Our results and implementation details help establish baselines for biologically motivated deep learning schemes going forward.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Evaluation & Benchmarks

Citations in the corpus

Stand-Alone Self-Attention in Vision Models (2019, positive): “This paper goes beyond [33] by removing convolutions and employing local self-attention across the entirety of the network.”
Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes (2018, neutral): “While this is consistent with the results of Lecun (1989), in a different setting Bartunov et al. (2018) found equivariance to not provide a substantial performance gain.”
A solution to the learning dilemma for recurrent networks of spiking neurons (2019, background): “(25)) is essential for understanding e-prop, and it is the main driver behind the resulting qualitative jump in computing capabilities of RSNNs, which are attainable through biologically plausible learning.”
Theories of Error Back-Propagation in the Brain (2019, background): “However, much more work needs to be done both empirically and theoretically, for example, on how the networks scale to larger architectures [28], as well as linking theory to neurobiological data (see Outstanding Questions).”
Synaptic Plasticity Dynamics for Deep Continuous Local Learning (DECOLLE) (2018, background): “This makes DECOLLE signiﬁcantly cheaper to implement compared to BPTT for training SNN, e.g., SLAYER (Shrestha and Orchard, 2018) which scales spatially as O ( NT ), where T is the number of timesteps (see Appendix section 5.2 for details on scaling).”
Low-dimensional dynamics for working memory and time encoding (2020, background): “Methods have been proposed for biologically plausible learning in recurrent networks (62, 63), but it should not be assumed that these methods scale to harder problems, as this scaling has proven difficult for biological approximations to backpropagation for feedforward networks (64).”
Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future (2020, background): “That being said, several researchers are investigating means by which the brain could perform something like backpropagation[111, 112, 113, 114].”
Training Spiking Neural Networks Using Lessons From Deep Learning (2021, background): “Random feedback has struggled with more complex tasks though variants have been proposed that reduce the gap [139], [140], [141], [142].”
Manifold Mixup: Encouraging Meaningful On-Manifold Interpolation as a Regularizer (2018, background): “…which compute predictions from a stream of changing inputs, and pass these predictions onto other modules which return some kind of feedback signal (Lee et al., 2015; Scellier & Bengio, 2017; Whittington & Bogacz, 2017; Bartunov et al., 2018) (potentially gradients or targets for the prediction).”
Greedy Layerwise Learning Can Scale to ImageNet (2018, background): “9% (Bartunov et al., 2018; Xiao et al., 2019).”
Decoupled Greedy Learning of CNNs (2019, background): “One class of alternatives to standard back-propagation aims to avoid its biologically implausible aspects, most notably the weight transport problem (Bartunov et al., 2018; Nø kland, 2016; Lillicrap et al., 2014; Lee et al., 2014; Ororbia et al., 2018; Ororbia & Mali, 2019).”
Associated Learning: Decomposing End-to-End Backpropagation Based on Autoencoders and Target Propagation (2019, background): “Therefore, many works have proposed methods that more closely resemble the operations of biological neurons [11, 12, 13, 14, 15, 16].”
Manifold Mixup: Better Representations by Interpolating Hidden States (2018, background): “At a high level, we can imagine systems in the brain which compute predictions from a stream of changing inputs, and pass these predictions onto other modules which return some kind of feedback signal (Lee et al., 2015; Scellier & Bengio, 2017; Whittington & Bogacz, 2017; Bartunov et al., 2018).”
Burst-dependent synaptic plasticity can coordinate learning in hierarchical circuits (2020, result): “on two challenging image categorization datasets that previous biologically plausible algorithms have struggled with: CIFAR-10 and ImageNet [6].”
Training Neural Networks with Local Error Signals (2019, background): “These methods do not seem to scale to larger and more complicated problems like CIFAR10 and ImageNet (Bartunov et al., 2018).”
Shallow Learning For Deep Networks (2018, background): “None of them have scaled to datasets such as ImageNet, where end-to-end CNNs have proved absolutely critical (Bartunov et al., 2018).”