Deep Supervised Learning Using Local Errors

By Hesham Mostafa, Vishwajith Ramesh, Gert Cauwenberghs. 2017

129 citations recorded by the source corpus.

Error backpropagation is a highly effective mechanism for learning high-quality hierarchical features in deep networks. Updating the features or weights in one layer, however, requires waiting for the propagation of error signals from higher layers. Learning using delayed and non-local errors makes it hard to reconcile backpropagation with the learning mechanisms observed in biological neural networks as it requires the neurons to maintain a memory of the input long enough until the higher-layer errors arrive. In this paper, we propose an alternative learning mechanism where errors are generated locally in each layer using fixed, random auxiliary classifiers. Lower layers could thus be trained independently of higher layers and training could either proceed layer by layer, or simultaneously in all layers using local error information. We address biological plausibility concerns such as weight symmetry requirements and show that the proposed learning mechanism based on fixed, broad, and random tuning of each neuron to the classification categories outperforms the biologically-motivated feedback alignment learning technique on the CIFAR10 dataset, approaching the performance of standard backpropagation. Our approach highlights a potential biological mechanism for the supervised, or task-dependent, learning of feature hierarchies. In addition, we show that it is well suited for learning deep networks in custom hardware where it can drastically reduce memory traffic and data communication overheads. Code used to run all learning experiments is available under https://gitlab.com/hesham-mostafa/learning-using-local-erros.git.

Subfields: Biophysics, Electrical and Electronic Engineering, Materials Chemistry, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biophysics

In Electrical and Electronic Engineering

In Materials Chemistry

In Optimization & Training

Citations in the corpus

Why Should We Add Early Exits to Neural Networks? (2020, background): “Multi-exit architectures in their LT formulation provide an interesting perspective on biological plausibility of neural network training [46].”
Surrogate Gradient Learning in Spiking Neural Networks: Bringing the Power of Gradient-based optimization to spiking neural networks (2019, background): “Each layer builds on the features learned by the layer below it, and we empirically observe that higher layers are able to learn higher-quality features that allow their random and fixed auxiliary layers to better match the target [40].”
Training Neural Networks with Local Error Signals (2019, methodology): “Using supervised layer-wise loss functions, without ﬁne-tuning, has also been explored previously (Mostafa et al., 2017; Malach & Shalev-Shwartz, 2018; Mar-a quez et al., 2018).”
Training Spiking Neural Networks Using Lessons From Deep Learning (2021, background): “Such approaches exploit the fact that training a shallow network is easier than training a deep one and aim to address spatial credit assignment by ensuring that the error signal does not need to propagate too far [127], [144].”
Synaptic Plasticity Dynamics for Deep Continuous Local Learning (DECOLLE) (2018, methodology): “DECOLLE uses layerwise local classifiers [17], which enables gradients to be computed locally.”
Associated Learning: Decomposing End-to-End Backpropagation Based on Autoencoders and Target Propagation (2019, background): “In [31], every layer in a deep neural network is trained by a local classifier.”
Decoupled Greedy Learning of CNNs (2019, result): “We also note concurrent work in the context of biologically plausible models by (Nøkland & Eidnes, 2019) which improves on results from (Mostafa et al., 2018), showing an approach similar to a specific instantiation of the synchronous version of DGL.”