Training Neural Networks with Local Error Signals

By Arild Nøkland, Lars Hiller Eidnes. 2019

273 citations recorded by the source corpus.

Supervised training of neural networks for classification is typically performed with a global loss function. The loss function provides a gradient for the output layer, and this gradient is back-propagated to hidden layers to dictate an update direction for the weights. An alternative approach is to train the network with layer-wise loss functions. In this paper we demonstrate, for the first time, that layer-wise training can approach the state-of-the-art on a variety of image datasets. We use single-layer sub-networks and two different supervised loss functions to generate local error signals for the hidden layers, and we show that the combination of these losses help with optimization in the context of local learning. Using local errors could be a step towards more biologically plausible deep learning because the global error does not have to be transported back to hidden layers. A completely backprop free variant outperforms previously reported results among methods aiming for higher biological plausibility. Code is available this https URL

Subfields: Artificial Intelligence, Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Optimization & Training

Citations in the corpus

Why Should We Add Early Exits to Neural Networks? (2020, background): “For example, [50] further replaces the true label at each auxiliary exit with a randomly projected version, while at the same time increasing the supervised loss with a separate one that forces the embeddings of the network to be similar whenever the corresponding projected outputs are similar.”
Associated Learning: Decomposing End-to-End Backpropagation Based on Autoencoders and Target Propagation (2019, background): “Therefore, many works have proposed methods that more closely resemble the operations of biological neurons [11, 12, 13, 14].”
Greedy InfoMax for Biologically Plausible Self-Supervised Representation Learning (2019, methodology): “Finally, we see that we outperform the state-of-the-art biologically inspired Predsim model from Nøkland and Eidnes [2019], which trains individual layers of a VGG like architecture [Simonyan and Zisserman, 2014] using two supervised loss functions.”
Deep semantic segmentation of natural and medical images: a review (2019, background): “, 2015), training with local error signals (Nøkland and Eidnes, 2019) and several other techniques (Amit, 2019; Bellec et al.”
Putting An End to End-to-End: Gradient-Isolated Learning of Representations (2019, unknown): “2 Predsim [Nøkland and Eidnes, 2019] 80.”
Online Learned Continual Compression with Stacked Quantization Module (2019, methodology): “Notably unlike VQVAE-2 (Razavi et al., 2019), each module is learned greedily without backward communication between modules using the current estimate of z ( i − 1) q similar to Belilovsky et al. (2019); Nøkland & Eidnes (2019).”
Decoupled Greedy Learning of CNNs (2019, result): “We also note concurrent work of [27] which shows an approach similar to the synchronous version of DGL in the context of biological plausibility.”
Online Learned Continual Compression with Adaptive Quantization Modules (2019, methodology): “VQVAE-2 (Razavi et al., 2019) also uses a multi-scale hierarchical organization, where unlike our AQM, the top level models global information such as shape, while the bottom without passing gradients between modules similar to (Belilovsky et al., 2019; Nøkland & Eidnes, 2019).”