Direct Feedback Alignment Provides Learning in Deep Neural Networks

By Arild Nøkland. 2016

574 citations recorded by the source corpus.

Artificial neural networks are most commonly trained with the back-propagation algorithm, where the gradient for learning is provided by back-propagating the error, layer by layer, from the output layer to the hidden layers. A recently discovered method called feedback-alignment shows that the weights used for propagating the error backward don't have to be symmetric with the weights used for propagation the activation forward. In fact, random feedback weights work evenly well, because the network learns how to make the feedback useful. In this work, the feedback alignment principle is used for training hidden layers more independently from the rest of the network, and from a zero initial condition. The error is propagated through fixed random feedback connections directly from the output layer to each hidden layer. This simple method is able to achieve zero training error even in convolutional networks and very deep networks, completely without error back-propagation. The method is a step towards biologically plausible machine learning because the error signal is almost local, and no symmetric or reciprocal weights are required. Experiments show that the test performance on MNIST and CIFAR is almost as good as those obtained with back-propagation for fully connected networks. If combined with dropout, the method achieves 1.45% error on the permutation invariant MNIST task.

Subfields: Artificial Intelligence, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

Citations in the corpus

Why Should We Add Early Exits to Neural Networks? (2020, background): “Point (ii) has been explored in-depth in the literature on feedback alignment (and similar) [ 41, 52, 5 ] , in which it is shown that the transpose of the weight matrix can be replaced by a (ﬁxed) random matrix during the back-ward pass while still allowing for convergence of the algorithm.”
Training Neural Networks Using Features Replay (2018, background): “[1, 27] broke the local dependencies between successive layers and made all hidden layers receive error information from the output layer directly.”
Decoupled Neural Interfaces using Synthetic Gradients (2016, methodology): “…to use other styles of critics or error approximation techniques such as Feedback Alignment (Lillicrap et al., 2016), Direct Feedback Alignment (Nøkland, 2016), and Kickback (Balduzzi et al., 2014b)) – interestingly (Czarnecki et al., 2017) shows that they can all be framed in the synthetic…”
Learning Rapid-Temporal Adaptations (2017, methodology): “Direct feedback meta information Direct feedback (DF) meta information is inspired by feed-back alignment methods [21, 27] and on biologically plausible deep learning [4].”
Deep Supervised Learning Using Local Errors (2017, methodology): “[16,17,53].”
Decoupled Parallel Backpropagation with Convergence Guarantee (2018, background): “(Nøkland, 2016; Balduzzi et al., 2015) broke the local dependencies between successive layers and made all hidden layers receive error information from the output layer directly.”
Hebbian Learning based Orthogonal Projection for Continual Learning of Spiking Neural Networks (2024, background): “There are other variants of FA such as direct feedback alignment (DFA) (Nøkland, 2016) etc., and we consider FA in experiments.”
Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures (2018, methodology): “We consider both the classical variant of FA [23] with random feedback weights at each hidden layer, and the recently proposed Direct Feedback Alignment [25] (DFA) or Broadcast Feedback Alignment [35], which connect feedback from the output layer directly to all previous layers directly.”
Training Neural Networks with Local Error Signals (2019, background): “…plausible algorithms for error-driven learning have so far focused on how to transport the error back to hidden layers (Lillicrap et al., 2014; Nøkland, 2016), or how to transport a target back to hidden layers (Xie & Seung, 2003; Bengio, 2014; Lee et al., 2015b; Scellier & Bengio, 2017;…”
Decoupled Greedy Learning of CNNs (2019, methodology): “One class of alternatives to standard back-propagation aims to avoid its biologically implausible aspects, most notably the weight transport problem (Bartunov et al., 2018; Nø kland, 2016; Lillicrap et al., 2014; Lee et al., 2014).”
Rapid Adaptation with Conditionally Shifted Neurons (2017, methodology): “Direct feedback information Direct feedback (DF) information is inspired by feedback alignment methods (Lillicrap et al., 2016; Nøkland, 2016) and biologically plausible deep learning (Bengio et al.”
Understanding Synthetic Gradients and Decoupled Neural Interfaces (2017, methodology): “DFA does the same, except each layer directly uses the learning signal from the output layer rather than the subsequent local one.”
Backprop Evolution (2018, background): “A few attempts have been made to change the back-propagation equation with some degrees of success (e.g., Bengio et al. (1994); Lillicrap et al. (2014); Lee et al. (2015); Nøkland (2016); Liao et al. (2016)).”
Material for Decoupled Neural Interfaces using Synthetic Gradients (2017, methodology): “…to use other styles of critics or error approximation techniques such as Feedback Alignment (Lillicrap et al., 2016), Direct Feedback Alignment (Nøkland, 2016), and Kickback (Balduzzi et al., 2014)) – interestingly Czarnecki et al. (2017) shows that they can all be framed in the synthetic…”
Associated Learning: Decomposing End-to-End Backpropagation Based on Autoencoders and Target Propagation (2019, background): “Later, the authors in [12] suggested delivering error signals directly from the output layer using fixed weights.”
Dendritic cortical microcircuits approximate the backpropagation algorithm (2018, background): “Such weight symmetry is not essential for successful learning in a broad range of problems, as demonstrated in the main simulations and as observed before (Lee et al., 2015; Lillicrap et al., 2016; Nøkland, 2016).”
Surrogate Gradient Learning in Spiking Neural Networks: Bringing the Power of Gradient-based optimization to spiking neural networks (2019, background): “(c) DFA [38] directly propagates the errors from the top layer to the hidden layers.”