Learning in the Machine: Random Backpropagation and the Deep Learning Channel

By Pierre Baldi, Peter Sadowski, Zhiqin Lu. 2016

72 citations recorded by the source corpus.

Random backpropagation (RBP) is a variant of the backpropagation algorithm for training neural networks, where the transpose of the forward matrices are replaced by fixed random matrices in the calculation of the weight updates. It is remarkable both because of its effectiveness, in spite of using random matrices to communicate error information, and because it completely removes the taxing requirement of maintaining symmetric weights in a physical neural system. To better understand random backpropagation, we first connect it to the notions of local learning and learning channels. Through this connection, we derive several alternatives to RBP, including skipped RBP (SRPB), adaptive RBP (ARBP), sparse RBP, and their combinations (e.g. ASRBP) and analyze their computational complexity. We then study their behavior through simulations using the MNIST and CIFAR-10 bechnmark datasets. These simulations show that most of these variants work robustly, almost as well as backpropagation, and that multiplication by the derivatives of the activation functions is important. As a follow-up, we study also the low-end of the number of bits required to communicate error information over the learning channel. We then provide partial intuitive explanations for some of the remarkable properties of RBP and its variations. Finally, we prove several mathematical results, including the convergence to fixed points of linear chains of arbitrary length, the convergence to fixed points of linear autoencoders with decorrelated data, the long-term existence of solutions for linear systems with a single hidden layer and convergence in special cases, and the convergence to fixed points of non-linear chains, when the derivative of the activation functions is included.

Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Why Should We Add Early Exits to Neural Networks? (2020, background): “Point (ii) has been explored in-depth in the literature on feedback alignment (and similar) [5, 39, 49], in which it is shown that the transpose of the weight matrix can be replaced by a (fixed) random matrix during the backward pass while still allowing for convergence of the algorithm.”
Advancing Neuromorphic Computing With Loihi: A Survey of Results and Outlook (2021, background): “…synthetic gradients [45], eligibility propagation as a way to circumvent locking [47], skip connections in the backward path to avoid backpropagating data through multiple layers [48], or only adapting pretrained output layers during online learning instead of training an entire DNN from scratch.”
Understanding Synthetic Gradients and Decoupled Neural Interfaces (2017, background): “…SGs and other forms of approximate error propagation such as Feedback Alignment (Lillicrap et al., 2016), Direct Feedback Alignment (Nøkland, 2016; Baldi et al., 2016), and Kickback (Balduzzi et al., 2014), and show that all these error approximation schemes can be captured in a uniﬁed…”