How Important Is Weight Symmetry in Backpropagation?

By Qianli Liao, Joel Z. Leibo, Tomaso A. Poggio. 2015

195 citations recorded by the source corpus.

Gradient backpropagation (BP) requires symmetric feedforward and feedback connections — the same weights must be used for forward and backward passes. This "weight transport problem'' (Grossberg 1987) is thought to be one of the main reasons to doubt BP's biologically plausibility. Using 15 different classification datasets, we systematically investigate to what extent BP really depends on weight symmetry. In a study that turned out to be surprisingly similar in spirit to Lillicrap et al.'s demonstration (Lillicrap et al. 2014) but orthogonal in its results, our experiments indicate that: (1) the magnitudes of feedback weights do not matter to performance (2) the signs of feedback weights do matter — the more concordant signs between feedforward and their corresponding feedback connections, the better (3) with feedback weights having random magnitudes and 100% concordant signs, we were able to achieve the same or even better performance than SGD. (4) some normalizations/stabilizations are indispensable for such asymmetric BP to work, namely Batch Normalization (BN) (Ioffe and Szegedy 2015) and/or a "Batch Manhattan'' (BM) update rule.

Subfields: Artificial Intelligence, Computational Mechanics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

Citations in the corpus

Direct Feedback Alignment Provides Learning in Deep Neural Networks (2016, background): “Back-propagation with asymmetric weights was also explored in [8].”
Towards deep learning with segregated dendrites (2016, methodology): “This direct projection allowed us to build our local targets for each hidden layer using the plateaus driven by the output layer, thereby avoiding a ‘backward pass’ through the entire network as has been used in other models (Lillicrap et al., 2016; Lee et al., 2015; Liao et al., 2015).”
Deep Supervised Learning Using Local Errors (2017, background): “Maintaining sign-alignment is problematic in the feedback alignment technique as the sign of the feedback weights have to dynamically track the sign of the feedforward weights during training [53] which introduces a dynamic dependency between the two sets of weights.”
Backprop Evolution (2018, background): “A few attempts have been made to change the back-propagation equation with some degrees of success (e.g., Bengio et al. (1994); Lillicrap et al. (2014); Lee et al. (2015); Nøkland (2016); Liao et al. (2016)).”
Learning in the Machine: Random Backpropagation and the Deep Learning Channel (2016, background): “[23] recently explored the use of low-precision updates with RBP.”