Understanding Synthetic Gradients and Decoupled Neural Interfaces

By Wojciech M. Czarnecki, Grzegorz Swirszcz, Max Jaderberg, Simon Osindero, Oriol Vinyals, Koray Kavukcuoglu. 2017

94 citations recorded by the source corpus.

When training neural networks, the use of Synthetic Gradients (SG) allows layers or modules to be trained without update locking - without waiting for a true error gradient to be backpropagated - resulting in Decoupled Neural Interfaces (DNIs). This unlocked ability of being able to update parts of a neural network asynchronously and with only local information was demonstrated to work empirically in Jaderberg et al (2016). However, there has been very little demonstration of what changes DNIs and SGs impose from a functional, representational, and learning dynamics point of view. In this paper, we study DNIs through the use of synthetic gradients on feed-forward networks to better understand their behaviour and elucidate their effect on optimisation. We show that the incorporation of SGs does not affect the representational strength of the learning system for a neural network, and prove the convergence of the learning system for linear and deep linear models. On practical problems we investigate the mechanism by which synthetic gradient estimators approximate the true loss, and, surprisingly, how that leads to drastically different layer-wise representations. Finally, we also expose the relationship of using synthetic gradients to other error approximation techniques and find a unifying language for discussion and comparison.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

Citations in the corpus

Decoupled Neural Interfaces using Synthetic Gradients (2016, methodology): “It should be noted that while this paper introduces and shows empirical justification for the efficacy of DNIs and synthetic gradients, the work of (Czarnecki et al., 2017) delves deeper into the analysis and theoretical understanding of DNIs and synthetic gradients, confirming the convergence prope”
Decoupled Greedy Learning of CNNs (2019, methodology): “Jaderberg et al. (2017); Czarnecki et al. (2017) pro-pose and analyze DNI, a method that addresses the more challenging update locking.”
Optimization Landscape and Expressivity of Deep CNNs (2017, background): “In particular, (Czarnecki et al., 2017) have shown empirically that linear separability is often obtained already in the first few hidden layers of the trained networks.”
The loss surface and expressivity of deep convolutional neural networks (2017, background): “In particular, Czarnecki et al. (2017) have shown empirically that linear separability is often obtained already in the ﬁrst few hidden layers of the trained networks.”
Decoupled Parallel Backpropagation with Convergence Guarantee (2018, background): “For example, (Jaderberg et al., 2016; Czarnecki et al., 2017) proposed to remove the lockings in backpropagation by employing additional neural networks to approximate error gradients.”
Material for Decoupled Neural Interfaces using Synthetic Gradients (2017, methodology): “…approximation techniques such as Feedback Alignment (Lillicrap et al., 2016), Direct Feedback Alignment (Nøkland, 2016), and Kickback (Balduzzi et al., 2014)) – interestingly Czarnecki et al. (2017) shows that they can all be framed in the synthetic gradients framework presented in this paper.”