Decoupled Parallel Backpropagation with Convergence Guarantee

By Zhouyuan Huo, Bin Gu, Qian Yang, Heng Huang. 2018

107 citations recorded by the source corpus.

Backpropagation algorithm is indispensable for the training of feedforward neural networks. It requires propagating error gradients sequentially from the output layer all the way back to the input layer. The backward locking in backpropagation algorithm constrains us from updating network layers in parallel and fully leveraging the computing resources. Recently, several algorithms have been proposed for breaking the backward locking. However, their performances degrade seriously when networks are deep. In this paper, we propose decoupled parallel backpropagation algorithm for deep learning optimization with convergence guarantee. Firstly, we decouple the backpropagation algorithm using delayed gradients, and show that the backward locking is removed when we split the networks into multiple modules. Then, we utilize decoupled parallel backpropagation in two stochastic methods and prove that our method guarantees convergence to critical points for the non-convex problem. Finally, we perform experiments for training deep convolutional neural networks on benchmark datasets. The experimental results not only confirm our theoretical analysis, but also demonstrate that the proposed method can achieve significant speedup without loss of accuracy.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression (2019, methodology): “Some works also investigate methods for parallel backpropgation (Huo et al., 2018; Li et al., 2018).”
Training Neural Networks Using Features Replay (2018, methodology): “In [11], the authors proposed decoupled parallel backpropagation by using stale gradients, where modules are updated with the gradients from different timestamps (Figure 1 DDG).”
Decoupled Greedy Learning of CNNs (2019, methodology): “We use the exact optimization and network split points as in (Huo et al., 2018b).”
You Only Propagate Once: Accelerating Adversarial Training via Maximal Principle (2019, methodology): “This includes ADMM [34], synthetic gradients [14], delayed gradient [12], lifted machines [1,9,19].”
Associated Learning: Decomposing End-to-End Backpropagation Based on Autoencoders and Target Propagation (2019, background): “Thus, many studies have suggested replacing backpropagation by more biologically plausible methods or by gradient-free methods [24] in the hope of decreasing the computational time and memory consumption [23, 25, 26].”
FLOP: Federated Learning on Medical Datasets using Partial Networks (2021, methodology): “In future work, we intend to accelerate the training of the models following the techniques in [13, 35] and apply our algorithm to other tasks.”
torchgpipe: On-the-fly Pipeline Parallelism for Training Giant Models (2020, methodology): “It is benefitting to combine different types of parallelization strategies [16, 14, 26, 12, 9, 11, 7], and recent lines of research questions how to find an optimal strategy”
PipeDream: generalized pipeline parallelism for DNN training (2019, background): “[29] explored parallelizing the backward pass during training.”