Training Neural Networks Using Features Replay

By Zhouyuan Huo, Bin Gu, Heng Huang. 2018

78 citations recorded by the source corpus.

Training a neural network using backpropagation algorithm requires passing error gradients sequentially through the network. The backward locking prevents us from updating network layers in parallel and fully leveraging the computing resources. Recently, there are several works trying to decouple and parallelize the backpropagation algorithm. However, all of them suffer from severe accuracy loss or memory explosion when the neural network is deep. To address these challenging issues, we propose a novel parallel-objective formulation for the objective function of the neural network. After that, we introduce features replay algorithm and prove that it is guaranteed to converge to critical points for the non-convex problem under certain conditions. Finally, we apply our method to training deep convolutional neural networks, and the experimental results show that the proposed method achieves {faster} convergence, {lower} memory consumption, and {better} generalization error than compared methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

In Meta & Continual Learning

Citations in the corpus

Advances and Open Problems in Federated Learning (2019, background): “The ideas in [213, 207], where the authors break the dependencies between partial networks and reduced total centralized training time by parallelizing the computations in different parts, can be relevant here as well.”
Associated Learning: Decomposing End-to-End Backpropagation Based on Autoencoders and Target Propagation (2019, background): “Thus, many studies have suggested replacing backpropagation by more biologically plausible methods or by gradient-free methods [28] in the hope of decreasing the computational time and memory consumption [27, 29, 30].”
Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform (2018, methodology): “[43, 44] also proposed algorithms to break backward locking so that the backward pass can be performed parallel, which enables parallelism in model parallelism.”
Decoupled Greedy Learning of CNNs (2019, methodology): “We use the exact optimization and network split points as in (Huo et al., 2018b).”