Pipelined Backpropagation at Scale: Training Large Models without Batches
By Atli Kosson, Vitaliy Chiley, Abhinav Venigalla, Joel Hestness, Urs Koster. 2020
39 citations recorded by the source corpus.
New hardware can substantially increase the speed and efficiency of deep neural network training. To guide the development of future hardware architectures, it is pertinent to explore the hardware and machine learning properties of alternative training algorithms. In this work we evaluate the use of small batch, fine-grained Pipelined Backpropagation, an asynchronous pipeline parallel training algorithm that has significant hardware advantages. We introduce two methods, Spike Compensation and Linear Weight Prediction, that effectively mitigate the downsides caused by the asynchronicity of Pipelined Backpropagation and outperform existing techniques in our setting. We show that appropriate normalization and small batch sizes can also aid training. With our methods, fine-grained Pipelined Backpropagation using a batch size of one can match the accuracy of SGD for multiple networks trained on CIFAR-10 and ImageNet. Simple scaling rules allow the use of existing hyperparamaters for traditional training without additional tuning.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohammad Shoeybi
- Bryan Catanzaro
- Matei A. Zaharia
- M. Patwary
- D. Narayanan
- Amar Phanishayee
- Jared Casper
- Vijay Anand Korthikanti
In Computer Vision and Pattern Recognition
- Matei A. Zaharia
- D. Narayanan
- Mohammad Shoeybi
- M. Patwary
- Bryan Catanzaro
- Jared Casper
- Amar Phanishayee
- Dehao Chen
In Efficiency & Compression
- Mohammad Shoeybi
- Matei A. Zaharia
- Bryan Catanzaro
- M. Patwary
- D. Narayanan
- Vijay Anand Korthikanti
- Yuanzhong Xu
- Amar Phanishayee
In Optimization & Training
- Mohammad Shoeybi
- Bryan Catanzaro
- M. Patwary
- Matei A. Zaharia
- Jinliang Wei
- Shibo Wang
- Yuanzhong Xu
- Zongwei Zhou
Citations in the corpus
- Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM (2021, methodology): “Pipeline parallelism can also be implemented with relaxed semantics: PipeDream-2BW [30] maintains two weight versions and guarantees 1-stale weight updates without expensive flushes, while PipeMare [45] and Kosson et al. [23] use asynchoronous pipeline parallelism.”
- Overlap Communication with Dependent Computation via Decomposition in Large Deep Learning Models (2022, methodology): “Pipeline model parallelism [24, 29, 33, 34, 53] is another technique that have been proposed for large deep learning models to address the scaling challenges.”