Experiments on Parallel Training of Deep Neural Network using Model Averaging

By Hang Su. 2015

74 citations recorded by the source corpus.

In this work we apply model averaging to parallel training of deep neural network (DNN). Parallelization is done in a model averaging manner. Data is partitioned and distributed to different nodes for local model updates, and model averaging across nodes is done every few minibatches. We use multiple GPUs for data parallelization, and Message Passing Interface (MPI) for communication between nodes, which allows us to perform model averaging frequently without losing much time on communication. We investigate the effectiveness of Natural Gradient Stochastic Gradient Descent (NG-SGD) and Restricted Boltzmann Machine (RBM) pretraining for parallel training in model-averaging framework, and explore the best setups in term of different learning rate schedules, averaging frequencies and minibatch sizes. It is shown that NG-SGD and RBM pretraining benefits parameter-averaging based model training. On the 300h Switchboard dataset, a 9.3 times speedup is achieved using 16 GPUs and 17 times speedup using 32 GPUs with limited decoding accuracy loss.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

In ML Systems & Hardware

Citations in the corpus

Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization (2019, background): “Advantages of model averaging have been studied from an empirical point of view in [27, 7, 24, 35, 17, 20].”
On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization (2019, methodology): “Such a method is also faithful to the common practice “model averaging with momentum” used by practitioners for training deep neural networks in (Chen & Huo, 2016; McMahan et al., 2017; Su et al., 2018; Lin et al., 2018).”
Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms (2018, methodology): “…idea is to allow workers to perform τ local updates to the model instead of just computing gradients, and then periodically averaging the local models (Moritz et al., 2015; Zhang et al., 2016; Povey et al., 2014; Su & Chen, 2015; Chaudhari et al., 2017; Smith et al., 2018; Lin et al., 2018).”
Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learning (2018, background): “…average to obtain y every I iterations and performs local SGDs at each worker 3After the preprint (Yu, Yang, and Zhu 2018) of this paper is posted on ArXiv in July 2018, another work (Wang and Joshi 2018) subsequently analyzes the convergence rate of model averaging for non-convex optimization.”
Distributed deep learning networks among institutions for medical imaging (2018, methodology): “11 In asynchronous stochastic gradient descent, separate models are trained for each split of the data and the gradients of each separate model are transferred to a central model.”
Opportunities and obstacles for deep learning in biology and medicine (2017, background): “The massively parallel nature of GPUs allows additional optimizations, such as accelerated mini-batch gradient descent [379,380,383,384].”
Collaborative Deep Learning in Fixed Topology Networks (2017, background): “Natural Gradient Stochastic Gradient Descent (NG-SGD) based on model averaging [14] and staleness-aware async-SGD [15] have also been developed for distributed deep learning.”
Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGD (2018, background): “Periodic averaging has been shown to offer signiﬁcant speedup in deep neural network training (Moritz et al., 2015; Zhang et al., 2016; Su & Chen, 2015; Zhou & Cong, 2017; Lin et al., 2018).”
On the Convergence of Local Descent Methods in Federated Learning (2019, background): “Empirical advantages of model averaging is studied in [5,13,23,25,27,32].”
Variance Reduced Local SGD with Lower Communication Complexity (2019, methodology): “Local SGD has a more superior training speed performance than S-SGD, which has been empirically observed in various machine learning tasks [Povey et al., 2014, Su and Chen, 2015 Figure 2: Epoch loss for the identical case .”
A Linear Speedup Analysis of Distributed Deep Learning with Sparse and Quantized Communication (2018, methodology): “This approach has been used in practice [34, 43]; however, its convergence has not been well studied.”
Preparing Medical Imaging Data for Machine Learning (2020, background): “Different models are trained on each split of the data and finally the gradients are transferred to a central model (99,100).”
A Survey on Deep Learning (2018, methodology): “A more popular approach for data parallelism uses SGD and is known as update-based data parallelism [149], where the updates of the learning rate decay and momentum are transferred.”
Parallel Restarted SGD for Non-Convex Optimization with Faster Convergence and Less Communication (2018, background): “Various experimental results [12, 1, 11, 14, 5] observe that model averaging can provide superior performance for deep learning applications.”