Parallel training of Deep Neural Networks with Natural Gradient and Parameter Averaging

By Daniel Povey, Xiaohui Zhang, Sanjeev Khudanpur. 2014

250 citations recorded by the source corpus.

We describe the neural-network training framework used in the Kaldi speech recognition toolkit, which is geared towards training DNNs with large amounts of training data using multiple GPU-equipped or multicore machines. In order to be as hardwareagnostic as possible, we needed a way to use multiple machines without generating excessive network traffic. Our method is to average the neural network parameters periodically (typically every minute or two), and redistribute the averaged parameters to the machines for further training. Each machine sees different data. By itself, this method does not work very well. However, we have another method, an approximate and efficient implementation of Natural Gradient for Stochastic Gradient Descent (NG-SGD), which seems to allow our periodic-averaging method to work well, as well as substantially improving the convergence of SGD on a single machine.

Subfields: Artificial Intelligence, Optimization & Training, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

In Signal Processing

Citations in the corpus

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift (2015, methodology): “…whitening activations at every training step or at some interval, either by modifying the network directly or by changing the parameters of the optimization algorithm to depend on the network activation values (Wiesler et al., 2014; Raiko et al., 2012; Povey et al., 2014; Desjardins & Kavukcuoglu).”
Optimizing Neural Networks with Kronecker-factored Approximate Curvature (2015, methodology): “tational efﬁciency (see Section 8). We have found that each of these additional elements is important in order for K-FAC to work as 34 well as it does in various settings. Concurrently with this work Povey et al. (2015) has developed a neural network optimization method which uses a block-diagonal K”
Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization (2019, background): “Advantages of model averaging have been studied from an empirical point of view in [27, 7, 24, 35, 17, 20].”
Federated Learning for Keyword Spotting (2018, background): “The multi-core and multi-gpu distributed training setting has been speciﬁcally studied in the context of speech recognition in [4].”
Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms (2018, methodology): “, 2016), also referred to as model averaging (Povey et al., 2014), uses periodic communication to reduce the communication.”
Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learning (2018, background): “g the communication cost, averaging models periodically also has the advantage of reducing privacy and security risks in the federated learning scenario proposed in [11]. Various experimental results [12, 1, 11, 14, 5] observe that model averaging can provide superior performance for deep learning a”
cpSGD: Communication-efficient and differentially-private distributed SGD (2018, background): “Distributed stochastic gradient descent (SGD) is a basic building block of modern machine learning [25, 11, 9, 28, 1, 27, 5].”
Distributed Mean Estimation with Limited Communication (2016, background): “In a typical scenario of synchronized distributed learning, each client obtains a copy of a global model.”
Communication-Efficient Learning of Deep Networks from Decentralized Data (2016, neutral): “We investigate both of these approaches, but the speedups we achieve are due primarily to adding more computation on each client, once a minimum level of parallelism over clients is used.”
A Kronecker-factored approximate Fisher matrix for convolution layers (2016, methodology): “Our procedure is perhaps more closely analogous to earlier Kronecker product-based natural gradient approximations (Heskes, 2000; Povey et al., 2015) than to K-FAC itself.”
Natural Neural Networks (2015, neutral): “Similar techniques can be found in the preconditioning of the Kaldi speech recognition toolkit [16].”
Experiments on Parallel Training of Deep Neural Network using Model Averaging (2015, methodology): “We also show that model averaging beneﬁts a lot from NG-SGD and RBM based pretraining.”
Variance Reduced Local SGD with Lower Communication Complexity (2019, methodology): “Local SGD has a more superior training speed performance than S-SGD, which has been empirically observed in various machine learning tasks [Povey et al., 2014, Su and Chen, 2015 Figure 2: Epoch loss for the identical case .”
Deep neural network-based speaker embeddings for end-to-end speaker verification (2016, methodology): “, x2N are computed, and backpropagated to the DNN for parallelized stochastic gradient descent [28].”
Gradient-Leaks: Understanding and Controlling Deanonymization in Federated Learning (2018, background): “n Hk() (2) With respect to this setting, we discuss the data distribution in Section 3.3 and the optimization protocol in Section 3.4. 3.3 Decentralized Data Scenarios Distributed learning algorithms [15, 20, 42, 45, 58, 65, 76] typically assume data among the clients are (a) independent and identic”
Ensemble-Compression: A New Method for Parallel Training of Deep Neural Networks (2016, methodology): “3 and previous studies [20, 3] also revealed such problem.”
A Linear Speedup Analysis of Distributed Deep Learning with Sparse and Quantized Communication (2018, methodology): “This approach has been used in practice [34, 43]; however, its convergence has not been well studied.”
Fast Transformers with Clustered Attention (2020, methodology): “Following common practice for ﬂat-start lattice-free MMI training, we train over multiple gpus with weight averaging for synchronization as described in [22].”