Large Scale Distributed Deep Networks

By Joseph Dean, Greg S. Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Quoc V. Le, Mark Z. Mao, Marc'Aurelio Ranzato, Andrew W. Senior, Paul A. Tucker and 2 more. 2012

4,179 citations recorded by the source corpus.

Subfields: Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (2016, methodology): “Data parallelism is straightforward: we train n model replicas concurrently using a Downpour SGD algorithm [12].”
Layer Normalization (2016, background): “It is possible to speed-up the learning by computing gradients for different subsets of the training cases on different machines or splitting the neural network itself over many machines [Dean et al., 2012], but this can require a lot of communication and complex software.”
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift (2015, methodology): “The training was performed using a large-scale, distributed architecture (similar to (Dean et al., 2012)).”
Going deeper with convolutions (2014, methodology): “Also we are indebted to the DistBelief [4] team for their support especially to Rajat Monga, Jon Shlens, Alex Krizhevsky, Jeff Dean, Ilya Sutskever and Andrea Frome.”
Very Deep Convolutional Networks for Large-Scale Image Recognition (2014, background): “Convolutional networks (ConvNets) have recently enjoyed a great success in large-scale visual recognition [10, 16, 17, 19] which has become possible due to the large public image repositories, such as ImageNet [4], and high-performance computing syste ms, such as GPUs or large-scale distributed clus”
ADADELTA: An Adaptive Learning Rate Method (2012, methodology): “A recent ﬁrst order method called ADAGRAD [3] has shown remarkably good results on large scale learning tasks in a distributed environment [4].”
Pylearn2: a machine learning research library (2013, methodology): “For example, in Torch [9] or DistBelief [14] the Model s are modular, but to train a layer, the layer needs to implement at the very least a backpropogation method for computing derivatives.”
Distilling the Knowledge in a Neural Network (2015, methodology): “For the distillation we tried temperatures of [1,2, 5, 10] and used a relative weight of 0.”
Efficient Estimation of Word Representations in Vector Space (2013, methodology): “y then comes from H H. 3 2.3 Parallel Training of Neural Networks To train models on huge data sets, we have implemented several models on top of a large-scale distributed framework called DistBelief [6], including the feedforward NNLM and the new models proposed in this paper. The framework allows ”
Mesh-TensorFlow: Deep Learning for Supercomputers (2018, background): “Different distribution strategies (model-parallelism [7]) can solve these issues, but specifying these strategies can be complicated, and the current MIMD implementations generate very large programs which can be difficult to compile and to optimize.”
Federated Learning: Strategies for Improving Communication Efficiency (2016, background): “The above framework differs from conventional distributed machine learning [17, 11, 19, 22, 5, 4] due to the the large number of clients, highly unbalanced and non-i.”
One weird trick for parallelizing convolutional neural networks (2014, methodology): “The work of [Coates et al., 2013] of [Dean et al., 2012], which introduced this particular form of model parallelism for training a locally-connected neural network.”
Scaling Neural Machine Translation (2018, methodology): “…training of neural networks follows two main strategies: (i) model parallel evaluates different model layers on different workers (Coates et al., 2013) and (ii) data parallel keeps a copy of the model on each worker but distributes different batches to different machines (Dean et al., 2012).”
ZeRO-Offload: Democratizing Billion-Scale Model Training (2021, methodology): “Model parallelism [5, 28] partitions the model vertically and distributes the model partitions to multiple GPU devices in order to train large models.”
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2020, background): “Software Software systems supporting neural networks evolved together with the advancement of the underlying hardware (Dean et al., 2012; Bastien et al., 2012; Abadi et al., 2016; Paszke et al., 2017; Palkar & Zaharia, 2019).”
Memory-Efficient Pipeline-Parallel DNN Training (2020, background): “One of the solutions researchers and practitioners have turned to is model-parallel training (Dean et al., 2012; Chilimbi et al., 2014), where a model is partitioned over multiple accelerator devices.”
Don't Decay the Learning Rate, Increase the Batch Size (2017, methodology): “ study), while our work reduces the gap in convergence speed. Asynchronous-SGD is another popular strategy, which enables the use of multiple GPUs even when batch sizes are small (Recht et al., 2011; Dean et al., 2012). We do not consider asynchronous-SGD in this work, since the scaling rules enable”
Group Normalization (2018, background): “Moreover, the synchronized BN computation prevents using asynchronous solvers (ASGD [37]), a practical solution to large-scale training widely used in industry.”
Revisiting Unreasonable Effectiveness of Data in Deep Learning Era (2017, methodology): “To allow asynchrounous training of models on 50 GPUs, we adopt the Downpour SGD training scheme [8], where we use 17 parameter servers to store and update the model”
TensorFlow: A system for large-scale machine learning (2016, neutral): “Parameter servers emerged as an architecture for scalable topic modeling [66], and our previous system DistBelief [21] showed how a similar architecture could be applied to deep neural network training.”