Project Adam: Building an Efficient and Scalable Deep Learning Training System

By Trishul M. Chilimbi, Yutaka Suzue, Johnson Apacible, Karthik Kalyanaraman. 2014

827 citations recorded by the source corpus.

Subfields: Optimization & Training, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

In Scaling & Learning Theory

Citations in the corpus

Federated Learning: Strategies for Improving Communication Efficiency (2016, background): “The above framework differs from conventional distributed machine learning [17, 11, 19, 22, 5, 4] due to the the large number of clients, highly unbalanced and non-i.”
TensorFlow: A system for large-scale machine learning (2016, neutral): “Training a model can take several hours or days, even us-ing a large number of machines [21, 14].”
Deep Speech 2 : End-to-End Speech Recognition in English and Mandarin (2015, positive): “In contrast to previous large-scale training approaches that use parameter servers and asynchronous updates [18, 10], we use synchronous SGD, which is easier to debug while testing new ideas, and also converges faster for the same degree of data parallelism.”
PipeDream: Fast and Efficient Pipeline Parallel DNN Training (2018, methodology): “Model parallelism has also been used for DNNs, but traditionally only as a last resort when the working set of model training is too large to fit in a single worker’s memory or cache [25, 7, 10] (making data parallelism not an option).”
Revisiting Distributed Synchronous SGD (2016, methodology): “Both Dean et al. (2012) and Chilimbi et al. (2014) use versions of Async-SGD where the main potential problem is that each worker computes gradients over a potentially old version of the model.”
Deep Learning with Limited Numerical Precision (2015, background): “Some notable eﬀorts in this direction include distributed computing infrastructure using thousands of CPU cores (Dean et al., 2012; Chilimbi et al., 2014), or high-end graphics processors (GPUs) (Krizhevsky & Hinton, 2009), or a combination of CPUs and GPUs scaled-up to multiple nodes (Coates et…”
The Non-IID Data Quagmire of Decentralized Machine Learning (2019, background): “There are many large-scale ML systems that aim to enable efﬁcient ML training with centralized datasets (e.g., (Recht et al., 2011; Low et al., 2012; Chilimbi et al., 2014; Dean et al., 2012; Ho et al., 2013; Cui et al., 2014; Li et al., 2014a; Chen et al., 2015; Goyal et al., 2017)), where the…”
Error Feedback Fixes SignSGD and other Gradient Compression Schemes (2019, background): “However, at these scales communication of the gradients between the machines becomes a bottleneck hindering us from making full use of the impressive computational resources available in today’s data centers Chilimbi et al. [2014], Seide et al. [2014], Strom [2015].”
Don't Use Large Mini-Batches, Use Local SGD (2018, background): “Asynchronous SGD algorithms (Chilimbi et al., 2014; Dean et al., 2012) aim to improve overall training time at the expense of additional noise introduced from asynchrony, i.”
Exploiting Unintended Feature Leakage in Collaborative Learning (2018, background): “A common scaling approach is to partition the training dataset, concurrently train separate models on each subset, and exchange parameters via a parameter server [9, 11].”
DRACO: Byzantine-resilient Distributed Training via Redundant Gradients (2018, background): “Distributed and parallel implementations of stochastic optimization algorithms have become the de facto standard in large-scale model training (Li et al., 2014; Recht et al., 2011; Zhang et al., 2015; Agarwal et al., 2010; Abadi et al., 2016; Chen et al., 2015; Paszke et al., 2017a; Chilimbi et al.,”
Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training (2017, background): “Algorithm 1 Gradient Sparsification on node k Input: dataset χ Input: minibatch size b per node Input: the number of nodes N Input: optimization function SGD Input: init parameters w = {w[0], w[1], · · · , w[M ]} 1: G ← 0 2: for t = 0, 1, · · · do 3: Gt ← Gt−1 4: for i = 1, · · · , b do 5: Sample da”
QSGD: Communication-Optimal Stochastic Gradient Descent, with Applications to Training Neural Networks (2016, methodology): “In practical applications, communicating the gradients in each iteration has been observed to be a signiﬁcant performance bottleneck [31, 33, 7].”
Federated Optimization: Distributed Machine Learning for On-Device Intelligence (2016, methodology): “The practical usefulness has been demonstrated for instance by Google’s Downpour SGD [23] and Microsoft’s Project Adam [18].”
TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning (2017, background): “The training of large-scale models with huge amounts of data are often carried on distributed systems [1][2][3][4][5][6][7][8][9], where data parallelism is adopted to exploit the compute capability empowered by multiple workers [10].”
Perturbed Iterate Analysis for Asynchronous Stochastic Optimization (2015, methodology): “This behavior has been demonstrated in practice in state-of-the-art deep learning systems such as Google’s Downpour SGD [9] and Microsoft’s Project Adam [10].”