Deep learning with COTS HPC systems

By Adam Coates, Brody Huval, Tao Wang, David J. Wu, Bryan Catanzaro, Andrew Y. Ng. 2013

755 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

One weird trick for parallelizing convolutional neural networks (2014, methodology): “The work of [Coates et al., 2013] of [Dean et al., 2012], which introduced this particular form of model parallelism for training a locally-connected neural network.”
Scaling Neural Machine Translation (2018, methodology): “Distributed training of neural networks follows two main strategies: (i) model parallel evaluates different model layers on different workers (Coates et al., 2013) and (ii) data parallel keeps a copy of the model on each worker but distributes different batches to different machines (Dean et al.,…”
Beyond human-level accuracy: computational challenges in deep learning (2019, background): “Better cache tiling, kernel optimization and fusion techniques might also help [10, 11].”
Predicting Parameters in Deep Learning (2013, background): “n scaling deep networks has led to the construction of the largest artiﬁcial neural networks to date. It is now possible to train networks with tens of millions [14] or even over a billion parameters [8, 17, 4]. The largest networks (i.e. those of Dean et al. [8]) are trained using asynchronous SGD.”
Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations (2016, background): “Today, DNNs are almost exclusively trained on one or many very fast and power-hungry Graphic Processing Units (GPUs) (Coates et al., 2013).”
Massively Parallel Methods for Deep Reinforcement Learning (2015, background): “However, recent approaches have focused on massively distributed architectures that can learn from more data in parallel and therefore outperform training on a single machine (Coates et al., 2013; Dean et al., 2012).”
Learning both Weights and Connections for Efficient Neural Network (2015, background): “[6] scaled up a network to 10B parameters.”
cuDNN: Efficient Primitives for Deep Learning (2014, background): “The efficiency provided by these implementations allows researchers to explore significantly higher capacity networks, training them on larger datasets [7].”
Deep Learning with Limited Numerical Precision (2015, methodology): “or a combination of CPUs and GPUs scaled-up to multiple nodes (Coates et al., 2013; Wu et al., 2015).”
Compressing Neural Networks with the Hashing Trick (2015, background): “As data sets increase in size, so do the number of parameters in these neural networks in order to absorb the enormous amount of supervision (Coates et al., 2013).”
cpSGD: Communication-efficient and differentially-private distributed SGD (2018, background): “Distributed stochastic gradient descent (SGD) is a basic building block of modern machine learning [25, 11, 9, 28, 1, 27, 5].”
Deep Learning for IoT Big Data and Streaming Analytics: A Survey (2017, background): “[206] proposed a large-scale system, based on a cluster of GPU servers, which can perform the training of neural networks with 1 billion parameters on 3 machines in few days.”
Crypto-Nets: Neural Networks over Encrypted Data (2014, background): “Even without encryption, high throughput computing units such as GPUs or multi-node clusters are needed to make learning neural nets feasible on large datasets (Dean et al., 2012; Coates et al., 2013).”
Opportunities and obstacles for deep learning in biology and medicine (2017, background): “There now exist a number of algorithms [375,392,393], tools [394–396], and high-level libraries [397,398] for deep learning in a distributed environment, and it is possible to train very complex networks with limited infrastructure [399].”
Binarized Neural Networks (2016, background): “Today, DNNs are almost exclusively trained on one or many very fast and power-hungry Graphic Processing Units (GPUs) (Coates et al., 2013).”