FireCaffe: Near-Linear Acceleration of Deep Neural Network Training on Compute Clusters

By Forrest N. Iandola, Matthew W. Moskewicz, Khalid Ashraf, Kurt Keutzer. 2015

312 citations recorded by the source corpus.

Long training times for high-accuracy deep neural networks (DNNs) impede research into new DNN architectures and slow the development of high-accuracy DNNs. In this paper we present FireCaffe, which successfully scales deep neural network training across a cluster of GPUs. We also present a number of best practices to aid in comparing advancements in methods for scaling and accelerating the training of deep neural networks. The speed and scalability of distributed algorithms is almost always limited by the overhead of communicating between servers, DNN training is not an exception to this rule. Therefore, the key consideration here is to reduce communication overhead wherever possible, while not degrading the accuracy of the DNN models that we train. Our approach has three key pillars. First, we select network hardware that achieves high bandwidth between GPU servers - Infiniband or Cray interconnects are ideal for this. Second, we consider a number of communication algorithms, and we find that reduction trees are more efficient and scalable than the traditional parameter server approach. Third, we optionally increase the batch size to reduce the total quantity of communication during DNN training, and we identify hyperparameters that allow us to reproduce the small-batch accuracy while training with large batch sizes. When training GoogLeNet and Network-in-Network on ImageNet, we achieve a 47x and 39x speedup, respectively, when training on a cluster of 128 GPUs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In ML Systems & Hardware

Citations in the corpus

SCAFFOLD: Stochastic Controlled Averaging for On-Device Federated Learning (2019, background): “Unlike standard settings where models are trained using large datasets stored in a central server (Dean et al., 2012; Iandola et al., 2016; Goyal et al., 2017), in federated learning, the training data remains distributed over a large number of clients, which may be phones, other mobile devices,…”
MATCHA: Speeding Up Decentralized SGD via Matching Decomposition Sampling (2019, methodology): “There have been massive amount of work in the context of algorithms [37, 33, 36, 5, 26] and systems [16, 43, 9] that improve the communication efficiency of synchronous distributed SGD in a fully-connected network.”
PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization (2019, methodology): “Synchronous data-parallel SGD is the most common method for accelerating training of deep learning models (Dean et al., 2012; Iandola et al., 2015; Goyal et al., 2017).”
Distributed Learning over Unreliable Networks (2018, background): “2.3 AllReduce SGD and Decentralized Learning To overcome the limitations of the parameter-server model (Li et al., 2014), recent research, e.g.(Iandola et al., 2016) has explored communication patterns which do not depend on the existence of a coordinator node.”
Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks (2018, background): “…high-end GPUs available (NVIDIA P40, P100, K80, and GTX 1080), even if a less powerful GPU is used (say, one that is 10X slower) we can see that ﬁne-pruning is still signiﬁcantly more efﬁcient than training from scratch, which can take several days in the case of large models such as AlexNet [22].”
QSGD: Communication-Optimal Stochastic Gradient Descent, with Applications to Training Neural Networks (2016, background): “It gives upper bounds on the error probability of SGD, assuming unbiasedness, as well as sparsity bounds on the gradients.”
Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGD (2018, methodology): “For example, in the parameter server framework, the communication delay can be proportional to 2 log2(m) by exploiting a reduction tree structure (Iandola et al., 2016).”
SparkNet: Training Deep Networks in Spark (2015, background): “However, it is a frequently-used approach to parallelizing SGD (Noel et al., 2015; Iandola et al., 2015), especially when asynchronous updates are not an option (as in computational frameworks like MapReduce and Spark).”
Deep Leakage from Gradients (2019, background): “In general, distributed training can be classified into two categories: with a parameter server (centralized) [15, 19, 23] and without a parameter server (decentralized) [3, 30, 33]).”
How to scale distributed deep learning? (2016, background): “Examples of large-scale synchronous data parallel SGD for distributed deep learning are given in [1], [12], [2], [3], and [4].”
Demystifying Parallel and Distributed Deep Learning (2018, methodology): “ently, and thus the overall algorithm has to be Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis 1:23 Category Method Model Consistency Synchronization Synchronous[45,112,173,210,224,246] Stale-Synchronous[88,98,120,156,261] Asynchronous[55,56,127,182,190,205,260”
On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent (2018, background): “In the face of unprecedented growth in dataset size, a large body of work has attempted to scale SGD to train DNN models on increasingly large datasets, while keeping wall-clock time manageable [13, 9, 29, 7].”
Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training (2018, methodology): “Distributed Training of Deep Nets: To train larger models or to increase the minibatch size, distributed training on multiple compute nodes is used [8, 15, 6, 27, 28, 36, 16].”
Hessian-based Analysis of Large Batch Training and Robustness to Adversaries (2018, methodology): “In [19], an approximate numerical method was used to approximate the”
SqueezeBERT: What can computer vision teach NLP about efficient neural networks? (2020, background): “(9)However, reducing the number of parameters while retaining a high number of FLOPs can present other advantages, such as faster distributed training [19, 43] and superior energy-efficiency [44].”
TicTac: Accelerating Distributed Deep Learning with Communication Scheduling (2018, background): “The fraction of computation time relative to communication time can be increased by increasing the batch size (Iandola et al., 2016).”
Optimizing Network Performance for Distributed DNN Training on GPU Clusters: ImageNet/AlexNet Training in 1.5 Minutes (2019, background): “, 56/100 Gbps) for GPUs to exchange gradients [17], [19], [20], [21].”