GeePS: scalable deep learning on distributed GPUs with a GPU-specialized parameter server

By Henggang Cui, H. Zhang, G. R. Ganger, Phillip B. Gibbons, Eric P. Xing. 2016

337 citations recorded by the source corpus.

Large-scale deep learning requires huge computational resources to train a multi-layer neural network. Recent systems propose using 100s to 1000s of machines to train networks with tens of layers and billions of connections. While the computation involved can be done more efficiently on GPUs than on more traditional CPU cores, training such networks on a single GPU is too slow and training on distributed GPUs can be inefficient, due to data movement overheads, GPU stalls, and limited GPU memory. This paper describes a new parameter server, called GeePS, that supports scalable deep learning across GPUs distributed among multiple machines, overcoming these obstacles. We show that GeePS enables a state-of-the-art single-node GPU implementation to scale well, such as to 13 times the number of training images processed per second on 16 machines (relative to the original optimized single-node code). Moreover, GeePS achieves a higher training throughput with just four GPU machines than that a state-of-the-art CPU-only system achieves with 108 machines.

Subfields: Efficiency & Compression, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

In Optimization & Training

Citations in the corpus

TensorFlow: A system for large-scale machine learning (2016, positive): “We have extended the algorithm to differentiate conditional and iterative subcomputations ( § 3.4), and developed techniques for managing GPU memory when iterating (and accumulating intermediate values) over long sequences in the input data (similar to GeePS [19]).”
PipeDream: Fast and Efficient Pipeline Parallel DNN Training (2018, methodology): “Each GPU maintains a full copy of the model and trains on its own partition of data while periodically synchronizing weights with other GPUs, using either collective communication primitives [14] or Parameter Servers [28, 9].”
The Non-IID Data Quagmire of Decentralized Machine Learning (2019, methodology): “Evaluation Results We implement and evaluate SkewScout in a GPU parameter server system (Cui et al., 2016) based on Caffe (Jia et al.”
Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training (2018, background): “The fraction of time spent on communication is further increased when applying accelerators [16, 7, 38, 52, 48, 49, 44], as they decrease computation time while leaving communication time untouched.”
Asynchrony begets momentum, with an application to deep learning (2016, background): “Let vt denote the value read by the worker in charge of step and update t.”
Toward Understanding the Impact of Staleness in Distributed Machine Learning (2018, background): “wever, by allowing various workers to use stale versions of the model that do not always reﬂect the latest updates, non-synchronous systems can exhibit lower statistical efﬁciency (Chen et al., 2016; Cui et al., 2016). How statistical efﬁciency and system throughput trade off in distributed systems,”
DeepChain: Auditable and Privacy-Preserving Deep Learning with Blockchain-Based Incentive (2019, methodology): “To alleviate this problem, distributed deep learning training has been proposed recently, and some research work [30], [31], [32], [33], [34] and system implementations have been around, such as DistBelief [35], Torch [36], DeepImage [37] and Purine [38].”
Capuchin: Tensor-based GPU Memory Management for Deep Learning (2020, background): “[7, 21] tries to virtualize the GPU memory via leveraging host memory as the extended memory.”
Optimizing Network Performance for Distributed DNN Training on GPU Clusters: ImageNet/AlexNet Training in 1.5 Minutes (2019, methodology): “Various distributed DNN systems have been proposed to accelerate model training, such as MxNet [7], PyTorch [8], TensorFlow [9] and Petuum [10], Adam [11] and GeePS [12].”
Efficient Memory Management for GPU-based Deep Learning Systems (2019, methodology): “In GeePS [35], the decision to swap which layer or which tensor is made by the end user.”
Asynchronous Byzantine Machine Learning (2018, methodology): “We consider the general distributed model for machine learning, namely the parameter server (Dean et al., 2012; Abadi et al., 2016; Li et al., 2013; 2014b;a; Ho et al., 2013; Cui et al., 2016)1.”
PipeDream: generalized pipeline parallelism for DNN training (2019, background): “This approach improves hardware efficiency (time needed per iteration) over BSP by overlapping computation with communication, but also introduces staleness and reduces statistical efficiency (number of iterations needed to reach a particular target accuracy) [12, 20].”
Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systems (2020, methodology): “017). In addition, the data batch is sharded into multiple mini-batches and sent to each GPU worker thread (line5-10). Many recent machine learning system studies (Ho et al.,2013;Chilimbi et al.,2014;Cui et al., 2016;Alistarh et al.,2018) suggest that the parameter staleness shared among workers in ”
PipeDream: Pipeline Parallelism for DNN Training (2018, methodology): “While data parallel training works well with some popular models [1, 3], expected growth in DNNmodel sizes increases the overhead of communication.”
Gaia: Geo-Distributed Machine Learning Approaching LAN Speeds (2017, background): “…running across 11 Amazon EC2 global regions and on a cluster that emulates EC2 WAN bandwidth show that Gaia provides 1.8–53.5× speedup over two state-of-the-art distributed ML systems, and is within 0.94–1.40× of the speed of running the same ML algorithm on machines on a local area network (LAN).”
TicTac: Accelerating Distributed Deep Learning with Communication Scheduling (2018, methodology): “Smarter interleaving of computation and communication: Several layer-by-layer systems (Arnold, 2016; Cui et al., 2016; Zhang et al., 2017), where the models are sequential and obtaining the order is trivial (Cui et al.”