Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusters

By H. Zhang, Zeyu Zheng, Shizhen Xu, Wei Dai, Qirong Ho, Xiaodan Liang, Zhiting Hu, Jinliang Wei, Pengtao Xie, Eric P. Xing. 2017

378 citations recorded by the source corpus.

Deep learning models can take weeks to train on a single GPU-equipped machine, necessitating scaling out DL training to a GPU-cluster. However, current distributed DL implementations can scale poorly due to substantial parameter synchronization over the network, because the high throughput of GPUs allows more data batches to be processed per unit time than CPUs, leading to more frequent network synchronization. We present Poseidon, an efficient communication architecture for distributed DL on GPUs. Poseidon exploits the layered model structures in DL programs to overlap communication and computation, reducing bursty network communication. Moreover, Poseidon uses a hybrid communication scheme that optimizes the number of bytes required to synchronize each layer, according to layer properties and the number of machines. We show that Poseidon is applicable to different DL frameworks by plugging Poseidon into Caffe and TensorFlow. We show that Poseidon enables Caffe and TensorFlow to achieve 15.5x speed-up on 16 single-GPU machines, even with limited bandwidth (10GbE) and the challenging VGG19-22K network for image classification. Moreover, Poseidon-enabled TensorFlow achieves 31.5x speed-up with 32 single-GPU machines on Inception-V3, a 50% improvement over the open-source TensorFlow (20x speed-up).

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Neurology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

Citations in the corpus

Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes (2018, neutral): “[3][23] further improves the system throughput.”
DAPPLE: a pipelined data parallel approach for training large models (2020, background): “We consider pipelines training only if DP optimizations [5], [6], [8], [17]–[20] are unable to achieve satisfactory efficiency, or the model size is too large to fit in a device with a minimum required batch size.”
3LC: Lightweight and Effective Traffic Compression for Distributed Machine Learning (2018, background): “Recent performance studies show that in-datacenter distributed training can demand more band-width than local networks and even GPU interconnects currently oﬀer [3, 25, 39, 41].”
Gradient Compression Supercharged High-Performance Data Parallel DNN Training (2021, background): “Poseidon [82] synchronizes sufficient factors, which are compressed forms of gradients of fully connected layers in CV models.”
Tiresias: A GPU Cluster Manager for Distributed Deep Learning (2019, methodology): “Similar to [45], each job uses eight parameter servers – the same as the number of workers.”
PipeDream: Fast and Efficient Pipeline Parallel DNN Training (2018, methodology): “Despite optimizations such as Wait-free Backpropagation [47], where weight gradients are sent as soon as they are available, (common in modern parameter servers), communication stalls are inevitable in data-parallel training due to the structure of the DNN computation, and the fact that communicatio”
AGGREGATHOR: Byzantine Machine Learning via Robust Gradient Aggregation (2019, methodology): “In particular, we consider the image classiﬁcation task due to its wide adoption as a benchmark for distributed ML literature (Chilimbi et al., 2014; Abadi et al., 2016a; Zhang et al., 2017).”
A generic communication scheduler for distributed DNN training acceleration (2019, methodology): “This is also echoed by recent literature [9, 18, 39].”
Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systems (2020, background): “Posei-don (Zhang et al., 2017) develops two strategies wait-free back-propagation and hybrid communication to exploit the independency between parameters of layered model structures in deep neural networks and conquer communication bottleneck; while Gaia (Hsieh et al., 2017) even considered the…”
Zico: Efficient GPU Memory Sharing for Concurrent DNN Training (2021, background): “They are mostly static [15,25,37,45] and can be easily factored in.”
PipeDream: generalized pipeline parallelism for DNN training (2019, background): “Despite optimizations such as Wait-free Backpropagation [57], where weight gradients are sent as soon as they are available (common in modern frameworks), communication stalls are sometimes inevitable for large models where the time needed to synchronize gradients across workers can dominate…”
TicTac: Accelerating Distributed Deep Learning with Communication Scheduling (2018, methodology): “…of parameter representation (Vanhoucke et al., 2011; Courbariaux et al., 2015; Gupta et al., 2015), changing the network primitives to collective (e.g. all reduce) (Goyal et al., 2017; Cho et al., 2017; Amodei et al., 2015; You et al., 2017; Akiba et al., 2017) or broadcast (Zhang et al., 2017).”
SGD: Decentralized Byzantine Resilience (2019, methodology): “In particular, we consider the image classiﬁcation task due to its wide adoption as a benchmark for distributed ML literature [3, 15, 43].”
ATP: In-network Aggregation for Multi-tenant Learning (2021, background): “Prior efforts propose to improve gradient aggregation time by (1) smarter network scheduling – increasing the overlap between GPU/CPU computation and network transmission via fine-grained tensors transmission scheduling (per layer instead of the whole gradient or parameter) [32, 37, 52, 70]; combini”