Horovod: fast and easy distributed deep learning in TensorFlow

By Alexander Sergeev, Mike Del Balso. 2018

1,401 citations recorded by the source corpus.

Training modern deep learning models requires large amounts of computation, often provided by GPUs. Scaling computation from one GPU to many can enable much faster training and research progress but entails two complications. First, the training library must support inter-GPU communication. Depending on the particular methods employed, this communication may entail anywhere from negligible to significant overhead. Second, the user must modify his or her training code to take advantage of inter-GPU communication. Depending on the training library's API, the modification required may be either significant or minimal. Existing methods for enabling multi-GPU training under the TensorFlow library entail non-negligible communication overhead and require users to heavily modify their model-building code, leading many researchers to avoid the whole mess and stick with slower single-GPU training. In this paper we introduce Horovod, an open source library that improves on both obstructions to scaling: it employs efficient inter-GPU communication via ring reduction and requires only a few lines of modification to user code, enabling faster, easier distributed training in TensorFlow. Horovod is available under the Apache 2.0 license at this https URL

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Optimization & Training

Citations in the corpus

ZeRO: Memory optimizations Toward Training Trillion Parameter Models (2019, methodology): “To address this issue, we use a performance-efficient constant-size fused buffer when the model becomes too large, similar to [29].”
Flower: A Friendly Federated Learning Research Framework (2020, background): “The optimization of distributed training with and without federated concepts has been covered from many angles (Dean et al., 2012; Jia et al., 2018; Chahal et al., 2018; Sergeev & Balso, 2018; Dryden et al., 2016).”
FedML: A Research Library and Benchmark for Federated Machine Learning (2020, background): “Distributed training libraries in PyTorch [40], TensorFlow [41], MXNet [42], and distributed training-specialized libraries such as Horovod [43] and BytePS [44] are designed for distributed training in data centers.”
Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates (2019, methodology): “A common alternative approach of PS is the AllReduce algorithm, which is typically implemented by MPI (Sergeev and Balso, 2018; Walker and Dongarra, 1996).”
Qsparse-Local-SGD: Distributed SGD With Quantization, Sparsification, and Local Computations (2019, methodology): “We build our compression scheme into the Horovod framework [SB18].”
Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learning (2018, methodology): “Our implementation uses Horovod (Sergeev and Del Balso 2018) for inter-worker communication and uses PyTorch 0.4 for algorithm implementations.”
IntelliCode compose: code generation using transformer (2020, methodology): “The ofﬂine training module of the IntelliCode Compose system is implemented as a Python library integrating PyTorch and Horovod with Adasum algorithm for gradient summation [28].”
ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Deep Learning and High Performance Computing (2020, methodology): “Both libraries used the Horovod framework [6] to train the models on distributed clusters such as Summit.”
Ray: A Distributed Framework for Emerging AI Applications (2017, methodology): “16xl instances connected by 25Gbps Ethernet, and workers allocated 4 GPUs per node as done in Horovod [53].”
Deep Leakage from Gradients (2019, background): “In general, distributed training can be classified into two categories: with a parameter server (centralized) [15, 19, 23] and without a parameter server (decentralized) [3, 30, 33]).”
Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes (2018, background): “Horovod [20] introduced gradient fusion strategy to the all-reduce algorithm, which reduces tensor fragmentation and improves bandwidth utilization.”
OpenSeq2Seq: Extensible Toolkit for Distributed and Mixed Precision Training of Sequence-to-Sequence Models (2018, methodology): “…training we support two modes, both following data-parallel approaches with synchronous updates: (1) multi-tower mode in which a separate TensorFlow graph is built on every GPU and (2) Horovod-based mode (Sergeev and Del Balso, 2018) which allows both multi-node as well as multi-GPU executions.”
DAPPLE: a pipelined data parallel approach for training large models (2020, background): “While the performance issue can be alleviated by optimizations such as local gradients accumulation[1, 5, 6] or computation and communication overlap techniques[27, 43],”
CSER: Communication-efficient SGD with Error Reset (2020, methodology): “Thus, GRBS is compatible with AllReduce [18, 24] and parameter server [7, 11, 12].”
A Hitchhiker's Guide On Distributed Training of Deep Neural Networks (2018, background): “It’s wide spread use in production systems like Horovord [38] and Tencent’s Framework [14] make it an important staple in modern distributed training frameworks.”
Wavelet: Efficient DNN Training with Tick-Tock Scheduling (2021, background): “Since parameter server needs human efforts to manually determine the roles of GPUs in use (i.e. either a parameter server or a worker) and is hard to determine server-to-worker ratio (Sergeev & Balso, 2018), collective communication gains more attention and is widely adopted.”