Optimization of Collective Reduction Operations

By Rolf Rabenseifner. 2004

223 citations recorded by the source corpus.

Subfields: Computer Networks and Communications, Hardware and Architecture, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

1. Nicolas Loizou
2. Sebastian U. Stich
3. Martin Jaggi
4. Tao Lin
5. Torsten Hoefler
6. Mahmoud Assran
7. Nicolas Ballas
8. Tal Ben-Nun

In Hardware and Architecture

1. Torsten Hoefler
2. Tal Ben-Nun
3. Xin Yuan
4. Song Han
5. Yang You
6. Tao Lin
7. J. Demmel
8. Aapo Kyrola

In Optimization & Training

1. Sebastian U. Stich
2. Martin Jaggi
3. Peter Richtárik
4. Nicolas Loizou
5. Kaiming He
6. Jianyu Wang
7. Ross B. Girshick
8. Torsten Hoefler

Citations in the corpus

• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour (2017, methodology): “For interserver allreduce, we implemented two of the best algorithms for bandwidth-limited scenarios: the recursive halving and doubling algorithm [29, 36] and the bucket algorithm (also known as the ring algorithm) [2].”
• Stochastic Gradient Push for Distributed Deep Learning (2018, methodology): “Typical implementations of ALLREDUCE have each node send and receive 2n−1 n B bytes, where B is the size (in bytes) of the tensor being reduced, and involve 2 log2(n) communication steps [13].”
• Don't Use Large Mini-Batches, Use Local SGD (2018, methodology): “More precisely, we use general all-reduce, e.g., recursive halving and doubling algorithm (Thakur et al., 2005; Rabenseifner, 2004), for gradient aggregation among K computation device.”
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training (2017, methodology): “With the all-reduce communication model [31, 32], the density of sparse data doubles at every aggregation step in the worst case.”
• Demystifying Parallel and Distributed Deep Learning (2018, methodology): “For most ranges of γm and P , one could use Rabenseifner’s algorithm [206], which combines reduce-scatter with”
• Bandwidth optimal all-reduce algorithms for clusters of workstations (2009, background): “Keywords: All-reduce, collective communication, tree topology, cluster of workstations”
• SparCML: High-Performance Sparse Communication for Machine Learning (2018, methodology): “This is implemented by a recursive halving technique [42].”
• Fast Deep Neural Network Training on Distributed Systems and Cloud TPUs (2019, background): “In particular, the all-reduce on N nodes have the scaling factor of OðlogNÞ or OðNÞ depending on the network topology [32], [33], [34], [35].”
• ImageNet Training in Minutes (2017, background): “In particular, the all-reduce on N nodes have the scaling factor of O(logN ) or O(N ) depending on the network topology [25, 26, 30, 31].”