Collective communication: theory, practice, and experience

By E. Chan, M. Heimlich, A. Purkayastha, R. V. D. Geijn. 2007

314 citations recorded by the source corpus.

Subfields: Computer Networks and Communications, Hardware and Architecture, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Hardware and Architecture

In ML Systems & Hardware

Citations in the corpus

Efficiently Scaling Transformer Inference (2022, neutral): “A.1 Cost of all-gather/reduce-scatter This is a general cost model that holds true for most real-world network topologies (Chan et al., 2007), not just the TPU’s torus topology.”
Demystifying Parallel and Distributed Deep Learning (2018, neutral): “d is T r ≥γmG, where γrepresents the size of a single data value andm is the number of values sent. This bound can be strengthened to T r ≥Llog 2 (P)+2γmG(P −1)/P if we disallow redundant computations[29]. Several practical algorithms exist for the parallel allreduce operation in different environme”
Integrated Model and Data Parallelism in Training Neural Networks (2017, background): “For the communication complexity we assume that all the compute nodes are connected and thus do not consider the topology of the interconnect, and we also do not consider network conflicts in our model [3].”
A Unified Coded Deep Neural Network Training Strategy based on Generalized PolyDot codes (2018, methodology): “We will use Broadcast and All-Gather protocols to prove our results on communication complexities in Appendix D.”
Integrated Model, Batch, and Domain Parallelism in Training Neural Networks (2017, background): “For the communication complexity we assume that all the compute nodes are connected and thus do not consider the topology of the interconnect, and we also do not consider network conflicts in our model [3].”
SparCML: High-Performance Sparse Communication for Machine Learning (2018, unknown): “Performing an allgather operation, where each node contributes exactly k elements, results in a performance lower bound in this cost model of log 2 ( P ) α + ( P − 1) kβ d [8].”
HybridFlow: A Flexible and Efficient RLHF Framework (2024, unknown): “The communication volumes for these operations are 𝑀 for DeepSpeed-Chat and 𝑡𝑝 − 1 𝑀 for HybridFlow-V, calculated following [13].”
Communication-Avoiding Parallel Sparse-Dense Matrix-Matrix Multiplication (2016, background): “Figure 1 shows the computational cuboid, an example voxel subset V , and its projections onto the A , B , and C planes.”