Interprocessor collective communication library (InterCom)

By Mike Barnett, Satya Gupta, G. David, PayneDepartment, N. W. Greenbrier, PkwyBeaverton, L. Shuler, R. V. D. Geijn, Jerrell, WattsDepartment. 1994

Subfields: ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In ML Systems & Hardware

Citations in the corpus

Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour (2017, methodology): “For interserver allreduce, we implemented two of the best algorithms for bandwidth-limited scenarios: the recursive halving and doubling algorithm [29, 36] and the bucket algorithm (also known as the ring algorithm) [2].”
Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes (2018, background): “To reduce the communication overhead when the number of nodes increases in distributed training, Baidu [9] introduces the ring-based all-reduce algorithm [4] to deep learning.”
Optimal bucket algorithms for large MPI collectives on torus interconnects (2010, methodology): “We now describe the bucket algorithm [2, 3, 10] for the Reduce-scatter collective.”
Optimized Broadcast for Deep Learning Workloads on Dense-GPU InfiniBand Clusters: MPI or NCCL? (2017, methodology): “y non-intuitive yet very powerful when scalability with respect to the message size ‘M’ is of higher importance than scalability with respect to the number of nodes ‘n’. The scatter-allgather schemes [7], [30] to realize a broadcast have been proposed to improve the performance for large-message bro”
Optimization of Collective Reduction Operations (2004, background): “Although principal work on optimizing collective routines is quite old [2], there is a lack of fast implementations for allreduce and reduce in MPI libraries for a wide range of number of processes and buffer sizes.”
Collective communication: theory, practice, and experience (2007, background): “Further evidence, from the Intel Touchstone Delta and Paragon systems, can be found in [ 20–22 ] .”
Optimization of Collective Communication Operations in MPICH (2005, methodology): “This model is similar to the one used by Van de Geijn [2] [16] [24], Hockney [11], and others.”
Improving the Performance of Collective Operations in MPICH (2003, methodology): “Some efforts have focused on using different algorithms for different message sizes, such as the work by Van de Geijn et al. for the Intel Paragon [1] [9] [14], by Rabenseifner on reduce and allreduce [11], and by Kale et al. on all-to-all communication [6].”
Parallel Spectral Clustering in Distributed Systems (2011, background): “Note that one often assumes the following cost model for transferring some data between two nodes [3]:”
Fast parallel algorithms for short-range molecular dynamics (1993, methodology): “Various algorithms have been developed for performing this operation eﬃciently on diﬀerent parallel machines and architectures [7, 22, 54].”