Efficient sparse collective communication and its application to accelerate distributed deep learning

By Jiawei Fei, Chen-Yu Ho, Atal Narayan Sahu, Marco Canini, Amedeo Sapio. 2020

142 citations recorded by the source corpus.

Efficient collective communication is crucial to parallel-computing applications such as distributed training of large-scale recommendation systems and natural language processing models. Existing collective communication libraries focus on optimizing operations for dense inputs, resulting in transmissions of many zeros when inputs are sparse. This counters current trends that see increasing data sparsity in large models. We propose OmniReduce, an efficient streaming aggregation system that exploits sparsity to maximize effective bandwidth use by sending only non-zero data blocks. We demonstrate that this idea is beneficial and accelerates distributed training by up to 8.2x. Even at 100 Gbps, OmniReduce delivers 1.4--2.9x better performance for network-bottlenecked DNNs.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

Citations in the corpus

Natural Compression for Distributed Deep Learning (2019, methodology): “To alleviate this, we exploit recently proposed OmniReduce (Fei et al., 2020).”
Scaling Distributed Machine Learning with In-Network Aggregation (2019, unknown): “For example, we are currently experimenting with new floating-point representations and protocols for sparse vector aggregations [27].”