3LC: Lightweight and Effective Traffic Compression for Distributed Machine Learning

By Hyeontaek Lim, D. Andersen, Michael Kaminsky. 2018

82 citations recorded by the source corpus.

The performance and efficiency of distributed machine learning (ML) depends significantly on how long it takes for nodes to exchange state changes. Overly-aggressive attempts to reduce communication often sacrifice final model accuracy and necessitate additional ML techniques to compensate for this loss, limiting their generality. Some attempts to reduce communication incur high computation overhead, which makes their performance benefits visible only over slow networks. We present 3LC, a lossy compression scheme for state change traffic that strikes balance between multiple goals: traffic reduction, accuracy, computation overhead, and generality. It combines three new techniques---3-value quantization with sparsity multiplication, quartic encoding, and zero-run encoding---to leverage strengths of quantization and sparsification techniques and avoid their drawbacks. It achieves a data compression ratio of up to 39--107X, almost the same test accuracy of trained models, and high compression speed. Distributed ML frameworks can employ 3LC without modifications to existing ML algorithms. Our experiments show that 3LC reduces wall-clock training time of ResNet-110--based image classifiers for CIFAR-10 on a 10-GPU cluster by up to 16--23X compared to TensorFlow's baseline design.

Subfields: Artificial Intelligence, Computer Networks and Communications, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Computer Vision and Pattern Recognition

Citations in the corpus

Natural Compression for Distributed Deep Learning (2019, background): “It is well-known (Seide et al., 2014; Alistarh et al., 2017; Zhang et al., 2017; Lin et al., 2018; Lim et al., 2018) that in many practical applications with common computing architectures, communication takes much more time than computation, creating a bottleneck in the entire training system.”
Software-hardware co-design for fast and scalable training of deep learning recommendation models (2021, background): “Alternatively, these papers [33, 34] address the communication overheads by using various quantization schemes to reduce communication volume.”
Gradient Compression Supercharged High-Performance Data Parallel DNN Training (2021, background): “For instance, AdaComp [12] needs map, reduce, filter, concat and extract common operators, while 3LC [36] needs reduce, map, concat, fil-”
High-performance, Distributed Training of Large-scale Deep Learning Recommendation Models (2021, background): “Alternatively, these papers [30, 31] address the communication overheads by using various quantization schemes to reduce communication volume.”