AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Training

By Chia-Yu Chen, Jungwook Choi, Daniel Brand, Ankur Agrawal, Kailash Gopalakrishnan. 2017

188 citations recorded by the source corpus.

Highly distributed training of Deep Neural Networks (DNNs) on future compute platforms (offering 100 of TeraOps/s of computational capacity) is expected to be severely communication constrained. To overcome this limitation, new gradient compression techniques are needed that are computationally friendly, applicable to a wide variety of layers seen in Deep Neural Networks and adaptable to variations in network architectures as well as their hyper-parameters. In this paper we introduce a novel technique - the Adaptive Residual Gradient Compression (AdaComp) scheme. AdaComp is based on localized selection of gradient residues and automatically tunes the compression rate depending on local activity. We show excellent results on a wide spectrum of state of the art Deep Learning models in multiple domains (vision, speech, language), datasets (MNIST, CIFAR10, ImageNet, BN50, Shakespeare), optimizers (SGD with momentum, Adam) and network parameters (number of learners, minibatch-size etc.). Exploiting both sparsity and quantization, we demonstrate end-to-end compression rates of ∼200× for fully-connected and recurrent layers, and ∼40× for convolutional layers, without any noticeable degradation in model accuracies.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training (2017, background): “Concurrently, Chen et al. (2017) proposed to automatically tunes the compression rate depending on local gradient activity, and gained compression ratio around 200 × for fully-connected layers and 40 × for convolutional layers with negligible degradation of top-1 accuracy on ImageNet dataset.”
Sparsified SGD with Memory (2018, background): “Recent work proposes adaptive tuning of the compression ratio [7].”
ATOMO: Communication-efficient Learning via Atomic Sparsification (2018, background): “Other approaches for low-communication training focus on sparsification of gradients, either by thresholding small entries or by random sampling [6, 22, 23, 24, 25, 26, 27, 28].”
Demystifying Parallel and Distributed Deep Learning (2018, background): “, top 1%) [3, 32, 222] and adaptive thresholds [69] to transmit only the “important” gradients, based on their absolute value.”
Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training (2018, background): “Therefore, many proposed complex compression techniques [43, 46, 11, 4, 50, 33] often fail in the communication-optimal AllReduce setting, resulting in longer wallclock time.”
Communication-Efficient Edge AI: Algorithms and Systems (2020, methodology): “…quantization to limited feedback in wireless communication [23] • Communicating only important gradients for minimizing communication bandwidth [47], [38] Second-Order Methods • Stochastic quasi-Newton methods [48], [49], [50] • Approximate Newton-type methods [51], [52], [53], [54] Federated…”
Gradient Compression Supercharged High-Performance Data Parallel DNN Training (2021, background): “For instance, AdaComp [12] needs map , reduce , filter , concat and extract common operators, while 3LC [36] needs reduce , map , concat , filter and extract .”
Hyper-Sphere Quantization: Communication-Efficient SGD for Federated Learning (2019, background): “…have been proposed to reduce the cost of gradient communication (Al-istarh et al., 2017; Bernstein et al., 2018; Wen et al., 2017; Yu et al., 2018; Chen et al., 2018; Wang et al., 2019; Stich et al., 2018; Suresh et al., 2017; Sattler et al., 2019; Tang et al., 2018; 2019; Assran et al., 2019;…”
A Hitchhiker's Guide On Distributed Training of Deep Neural Networks (2018, methodology): “AdaComp [53] exploits local gradient activity by dividing the residual vector of every layer into several ﬁxed size bins using the maximum absolute value in each bin as a threshold to ﬁnd salient gradients.”
A Linear Speedup Analysis of Distributed Deep Learning with Sparse and Quantized Communication (2018, background): “Though numerous variants of gradient quantization and/or sparsiﬁcation have been proposed and successfully applied to different deep learning tasks [42, 45, 37, 48, 18, 7, 16], their impact to the convergence rate of distributed SGD (especially for non-convex optimization) is still unclear.”
A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters (2020, methodology): “Prior work on top of allreduce and PS, like tensor compression [21, 45], can directly apply to BytePS.”
ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction (2021, methodology): “, 2021b,a), and distributed deep learning (Chen et al., 2018; Liu et al., 2020).”