Communication Quantization for Data-Parallel Training of Deep Neural Networks

By Nikoli Dryden, Tim Moon, Sam Ade Jacobs, Brian Van Essen. 2016

215 citations recorded by the source corpus.

Subfields: Efficiency & Compression, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

In ML Systems & Hardware

Citations in the corpus

Training Neural Networks with Fixed Sparse Masks (2021, background): “An orthogonal approach to reducing communication costs is to have workers only update a small subset of the model’s parameters [2, 13, 44, 45].”
Flower: A Friendly Federated Learning Research Framework (2020, background): “The optimization of distributed training with and without federated concepts has been covered from many angles (Dean et al., 2012; Jia et al., 2018; Chahal et al., 2018; Sergeev & Balso, 2018; Dryden et al., 2016).”
Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization (2019, background): “Gradient vector sparsification is another approach analyzed in [4, 40, 39, 5, 30, 34, 9, 3, 36, 21, 32].”
Natural Compression for Distributed Deep Learning (2019, methodology): “While the focus of this paper was on unbiased quantizations, it is possible to combine our natural quantization mechanisms in conjunction with biased techniques, such as the TopK sparsifier proposed in (Dryden et al., 2016; Aji & Heafield, 2017) and recently analyzed in (Alistarh et al.”
Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learning (2018, methodology): “In Theorem 1 and Corollary 1, we follow the convention in literature (Ghadimi and Lan 2013) (Lian et al. 2017) (Alistarh et al. 2017) to use the (average) expected squared gradient norm to characterize the convergence rate.”
Local SGD Converges Fast and Communicates Little (2018, background): “A very aggressive—and promising—sparsification method is to keep only very few coordinates of the stochastic gradient by considering only the coordinates with the largest magnitudes [3, 9, 18, 32, 35, 36, 37].”
Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training (2017, background): “Therefore, Dryden et al. (2016) chose a ﬁxed proportion of positive and negative gradient updates separately, and Aji & Heaﬁeld (2017) proposed Gradient Dropping to sparsify the gradients by a single threshold based on the absolute value.”
Sparsified SGD with Memory (2018, methodology): “As the usefulness of SGD with sparsification techniques has already been shown in practical applications [1, 9, 21, 33, 37] we focus here on a few particular aspects.”
Sparse Communication for Distributed Gradient Descent (2017, methodology): “This network communication is costly, so prior work developed two ways to approximately compress network trafﬁc: 1-bit quantization (Seide et al., 2014) and sending sparse matrices by dropping small updates (Strom, 2015; Dryden et al., 2016).”
The Convergence of Sparsified Gradient Methods (2018, background): “[8] and Aji and Heafield [2] considered sparsifying the gradient updates by only applying the top K components, taken at at every node, in every iteration, for K corresponding to < 1% of the dimension, and accumulating the error.”
Demystifying Parallel and Distributed Deep Learning (2018, background): “2,128,152] Sharded PS[39,56,120,137,142,244,254,255] Hierarchical PS[88,107,253] Decentralized[9,54,121,155] Compression Quantization[5,35,50,51,55,63,87,90,109,132,151,202,210,237,265] Sparsification[3,32,67,158,206,214,224] Other Methods[41,106,111,129,150,243,254] Training Distribution Model Cons”
Stochastic Distributed Learning with Gradient Quantization and Variance Reduction (2019, background): “Experimentally is has been shown that deterministic masking—for instance selecting the largest components in absolute value (Dryden et al., 2016; Aji & Heaﬁeld, 2017)—can outperform the random techniques.”
Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training (2018, background): “Therefore, many proposed complex compression techniques [43, 46, 11, 4, 50, 33] often fail in the communication-optimal AllReduce setting, resulting in longer wallclock time.”
On the Convergence of Quantized Parallel Restarted SGD for Serverless Learning (2020, unknown): “, quantization [1,23,26,20] and sparsification [5].”
Variance-based Gradient Compression for Efficient Distributed Deep Learning (2018, background): “Dryden et al. (2016) and Aji & Heaﬁeld (2017) proposed sparsifying gradient elements multiple times to utilize a kind of allreduce for sparsiﬁcation-based compressions.”
meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting (2017, methodology): “There are also prior studies focusing on reducing the communication cost in distributed systems (Seide et al., 2014; Dryden et al., 2016), by quantizing each value of the gradient from 32-bit ﬂoat to only 1-bit.”
PopSGD: Decentralized Stochastic Gradient Descent in the Population Model (2019, background): “In particular, there has been signiﬁcant progress on communication-reduced variants of SGD (e.g. [43, 46, 7, 51, 3, 22, 25]), asynchronous variants (e.g. [41, 42, 23, 6]), as well as large-batch or periodic model averaging methods, which aim to reduce the frequency of communication(e.g. [24, 53]…”
SparCML: High-Performance Sparse Communication for Machine Learning (2018, unknown): “Top-k SGD. Recent work proposes the following communication-reduced SGD variant called Top-k [2,13]: each node communicates only the k largest (by magnitude) components of it’s gradient vector ∇ F ( (cid:126)x t ) , instead of all values in the traditional method.”
A Survey on Methods and Theories of Quantized Neural Networks (2018, methodology): “In [ Dryden et al. , 2016 ] , the authors proposed a simple adaptive quantization method to select a proportion of gradients to be quantized and sent.”
Model Pruning Enables Efficient Federated Learning on Edge Devices (2019, background): “, send gradients whose magnitudes are above a certain threshold (Strom, 2015), or send a fixed proportion of gradients to the server (Dryden et al., 2016).”