1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs

By Frank Torsten Bernd Seide, Hao Fu, Jasha Droppo, Gang Li, Dong Yu. 2014

1,100 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Optimization & Training

Citations in the corpus

Private Counting from Anonymous Messages: Near-Optimal Accuracy with Vanishing Communication Overhead (2020, background): “The vector version of this problem captures, e.g., the case where gradients have been quantized to bits in order to reduce the communication cost (e.g., the 1-bit SGD of Seide et al. (2014)).”
Distributed Fixed Point Methods with Compressed Iterates (2019, methodology): “tion and decentralization [31], as well as exchanging only compressed or quantized information between the computing units [29]. Usually, the exchanged information is usually some compressed gradient [1,7,27] or compressed model update [4,25] in a distributed master-worker setting. We note that in t”
Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates (2019, methodology): “Various approaches have been proposed to reduce the communication overhead of distributed SGD, such as quantization (Seide et al., 2014; Strom, 2015; Wen et al., 2017; Alistarh et al., 2016; Bernstein et al., 2018; Karimireddy et al., 2019; Zheng et al., 2019) and sparsification (Aji and Heafield,…”
Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization (2019, background): “Gradient vector sparsification is another approach analyzed in [4, 40, 39, 5, 30, 34, 9, 3, 36, 21, 32].”
Communication Efficient Decentralized Training with Multiple Local Updates (2019, background): “The more direct approach is to reduce the size of the messages through gradient compression or sparsiﬁcation [32, 23, 51, 40, 42, 8].”
FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization (2019, background): “…in distributed learning has recently gained considerable attention for both master-worker [Alistarh et al., 2017, Bernstein et al., 2018, Seide et al., 2014, Smith et al., 2016] and masterless topologies [Koloskova et al., 2019, Reisizadeh et al., 2019a, Wang et al., 2019, Zhang et al.,…”
Robust and Communication-Efficient Collaborative Learning (2019, methodology): “In particular, efﬁcient gradient-compression methods are provided in (Alistarh et al., 2017; Seide et al., 2014; Bernstein et al., 2018) and deployed in the distributed master-worker setting.”
Decentralized Deep Learning with Arbitrary Communication Compression (2019, methodology): “Communication compression with quantization has been popularized in the deep learning community by the reported successes in (Seide et al., 2014; Strom, 2015).”
$\texttt{DeepSqueeze}$: Decentralization Meets Error-Compensated Compression (2019, background): “One example is the 1-Bit SGD [Seide et al., 2014], which compresses the entries in gradient vector into ±1 depends on its sign.”
Qsparse-Local-SGD: Distributed SGD With Quantization, Sparsification, and Local Computations (2019, background): “ent methods has decades rich history [26] and its recent use in training deep neural networks [16], [27] has re-ignited interest.”
Natural Compression for Distributed Deep Learning (2019, background): “It is well-known (Seide et al., 2014; Alistarh et al., 2017; Zhang et al., 2017; Lin et al., 2018; Lim et al., 2018) that in many practical applications with common computing architectures, communication takes much more time than computation, creating a bottleneck in the entire training system.”
DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression (2019, methodology): “…studied to improve the performance of distributed learning, such as communication efﬁcient learning (Alistarh et al., 2017; Bernstein et al., 2018a; Seide et al., 2014), de-centralized learning (He et al., 2018b; Lian et al., 2017b), and asynchronous learning (Agarwal & Duchi, 2011; Lian et al.,…”
On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization (2019, methodology): “To reduce the communication cost at each iteration, compression or sparsification based parallel SGD are considered in (Seide et al., 2014; Strom, 2015; Aji & Heafield, 2017; Wen et al., 2017; Alistarh et al., 2017).”
PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization (2019, methodology): “Notable examples include replacing the coordinates of the gradient with only their sign (Seide et al., 2014; Carlson et al., 2015; Bernstein et al., 2018, 2019; Karimireddy et al., 2019), quantizing the individual coordinates (Alistarh et al., 2017; Wen et al., 2017), and low-rank approximation of t”
Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication (2019, background): “Techniques that do not directly quantize gradients, but instead maintain additional states are known to perform better in theory and practice (Seide et al., 2014; Lin et al., 2018; Stich et al., 2018), an approach that we pick up here.”
Error Feedback Fixes SignSGD and other Gradient Compression Schemes (2019, methodology): “Of these, Seide et al. [2014], Strom [2015], Wen et al. [2017] are all sign-based compression schemes.”
Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learning (2018, background): “For example, quantized SGD studied in (Seide et al. 2014)(Alistarh et al.”
cpSGD: Communication-efficient and differentially-private distributed SGD (2018, background): “To address this issue, several recent works have focused on reducing the communication cost of distributed learning algorithms via gradient quantization and sparsification [32, 17, 33, 20, 21, 4, 34].”
Local SGD Converges Fast and Communicates Little (2018, background): “A very aggressive—and promising—sparsification method is to keep only very few coordinates of the stochastic gradient by considering only the coordinates with the largest magnitudes [3, 9, 18, 32, 35, 36, 37].”