signSGD with Majority Vote is Communication Efficient And Byzantine Fault Tolerant
By Jiawei Zhao, Kamyar Azizzadenesheli, Anima Anandkumar. 2018
50 citations recorded by the source corpus.
Training neural networks on large datasets can be accelerated by distributing the workload over a network of machines. As datasets grow ever larger, networks of hundreds or thousands of machines become economically viable. The time cost of communicating gradients limits the effectiveness of using such large machine counts, as may the increased chance of network faults. We explore a particularly simple algorithm for robust, communication-efficient learning---signSGD. Workers transmit only the sign of their gradient vector to a server, and the overall update is decided by a majority vote. This algorithm uses $32\times$ less communication per iteration than full-precision, distributed SGD. Under natural conditions verified by experiment, we prove that signSGD converges in the large and mini-batch settings, establishing convergence for a parameter regime of Adam as a byproduct. We model adversaries as those workers who may compute a stochastic gradient estimate and manipulate it, but may not coordinate with other adversaries. Aggregating sign gradients by majority vote means that no individual worker has too much power. We prove that unlike SGD, majority vote is robust when up to 50% of workers behave adversarially. On the practical side, we built our distributed training system in Pytorch. Benchmarking against the state of the art collective communications library (NCCL), our framework---with the parameter server housed entirely on one machine---led to a 25% reduction in time for training resnet50 on Imagenet when using 15 AWS p3.2xlarge machines.
Subfields: Radiology, Nuclear Medicine and Imaging
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Radiology, Nuclear Medicine and Imaging
Citations in the corpus
- $\texttt{DeepSqueeze}$: Decentralization Meets Error-Compensated Compression (2019, methodology): “The theoretical guarantee of this method is given in Bernstein et al. [2018].”
- DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression (2019, methodology): “…are many methods being studied to improve the performance of distributed learning, such as communication efficient learning (Alistarh et al., 2017; Bernstein et al., 2018a; Seide et al., 2014), decentralized learning (He et al., 2018b; Lian et al., 2017), and asynchronous learning (Agarwal &…”
- Robust and Communication-Efficient Federated Learning From Non-i.i.d. Data (2019, methodology): “2) SignSGD: The quantization method signSGD [29] (see green line in Fig.”
- Scaling Distributed Machine Learning with In-Network Aggregation (2019, unknown): “Other work has focused on gradient exchange for SGD, using fixed-point quantization, dithering, or sparsification to reduce both the number of bits and the gradient elements transmitted [7, 8, 60, 69, 88, 95, 99].”
- Privacy and Robustness in Federated Learning: Attacks and Defenses (2020, methodology): “SIGNSGD with majority vote [30], [139] offers a promising solution to realize the first four goals.”
- 1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed (2021, unknown): “The theoretical guarantee of this method is given in Bernstein et al. (2018).”
- Sparse Binary Compression: Towards Distributed Deep Learning with minimal Communication (2018, background): “Motivated by the central limit theorem and empirical investigations by [24] we can make the simplifying assumption (a) that this noise n is normally distributed at every time-step n ∼ N (0, σ(2)) with the variance being constant in time V(n) = σ(2) for all t = 1, .”