signSGD with Majority Vote is Communication Efficient and Fault Tolerant

By Jeremy Bernstein, Jiawei Zhao, Kamyar Azizzadenesheli, Anima Anandkumar. 2018

314 citations recorded by the source corpus.

Subfields: Optimization & Training, Federated & Distributed Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

In Federated & Distributed Learning

Citations in the corpus

SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum (2019, background): “Note that various approaches to gradient compression have been proposed to reduce the communication overhead for ALLREDUCE and decentralized learning methods (Alistarh et al., 2007; Wen et al., 2007; Bernstein et al., 2019; Karimireddy et al., 2019; Koloskova et al., 2019b; Vogels et al., 2019).”
PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization (2019, methodology): “All three methods scale reasonably well with the optimized NCCL backend, although Signum has a slope less than 1 in the log-log plot, indicating sub-linear scaling.”
Robust and Communication-Efficient Federated Learning From Non-i.i.d. Data (2019, methodology): “As it violates assumption (2), a non-iid distribution of the local data renders existing convergence guarantees as formulated in [19][20][29][21] inapplicable and has dramatic effects on the practical performance of communication-efficient distributed training algorithms as we will demonstrate in th”
Stochastic Gradient Push for Distributed Deep Learning (2018, background): “2XLARGE machines commonly used for distributed training (Bernstein et al., 2019).”
Distributed Training with Heterogeneous Data: Bridging Median- and Mean-Based Algorithms (2019, methodology): “The reason of assuming homogeneous data in existing literature [Bernstein et al., 2018a,b, Yin et al., 2018] is exactly to ensure that the median is close to mean.”
Variance Reduced Local SGD with Lower Communication Complexity (2019, methodology): “…bottlenecks, many distributed SGD-based methods are proposed, such as lossy compression methods [Alistarh et al., 2017, Aji and Heaﬁeld, 2017, Bernstein et al., 2019, Lin et al., 2018b, Karimireddy et al., 2019, Tang et al., 2019], which use inexact approximations or partial data to…”
Learning from History for Byzantine Robust Optimization (2020, methodology): “…of the updates from the different machines with more robust aggregation rules such as coordinate-wise median method (Yin et al., 2018), geometric median meth-ods (Blanchard et al., 2017; Chen et al., 2017; Pillutla et al., 2019), majority voting (Bernstein et al., 2018; Jin et al., 2020) etc.”
Uncertainty Principle for Communication Compression in Distributed and Federated Learning and the Search for an Optimal Compressor (2020, unknown): “…(Alistarh et al., 2018), standard random dithering (Goodall, 1951; Roberts, 1962; Alistarh et al., 2017), natural dithering (Horv´ath et al., 2019a), ternary quantization (Wen et al., 2017), and scaled sign quantization (Karimireddy et al., 2019b; Bernstein et al., 2018, 2019; Liu et al., 2019).”
Communication-Efficient Distributed Blockwise Momentum SGD with Error-Feedback (2019, methodology): “They are compared with (i) distributed synchronous SGD (with full-precision gradient); (ii) distributed synchronous SGD (full-precision gradient) with momentum (SGDM); (iii) signSGD with majority vote [3]; and (iv) signum with majority vote [4].”
Threats to Federated Learning: A Survey (2020, background): “Instead, sharing less sensitive information (e.g., SIGNSGD [Bernstein et al. , 2018]) or only sharing model predictions [ Chang et al. , 2019 ] in a black-box manner may result in more robust privacy protection in FL. Threats to VFL: In VFL, there may only be one party who owns labels for the given…”
Byzantine-Robust Learning on Heterogeneous Datasets via Resampling (2020, methodology): “In a third approach, (Bernstein et al., 2018) propose to use the signs of the gradients and then aggregate them by majority vote, however, (Karimireddy et al., 2019) show that it may not always converge.”
Privacy and Robustness in Federated Learning: Attacks and Defenses (2020, background): “SIGNSGD with majority vote [30], [139] offers a promising solution to realize the first four goals.”
Back to the Drawing Board: A Critical Evaluation of Poisoning Attacks on Production Federated Learning (2021, background): “Example AGRs are Median [70], Trimmed-mean [70], and sign-SGD with majority voting [6].”
Election Coding for Distributed Learning: Protecting SignSGD against Byzantine Attacks (2019, methodology): “For communicating 1-bit compressed gradients, a recent work [6] analyzed the Byzantine-tolerance of the naive SIGNSGD-MV scheme.”