DETOX: A Redundancy-based Framework for Faster and More Robust Gradient Aggregation

By Shashank Rajput, Hongyi Wang, Zachary B. Charles, Dimitris Papailiopoulos. 2019

149 citations recorded by the source corpus.

To improve the resilience of distributed training to worst-case, or Byzantine node failures, several recent approaches have replaced gradient averaging with robust aggregation methods. Such techniques can have high computational costs, often quadratic in the number of compute nodes, and only have limited robustness guarantees. Other methods have instead used redundancy to guarantee robustness, but can only tolerate limited number of Byzantine failures. In this work, we present DETOX, a Byzantine-resilient distributed training framework that combines algorithmic redundancy with robust aggregation. DETOX operates in two steps, a filtering step that uses limited redundancy to significantly reduce the effect of Byzantine nodes, and a hierarchical aggregation step that can be used in tandem with any state-of-the-art robust aggregation method. We show theoretically that this leads to a substantial increase in robustness, and has a per iteration runtime that can be nearly linear in the number of compute nodes. We provide extensive experiments over real distributed setups across a variety of large-scale machine learning tasks, showing that DETOX leads to orders of magnitude accuracy and speedup improvements over many state-of-the-art Byzantine-resilient approaches.

Subfields: Artificial Intelligence, Optimization & Training, Federated & Distributed Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

In Federated & Distributed Learning

Citations in the corpus

Attack of the Tails: Yes, You Really Can Backdoor Federated Learning (2020, background): “Detox [74] combines ideas from robust estimation and coding theory to trade-off between performance and robustness.”
Salvaging Federated Learning by Local Adaptation (2020, background): “To limit the inﬂu-ence of individual participants, several robust, “Byzantine-tolerant” aggregation schemes have been proposed (Blan-chard et al., 2017; El Mhamdi et al., 2018; Damaskinos et al., 2019; Rajput et al., 2019; Chen et al., 2017).”
Data Encoding for Byzantine-Resilient Distributed Optimization (2019, neutral): “Note that [26] combines both redundant computations and do a hierarchical robust aggregation and not is directly comparable to ours.”
Election Coding for Distributed Learning: Protecting SignSGD against Byzantine Attacks (2019, background): “A recent work [23] suggested a framework DETOX which combines two existing schemes: computing redundant gradients and robust aggregation methods.”
Federated Variance-Reduced Stochastic Gradient Descent With Robustness to Byzantine Attacks (2019, background): “Related works also include adversarial learning in distributed principal component analysis [17], escaping from saddle points in non-convex distributed learning under Byzantine attacks [18], and leveraging redundant gradients to improve robustness [19], [20].”
Byzantine-Resilient High-Dimensional SGD with Local Iterations on Heterogeneous Data (2020, unknown): “, 2018; 2019) or by explicitly introducing redundancy in the system via coding-theoretic solutions (Chen et al., 2018; Data et al., 2021; Rajput et al., 2019); however, these approaches fall short of in the FL setting.”
Privacy-Enhanced Federated Learning Against Poisoning Adversaries (2021, methodology): “2) Variations in gradient distributions such as Detox [20].”
Byzantine-Robust Learning on Heterogeneous Datasets via Resampling (2020, background): “If we assume that the server has the entire training dataset and can control the distribution of samples to good workers, Xie et al. (2018b); Chen et al. (2018); Rajput et al. (2019) show that non-iid-ness can be overcome.”
Learning from History for Byzantine Robust Optimization (2020, background): “…such settings where we have full control over the workers (e.g. within a datacenter) coding theory based solutions which can correct for the mistakes have also been proposed (Chen et al., 2018; Rajput et al., 2019; Gupta & Vaidya, 2019; Konstantinidis & Ramamoorthy, 2020; Data et al., 2018; 2019).”
FLTrust: Byzantine-robust Federated Learning via Trust Bootstrapping (2020, unknown): “, [9], [12], [29], [34], [44], [47], [48], [31]) that aim to tolerate Byzantine client failures.”