DRACO: Byzantine-resilient Distributed Training via Redundant Gradients

By Lingjiao Chen, Hongyi Wang, Zachary B. Charles, Dimitris Papailiopoulos. 2018

274 citations recorded by the source corpus.

Distributed model training is vulnerable to byzantine system failures and adversarial compute nodes, i.e., nodes that use malicious updates to corrupt the global model stored at a parameter server (PS). To guarantee some form of robustness, recent work suggests using variants of the geometric median as an aggregation rule, in place of gradient averaging. Unfortunately, median-based rules can incur a prohibitive computational overhead in large-scale settings, and their convergence guarantees often require strong assumptions. In this work, we present DRACO, a scalable framework for robust distributed training that uses ideas from coding theory. In DRACO, each compute node evaluates redundant gradients that are used by the parameter server to eliminate the effects of adversarial updates. DRACO comes with problem-independent robustness guarantees, and the model that it trains is identical to the one trained in the adversary-free setup. We provide extensive experiments on real datasets and distributed setups across a variety of large-scale models, where we show that DRACO is several times, to orders of magnitude faster than median-based approaches.

Subfields: Artificial Intelligence, Federated & Distributed Learning, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Federated & Distributed Learning

In ML Systems & Hardware

Citations in the corpus

Attack of the Tails: Yes, You Really Can Backdoor Federated Learning (2020, methodology): “Draco [72] is provides problem-independent robustness guarantees while being significantly faster than median-based approaches using elements from coding theory.”
Robust Aggregation for Federated Learning (2019, background): “Byzantine robustness, resilience to arbitrary behavior of some devices [51], was studied in distributed optimization with gradient aggregation [5, 16, 20, 21, 23, 88].”
DETOX: A Redundancy-based Framework for Faster and More Robust Gradient Aggregation (2019, background): “To the best of our knowledge, D RACO [7] and B ULYAN [10] are the only proposed methods that guarantee strong Byzantine resilience.”
Data Encoding for Byzantine-Resilient Distributed Optimization (2019, background): “In this section, we do a detailed comparison with [11] and [29], which are the closest related works that also combat”
A Little Is Enough: Circumventing Defenses For Distributed Learning (2019, background): “While all other defenses take a statistics-based approach, the authors of DRACO [7] suggest a coding-based defense.”
Analyzing Federated Learning through an Adversarial Lens (2018, background): “…also has a connection to a line of work on defending against Byzantine adversaries which consider a threat model where the malicious agents can send arbitrary gradient updates (Blanchard et al., 2017; Chen et al., 2017b; Mhamdi et al., 2018; Chen et al., 2018; Yin et al., 2018) to the server.”
Fall of Empires: Breaking Byzantine-tolerant SGD by Inner Product Manipulation (2019, methodology): “DRACO (Chen et al., 2018) uses coding theory to ensure robustness, and is different from the other Byzantine-tolerant SGD algorithms.”
Byzantine-Robust Federated Machine Learning through Adaptive Model Averaging (2019, methodology): “Other approaches such as DRACO (Chen et al. 2018) require to add some redundancy to achieve robustness and the data provided by the clients may require leaving their facilities to build client nodes with partially overlapping datasets.”
Learning from History for Byzantine Robust Optimization (2020, background): “within a datacenter) coding theory based solutions which can correct for the mistakes have also been proposed (Chen et al., 2018; Rajput et al., 2019; Gupta & Vaidya, 2019; Konstantinidis & Ramamoorthy, 2020; Data et al., 2018; 2019).”
Zeno++: robust asynchronous SGD with arbitrary number of Byzantine workers (2019, background): “[4] use coding theory and majority voting to recover correct gradients.”
Data Encoding for Byzantine-Resilient Distributed Gradient Descent (2018, methodology): “Our method gives deterministic guarantees, distinct from these works, but similar in spirit to [8], which is the closest related work.”
Distributed Gradient Descent Algorithm Robust to an Arbitrary Number of Byzantine Attackers (2019, methodology): “[25] propose DRACO algorithm that uses ideas from coding theory to determine which machines are under attack.”
Privacy preservation in federated learning: An insightful survey from the GDPR perspective (2020, background): “erverisunabletoinspectthedata oradministerthetrainingprocessatalocalnode.This,however,prohibitsthetransparencyofthetrainingprocess;thus, imposesanewvulnerabilityofanewtypeofattackcalled modelpoisoning[12,87,22,41,9,6].Generally,modelpoisoningattacksaimatmanipulatingtrainingprocessbyfeedingpoisonedlo”
Data Encoding Methods for Byzantine-Resilient Distributed Optimization (2019, methodology): “In contrast, the storage redundancy factor required in [7] is 2 t + 1 , growing linearly with the number of corrupt worker nodes.”
Election Coding for Distributed Learning: Protecting SignSGD against Byzantine Attacks (2019, methodology): “Compared to the codes in [9], our codes have the following two advantages.”