Distributed Gradient Descent Algorithm Robust to an Arbitrary Number of Byzantine Attackers
By Xinyang Cao, Lifeng Lai. 2019
94 citations recorded by the source corpus.
Due to the growth of modern dataset size and the desire to harness computing power of multiple machines, there is a recent surge of interest in the design of distributed machine learning algorithms. However, distributed algorithms are sensitive to Byzantine attackers who can send falsified data to prevent the convergence of algorithms or lead the algorithms to converge to value of the attackers’ choice. Some recent work proposed interesting algorithms that can deal with the scenario when up to half of the workers are compromised. In this paper, we propose a novel algorithm that can deal with an arbitrary number of Byzantine attackers. The main idea is to ask the parameter server to randomly select a small clean dataset and compute noisy gradient using this small dataset. This noisy gradient will then be used as a ground truth to filter out information sent by compromised workers. We show that the proposed algorithm converges to the neighborhood of the population minimizer regardless the number of Byzantine attackers. We further provide numerical examples to show that the proposed algorithm can benefit from the presence of good workers and achieve better performance than existing algorithms.
Subfields: Artificial Intelligence, Electrical and Electronic Engineering
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sham M. Kakade
- Zaïd Harchaoui
- Krishna Pillutla
- Minghong Fang
- Haibo Yang
- Xin Zhang
- Jia Liu
- Francis R. Bach
In Electrical and Electronic Engineering
Citations in the corpus
- Robust Aggregation for Federated Learning (2019, background): “Byzantine robustness, resilience to arbitrary behavior of some devices [51], was studied in distributed optimization with gradient aggregation [5, 16, 20, 21, 23, 88].”
- Byzantine-Resilient Stochastic Gradient Descent for Distributed Learning: A Lipschitz-Inspired Coordinate-wise Median Approach (2019, background): “In [11], the authors proposed to let the parameter server keep a small set of data to compute an estimate of the true gradient, which is used as a benchmark to filter out suspicious gradients.”