Zeno: Byzantine-suspicious stochastic gradient descent

By Cong Xie, Oluwasanmi Koyejo, Indranil Gupta. 2018

48 citations recorded by the source corpus.

We propose Zeno, a new robust aggregation rule, for distributed synchronous Stochastic Gradient Descent~(SGD) under a general Byzantine failure model. The key idea is to suspect the workers that are potentially malicious, and use a ranking-based preference mechanism. This allows us to generalize beyond past work--in our case, the number of malicious workers can be arbitrarily large, and we use only the weakest assumption on honest workers~(at least one honest worker). We prove the convergence of SGD under these scenarios. Empirical results show that Zeno outperforms existing approaches under various attacks.

Subfields: Artificial Intelligence, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

Citations in the corpus

How To Backdoor Federated Learning (2018, background): “[5], or even unmodified and equally distributed [13, 69, 71].”
DETOX: A Redundancy-based Framework for Faster and More Robust Gradient Aggregation (2019, background): “Byzantine-tolerant ML has been extensively studied in recent years [13, 14, 15, 16, 17, 2], establishing that while average-based gradient methods are susceptible to adversarial nodes, median-based update methods can in some cases achieve better convergence, while being robust to some attacks.”
Zeno++: Robust Fully Asynchronous SGD (2019, methodology): “n, we introduce Zeno++, a Byzantine-tolerant asynchronous SGD algorithm based on inner-product validation. Zeno++ is an computationally efﬁcient version of its prototype: Zeno+. 3 4.1 Zeno+ Like Zeno [18], we compute a score for each candidate gradient estimator by using the stochastic zero-order or”
RSA: Byzantine-Robust Stochastic Aggregation Methods for Distributed Learning from Heterogeneous Datasets (2018, unknown): “Leveraging additional data, (Xie, Koyejo, and Gupta 2018c) studies the trustworthy score-based schemes that guarantee ef-ﬁcient learning even when there is only one non-Byzantine worker, but additional data may not always be available in practice.”
Zeno++: robust asynchronous SGD with arbitrary number of Byzantine workers (2019, methodology): “[18] propose synchronous SGD algorithms for an unbounded number of Byzantine workers.”
Distributed Gradient Descent Algorithm Robust to an Arbitrary Number of Byzantine Attackers (2019, methodology): “13 and 16, we plot the impact of choosing different cutoff values in Zeno.”
Byzantine-Resilient Stochastic Gradient Descent for Distributed Learning: A Lipschitz-Inspired Coordinate-wise Median Approach (2019, background): “examples that Krum’s performance could be comparable to other methods, which is consistent with the observations in [4], [7].”
SGD: Decentralized Byzantine Resilience (2019, background): “Over the past two years, a growing body of work [10, 5, 14, 42, 7, 18, 40, 41] (and about 30 other papers) encompassed resilience to Poisoning inside a wider project, Byzantine Resilient Machine Learning .”