Phocas: dimensional Byzantine-resilient stochastic gradient descent

By Cong Xie, Oluwasanmi Koyejo, Indranil Gupta. 2018

62 citations recorded by the source corpus.

We propose a novel robust aggregation rule for distributed synchronous Stochastic Gradient Descent~(SGD) under a general Byzantine failure model. The attackers can arbitrarily manipulate the data transferred between the servers and the workers in the parameter server~(PS) architecture. We prove the Byzantine resilience of the proposed aggregation rules. Empirical analysis shows that the proposed techniques outperform current approaches for realistic use cases and Byzantine attack scenarios.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Zeno++: Robust Fully Asynchronous SGD (2019, background): “Chen et al. (2017); Su & Vaidya (2016; 2018); Yin et al. (2018); Xie et al. (2018) use robust statistics (Huber, 2004) Recent years have witnessed an increasing number of large-scale machine learning algorithms, including asynchronous SGD (Zinkevich et al., 2009; Lian et al., 2018; Zheng et al.,…”
Practical Distributed Learning: Secure Machine Learning with Communication-Efficient Local Updates (2019, methodology): “Approaches based on robust statistics are often used to address security issues in the PS architecture (Yin et al., 2018; Xie et al., 2018).”
Distributed Gradient Descent Algorithm Robust to an Arbitrary Number of Byzantine Attackers (2019, methodology): “[23] compare the performance of algorithms that use geometric median, marginal median and mean median to update parameters.”
SLSGD: Secure and Efficient Distributed On-device Machine Learning (2019, methodology): “Approaches based on robust statistics are often used to address security issues in the PS architecture [30,27].”
Zeno: Distributed Stochastic Gradient Descent with Suspicion-based Fault-tolerance (2018, background): “However, when the magnitude is enlarged, Mean will fail, as pointed out in Xie et al. (2018).”
Robust Federated Recommendation System (2020, methodology): “The current defense methods employ the existing deﬁnitions of Byzantine resilience [7; 23; 39].”
Byzantine-Resilient Stochastic Gradient Descent for Distributed Learning: A Lipschitz-Inspired Coordinate-wise Median Approach (2019, background): “, [2]–[8]) have two main limitations: i) requiring the knowledge of the number of malicious workers, which is often infeasible in practice; and ii) suffering from high computational complexity in stochastic subgradient screening and aggregation mechanisms (see Section II for detailed discussions).”
RSA: Byzantine-Robust Stochastic Aggregation Methods for Distributed Learning from Heterogeneous Datasets (2018, unknown): “Leveraging additional data, (Xie, Koyejo, and Gupta 2018c) studies the trustworthy score-based schemes that guarantee ef-ﬁcient learning even when there is only one non-Byzantine worker, but additional data may not always be available in practice.”
Secure Byzantine-Robust Machine Learning (2020, background): “Further, [37] extends the Byzantine setting to attackers manipulating data transfer between workers and server and [38] extends it to tolerate an arbitrary number of Byzantine workers.”