RSA: Byzantine-Robust Stochastic Aggregation Methods for Distributed Learning from Heterogeneous Datasets

By Xiang Wang, Dingxian Wang, Canran Xu, Xiangnan He, Yixin Cao, Tat-seng Chua. 2018

664 citations recorded by the source corpus.

In this paper, we propose a class of robust stochastic subgradient methods for distributed learning from heterogeneous datasets at presence of an unknown number of Byzantine workers. The Byzantine workers, during the learning process, may send arbitrary incorrect messages to the master due to data corruptions, communication failures or malicious attacks, and consequently bias the learned model. The key to the proposed methods is a regularization term incorporated with the objective function so as to robustify the learning task and mitigate the negative effects of Byzantine attacks. The resultant subgradient-based algorithms are termed Byzantine-Robust Stochastic Aggregation methods, justifying our acronym RSA used henceforth. In contrast to most of the existing algorithms, RSA does not rely on the assumption that the data are independent and identically distributed (i.i.d.) on the workers, and hence fits for a wider class of applications. Theoretically, we show that: i) RSA converges to a near-optimal solution with the learning error dependent on the number of Byzantine workers; ii) the convergence rate of RSA under Byzantine attacks is the same as that of the stochastic gradient descent method, which is free of Byzantine attacks. Numerically, experiments on real dataset corroborate the competitive performance of RSA and a complexity reduction compared to the state-of-the-art alternatives.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Robust Aggregation for Federated Learning (2019, background): “We note that [55] also aggregate model parameters rather than gradients by framing the problem in terms of consensus optimization.”
Data Encoding for Byzantine-Resilient Distributed Optimization (2019, neutral): “, stochastic gradient descent) [10], [21], [23], [25]–[27], [30]–[32] and also with deterministic methods (e.”
Ditto: Fair and Robust Federated Learning Through Personalization (2020, background): “In terms of defenses, robust aggregation is a common strategy to mitigate the effect of malicious updates (Blanchard et al., 2017; Pillutla et al., 2019; Sun et al., 2019; Li et al., 2019; He et al., 2020).”
Trustworthy AI: From Principles to Practices (2021, background): “Some learning paradigms other than conventional models are also considered to be explainable, such as causal inference [197, 259] and the knowledge graph [345].”
Back to the Drawing Board: A Critical Evaluation of Poisoning Attacks on Production Federated Learning (2021, background): “In particular, (1) SignSGD + majority voting, Krum, Bulyan, RSA, and certiﬁed defenses incur signiﬁcant performance losses, (2) certiﬁed defenses incur high memory cost to clients, (3) knowledge transfer based defenses require public data and incur high performance losses in cross-device, non-iid FL”
Secure Byzantine-Robust Machine Learning (2020, methodology): “d dataset have appeared only recently [21, 13].”
An Efficient Framework for Clustered Federated Learning (2020, methodology): “personal devices are usually non-i.i.d. Several formulations and solutions have been proposed to tackle this problem. A line of research focuses on learning a single global model from non-i.i.d. data [48, 33, 20, 36, 23, 29]. Other lines of research focus more on learning personalized models [37, 35”
Multi-center federated learning: clients clustering for better personalization (2020, methodology): “[12] proposed a robust FL comprising three steps: 1) learning a local model on each device, 2) clustering model parameters to multiple groups, each being a homogeneous dataset, and 3) running a robust distributed optimization [50] in each cluster.”
Learning from History for Byzantine Robust Optimization (2020, methodology): “Numerous variations have been proposed which can handle non-iid data with varying degrees of success (Li et al., 2019; Ghosh et al., 2019; Chen et al., 2019; Peng et al., 2020; Data & Diggavi, 2020; He et al., 2020a; El-Mhamdi et al., 2020; Dong et al., 2020).”
Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses (2020, methodology): “Another Byzantine-robust method called RSA (Li et al. 2019) penalizes parameter updates which move far away from the previous parameter vector and provides theoretical robustness guarantees.”
From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2022, background): “, Abstract Policy Graph [247], Knowledge graph [270], Flow graph [213] and Finite State Automata [103].”
Disentangled Graph Collaborative Filtering (2020, background): “history with users [21], item knowledge [15, 32, 39, 41, 42], and user reviews [5, 22], or conduct psychology experiments [24] to establish the ground truth on user intents and further interpret disentangled representations better.”
FLTrust: Byzantine-robust Federated Learning via Trust Bootstrapping (2020, background): “[26] introduced a norm regularization term into the loss function.”
Mitigating Byzantine Attacks in Federated Learning (2020, background): “Particularly, when the incorporated regularization term is based on ℓ 1 -norm penalty, [12] proposes the following set of updates: where δ is a regularization parameter, and sign ( ) is the coordinate-wise standard signum function.”