Securing Distributed Machine Learning in High Dimensions

By Lili Su, Jiaming Xu. 2018

37 citations recorded by the source corpus.

We consider securing a distributed machine learning system wherein the data is kept confidential by its providers who are recruited as workers to help the learner to train a $d$--dimensional model. In each communication round, up to $q$ out of the $m$ workers suffer Byzantine faults; faulty workers are assumed to have complete knowledge of the system and can collude to behave arbitrarily adversarially against the learner. We assume that each worker keeps a local sample of size $n$. (Thus, the total number of data points is $N=nm$.) Of particular interest is the high-dimensional regime $d \gg n$. We propose a secured variant of the classical gradient descent method which can tolerate up to a constant fraction of Byzantine workers. We show that the estimation error of the iterates converges to an estimation error $O(\sqrt{q/N} + \sqrt{d/N})$ in $O(\log N)$ rounds. The core of our method is a robust gradient aggregator based on the iterative filtering algorithm proposed by Steinhardt et al. \cite{Steinhardt18} for robust mean estimation. We establish a uniform concentration of the sample covariance matrix of gradients, and show that the aggregated gradient, as a function of model parameter, converges uniformly to the true gradient function. As a by-product, we develop a new concentration inequality for sample covariance matrices of sub-exponential distributions, which might be of independent interest.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Qiang Yang
2. Lichao Sun
3. Philip S. Yu
4. Lingjuan Lyu
5. Han Yu
6. Yang Liu
7. Bingsheng He
8. Qinbin Li

In Electrical and Electronic Engineering

1. Qing Ling
2. Lingjuan Lyu
3. Lichao Sun
4. Jun Zhao
5. Bingsheng He
6. Tianyi Chen
7. Philip S. Yu
8. Kannan Ramchandran

Citations in the corpus

• Federated Machine Learning (2019, background): “In [62], authors expose a potential security issue associated with gradient exchanges between different parties, and propose a secured variant of the gradient descent method and show that it tolerates up to a constant fraction of Byzantine workers.”
• Defending Against Saddle Point Attack in Byzantine-Robust Distributed Learning (2018, methodology): “For convex losses, the aggregation subroutines analyzed in the literature include geometric median [26, 17], median and trimmed mean [69], iterative filtering for the high dimensional setting [63], and martingale-based methods for the SGD setting [3].”
• Threats to Federated Learning: A Survey (2020, background): “Exchanging gradients during FL training can result in serious privacy leakage [Phong et al. , 2018; Su and Xu, 2018; Melis et al. , 2019; Zhu et al. , 2019].”
• A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection (2019, background): “fault [19], [25], [161] is also a common issue in distributed learning, where the parties may behave arbitrarily badly and upload random updates.”
• Distributed Gradient Descent Algorithm Robust to an Arbitrary Number of Byzantine Attackers (2019, methodology): “[27] propose an approximate gradient descent algorithm that employs iterative filtering for robust gradient aggregation in high dimensional estimation.”
• RSA: Byzantine-Robust Stochastic Aggregation Methods for Distributed Learning from Heterogeneous Datasets (2018, unknown): “On the other hand, the existing gradient aggregation meth-ods generally require to design nontrivial subroutines to aggregate gradients, and hence incur relatively high complexities (Xie, Koyejo, and Gupta 2018a; Blanchard et al. 2017; Su and Xu 2018).”
• Mitigating Byzantine Attacks in Federated Learning (2020, background): “To address Byzantine failures in federated learning, a number of strategies have been proposed recently [7]–[10].”
• Privacy and Robustness in Federated Learning: Attacks and Defenses (2020, methodology): “strate that exchanging gradients in FL can also leak sensitive information about the participants’ private data to either passive or active inference attacks [19], [20], [73], [74], [75].”
• Federated Variance-Reduced Stochastic Gradient Descent With Robustness to Byzantine Attacks (2019, methodology): “Robust aggregations other than the geometric median are available, including the median [12], Krum [15], marginal trimmed mean [13], and iterative filtering [14].”