Differentially Private Distributed Online Learning

By Chencheng Li, Pan Zhou, Li Xiong, Qian Wang, Ting Wang. 2015

83 citations recorded by the source corpus.

Online learning has been in the spotlight from the machine learning society for a long time. To handle massive data in Big Data era, one single learner could never efficiently finish this heavy task. Hence, in this paper, we propose a novel distributed online learning algorithm to solve the problem. Comparing to typical centralized online learner, the distributed learners optimize their own learning parameters based on local data sources and timely communicate with neighbors. However, communication may lead to a privacy breach. Thus, we use differential privacy to preserve the privacy of learners, and study the influence of guaranteeing differential privacy on the utility of the distributed online learning algorithm. Furthermore, by using the results from Kakade and Tewari (2009), we use the regret bounds of online learning to achieve fast convergence rates for offline learning algorithms in distributed scenarios, which provides tighter utility performance than the existing state-of-the-art results. In simulation, we demonstrate that the differentially private offline learning algorithm has high variance, but we can use mini-batch to improve the performance. Finally, the simulations show that the analytical results of our proposed theorems are right and our private distributed online learning algorithm is a general framework.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Edwige Cyffers
2. En Zhu
3. Xinwang Liu
4. Yanjun Qi
5. Aurélien Bellet
6. Jianping Yin
7. Beilun Wang
8. Ji Gao

Citations in the corpus

• Privacy Amplification by Decentralization (2020, methodology): “In LDP, random perturbations are performed locally by each user, making it convenient to design private versions of fully de-centralized algorithms in this model (see e.g., Huang et al., 2015; Bellet et al., 2018; Li et al., 2018; Cheng et al., 2019; Zhang et al., 2018; Xu et al., 2020).”
• Orthogonal Gradient Descent for Continual Learning (2019, unknown): “…previous knowledge (i.e. through the model prediction gradients not the actual data) is also related differential private learning (Wu et al., 2017; Li et al., 2018; Pihur et al., 2018; Han et al., 2018) and federated learning (Bonawitz et al., 2019; Smith et al., 2017; Vepakomma et al., 2018).”
• Understand Dynamic Regret with Switching Cost for Online Decision Making (2019, background): “Other problems include online learning [29, 39, 42, 43], online recommendation [41], online classi€cation [6, 18], online portfolio selection [28], and model predictive control [36] are usually studied in the seŠing of OCO.”
• A Theoretical Framework for Robustness of (Deep) Classifiers against Adversarial Examples (2016, background): “Recent studies have proposed various strategies (Xie et al., 2014; Bojarski et al., 2014; Stoddard et al., 2014; Li & Zhou, 2015; Rajkumar & Agarwal, 2012; Dwork, 2011; Nock et al., 2015) to preserve the privacy of data such as differential privacy.”