DP-ADMM: ADMM-Based Distributed Learning With Differential Privacy

By Zonghao Huang, Rui Hu, Yuanxiong Guo, Eric Chan-Tin, Yanmin Gong. 2018

236 citations recorded by the source corpus.

Alternating direction method of multipliers (ADMM) is a widely used tool for machine learning in distri-buted settings where a machine learning model is trained over distributed data sources through an interactive process of local computation and message passing. Such an iterative process could cause privacy concerns of data owners. The goal of this paper is to provide differential privacy for ADMM-based distributed machine learning. Prior approaches on differentially private ADMM exhibit low utility under high privacy guarantee and assume the objective functions of the learning problems to be smooth and strongly convex. To address these concerns, we propose a novel differentially private ADMM-based distributed learning algorithm called DP-ADMM, which combines an approximate augmented Lagrangian function with time-varying Gaussian noise addition in the iterative process to achieve higher utility for general objective functions under the same differential privacy guarantee. We also apply the moments accountant method to analyze the end-to-end privacy loss. The theoretical analysis shows that the DP-ADMM can be applied to a wider class of distributed learning problems, is provably convergent, and offers an explicit utility-privacy tradeoff. To our knowledge, this is the first paper to provide explicit convergence and utility properties for differentially private ADMM-based distributed learning algorithms. The evaluation results demonstrate that our approach can achieve good convergence and model accuracy under high end-to-end differential privacy guarantee.

Subfields: Artificial Intelligence, Computer Networks and Communications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xueru Zhang
2. Mohammad Mahdi Khalili
3. M. Liu
4. Peng Liu
5. Farhad Farokhi
6. Mohamed Ali Kâafar
7. Di Niu
8. Nan Wu

In Computer Networks and Communications

1. Xueru Zhang
2. Mohammad Mahdi Khalili
3. M. Liu
4. Peng Liu
5. Di Niu
6. Linglong Kong
7. Yaochen Hu
8. Aurélien Bellet

Citations in the corpus

• Learning Privately over Distributed Features: An ADMM Sharing Approach (2019, methodology): “…Li et al. (2014a,b); Hsieh et al. (2017); Ho et al. (2013) and ADMM-based methods Zhang et al. (2018); Shi et al. (2014); Zhang and Zhu (2016a); Huang et al. (2018), have been proposed to enable learning from distributed samples, since collecting data for centralized learning will incur…”
• The Value of Collaboration in Convex Machine Learning with Differential Privacy (2019, background): “In [20], [21], the privacy budget was kept constant and therefore by communicating more, as the number of the iterations grows, the privacy guarantee weakens.”
• Recycled ADMM: Improving the Privacy and Accuracy of Distributed Algorithms (2019, methodology): “[23] applies the first-order approximation to the augmented Lagrangian in all iterations; however, this method requires a central server to average all updated primal variables over the network in each iteration.”