Improving the Privacy and Accuracy of ADMM-Based Distributed Algorithms

By Xueru Zhang, Mohammad Mahdi Khalili, M. Liu. 2018

95 citations recorded by the source corpus.

Alternating direction method of multiplier (ADMM) is a popular method used to design distributed versions of a machine learning algorithm, whereby local computations are performed on local data with the output exchanged among neighbors in an iterative fashion. During this iterative process the leakage of data privacy arises. A differentially private ADMM was proposed in prior work (Zhang & Zhu, 2017) where only the privacy loss of a single node during one iteration was bounded, a method that makes it difficult to balance the tradeoff between the utility attained through distributed computation and privacy guarantees when considering the total privacy loss of all nodes over the entire iterative process. We propose a perturbation method for ADMM where the perturbed term is correlated with the penalty parameters; this is shown to improve the utility and privacy simultaneously. The method is based on a modified ADMM where each node independently determines its own penalty parameter in every iteration and decouples it from the dual updating step size. The condition for convergence of the modified ADMM and the lower bound on the convergence rate are also derived.

Subfields: Artificial Intelligence, Computer Networks and Communications, Federated & Distributed Learning, Privacy & Security

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Federated & Distributed Learning

In Privacy & Security

Citations in the corpus

Privacy Amplification by Decentralization (2020, methodology): “In LDP, random perturbations are performed locally by each user, making it convenient to design private versions of fully de-centralized algorithms in this model (see e.g., Huang et al., 2015; Bellet et al., 2018; Li et al., 2018; Cheng et al., 2019; Zhang et al., 2018; Xu et al., 2020).”
Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization (2019, background): “In this paper, we are particularly interested in synchronous distributed stochastic gradient descent algorithms for non-convex optimization problems mainly due to their recent successes and popularity in deep learning models [26, 29, 46, 47].”
Learning Privately over Distributed Features: An ADMM Sharing Approach (2019, methodology): “…machine learning schemes, including gradient-based meth-ods Li et al. (2014a,b); Hsieh et al. (2017); Ho et al. (2013) and ADMM-based methods Zhang et al. (2018); Shi et al. (2014); Zhang and Zhu (2016a); Huang et al. (2018), have been proposed to enable learning from distributed samples,…”
DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression (2019, background): “such as differentially private distributed optimization (Jayaraman et al., 2018; Zhang et al., 2018), adaptive distributed ADMM (Xu et al.”
Cross-Iteration Coded Computing (2018, background): “Specifically, we focus on reducing the number of communication rounds, which is often the bottleneck in distributed computing [1]–[5].”
DP-ADMM: ADMM-Based Distributed Learning With Differential Privacy (2018, background): “Among the works on distributed learning with differential privacy, most of them focus on subgradient-based algorithms [35]–[38] and only a few works consider ADMMbased methods [1], [2], [17]–[19].”
Recycled ADMM: Improving the Privacy and Accuracy of Distributed Algorithms (2019, methodology): “In our prior work [22], the total privacy loss of the whole network over the entire iterative process is considered.”