AdaCliP: Adaptive Clipping for Private SGD

By Venkatadheeraj Pichapati, Ananda Theertha Suresh, Felix X. Yu, Sashank J. Reddi, Sanjiv Kumar. 2019

149 citations recorded by the source corpus.

Privacy preserving machine learning algorithms are crucial for learning models over user data to protect sensitive information. Motivated by this, differentially private stochastic gradient descent (SGD) algorithms for training machine learning models have been proposed. At each step, these algorithms modify the gradients and add noise proportional to the sensitivity of the modified gradients. Under this framework, we propose AdaCliP, a theoretically motivated differentially private SGD algorithm that provably adds less noise compared to the previous methods, by using coordinate-wise adaptive clipping of the gradient. We empirically demonstrate that AdaCliP reduces the amount of added noise and produces models with better accuracy.

Subfields: Artificial Intelligence, Optimization & Training, Privacy & Security

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

In Privacy & Security

Citations in the corpus

Advances and Open Problems in Federated Learning (2019, neutral): “Hence, similar to recent works on user-level DP in the central model [24], figuring out how to adaptively bound the contributions of users and clip the model parameters remains an interesting research direction [381, 324].”
Practical and Private (Deep) Learning without Sampling or Shuffling (2021, background): “Differentially private empirical risk minimization (ERM) and private online learning are well-studied areas in the privacy literature [1, 2, 3, 6, 7, 14, 29, 35, 37, 38, 43, 51, 57, 58, 65, 67, 68, 74] 7 .”
Privacy Amplification via Random Check-Ins (2020, background): “We formally deﬁne the random check-ins procedure, describe a fully distributed DP-SGD protocol with random check-ins, and analyze its privacy and utility guarantees.”
Differentially Private Learning with Adaptive Clipping (2019, neutral): “Pichapati et al. (2019) propose a method for coordinate-wise adaptive clipping for DP-SGD, however it also contains a hyperparameter ( h 2 ) that is difﬁcult to tune.”
Large-Scale Differentially Private BERT (2021, background): “Other works also considered adaptive clipping in the context of DP-SGD (Andrew et al., 2019; Pichapati et al., 2019) as well as adaptive learning rates (Koskela and Honkela, 2020).”
Characterizing Private Clipped Gradient Descent on Convex Generalized Linear Problems (2020, unknown): “Over the past few years, there has been tremendous progress in differentially private convex empirical risk minimization (ERM) [9, 7, 38, 1, 6, 29, 44, 21, 33, 40, 15].”
Adversary Instantiation: Lower Bounds for Differentially Private Machine Learning (2021, background): “Theoretical work producing upper bounds of privacy is extensive [7, 4, 51, 1, 38, 55, 25, 47, 20, 21, 1, 4, 51, 40].”
Understanding Gradient Clipping in Private SGD: A Geometric Perspective (2020, methodology): “In Pichapati et al. [2019], an adaptive gradient clipping method is analyzed and the divergence is characterized by a bias depending on the di ﬀ erence between the clipped and unclipped gradients.”
Deep Learning with Gaussian Differential Privacy (2019, unknown): “This technique has been extended to a variety of situations by follow-up work [39, 46].”