Private Stochastic Non-Convex Optimization: Adaptive Algorithms and Tighter Generalization Bounds
By Yingxue Zhou, Xiangyi Chen, Mingyi Hong, Zhiwei Steven Wu, Arindam Banerjee. 2020
31 citations recorded by the source corpus.
We study differentially private (DP) algorithms for stochastic non-convex optimization. In this problem, the goal is to minimize the population loss over a $p$-dimensional space given $n$ i.i.d. samples drawn from a distribution. We improve upon the population gradient bound of ${\sqrt{p}}/{\sqrt{n}}$ from prior work and obtain a sharper rate of $\sqrt[4]{p}/\sqrt{n}$. We obtain this rate by providing the first analyses on a collection of private gradient-based methods, including adaptive algorithms DP RMSProp and DP Adam. Our proof technique leverages the connection between differential privacy and adaptive data analysis to bound gradient estimation error at every iterate, which circumvents the worse generalization bound from the standard uniform convergence argument. Finally, we evaluate the proposed algorithms on two popular deep learning tasks and demonstrate the empirical advantages of DP adaptive gradient methods over standard DP SGD.
Subfields: Artificial Intelligence, Computational Theory and Mathematics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Zheng Xu
- Abhradeep Thakurta
- Steve Chien
- H. Brendan McMahan
- Sergei Vassilvitskii
- Alexey Kurakin
- Hussein Hazimeh
- Natalia Ponomareva
In Computational Theory and Mathematics
- Zheng Xu
- H. Brendan McMahan
- Natalia Ponomareva
- Alexey Kurakin
- Sergei Vassilvitskii
- Carson E. Denison
- Steve Chien
- Abhradeep Thakurta
Citations in the corpus
- How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy (2023, methodology): “This simple strategy, e.g., DP-Adam, DP-AdaGrad etc., has been used in many previous works (Zhou et al., 2020; Anil et al., 2021; Li et al., 2022b; Yu et al., 2021) and is probably most commonly used by practitioners, who “privatize” for DP training their best performing optimizer from a…”