Practical and Private (Deep) Learning without Sampling or Shuffling
By Peter Kairouz, B. McMahan, Shuang Song, Om Thakkar, Abhradeep Thakurta, Zheng Xu. 2021
253 citations recorded by the source corpus.
We consider training models with differential privacy (DP) using mini-batch gradients. The existing state-of-the-art, Differentially Private Stochastic Gradient Descent (DP-SGD), requires privacy amplification by sampling or shuffling to obtain the best privacy/accuracy/computation trade-offs. Unfortunately, the precise requirements on exact sampling and shuffling can be hard to obtain in important practical scenarios, particularly federated learning (FL). We design and analyze a DP variant of Follow-The-Regularized-Leader (DP-FTRL) that compares favorably (both theoretically and empirically) to amplified DP-SGD, while allowing for much more flexible data access patterns. DP-FTRL does not use any form of privacy amplification. The code is available at https://github.com/google-research/federated/tree/master/dp_ftrl and https://github.com/google-research/DP-FTRL .
Subfields: Artificial Intelligence, Optimization & Training, Privacy & Security
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Zheng Xu
- Abhradeep Thakurta
- Steve Chien
- H. Brendan McMahan
- Michael G. Rabbat
- Sergei Vassilvitskii
- Alexey Kurakin
- Hussein Hazimeh
In Optimization & Training
- Abhradeep Thakurta
- Zheng Xu
- Steve Chien
- H. Brendan McMahan
- Kshitiz Malik
- Michael G. Rabbat
- Natalia Ponomareva
- Alexey Kurakin
In Privacy & Security
- Zheng Xu
- Abhradeep Thakurta
- H. Brendan McMahan
- Steve Chien
- Sergei Vassilvitskii
- Alexey Kurakin
- Natalia Ponomareva
- Carson E. Denison
Citations in the corpus
- FedScale: Benchmarking Model and System Performance of Federated Learning (2021, background): “Instead of training a single global model, some efforts resort to training a mixture of models [19, 22], clustering clients over training [27], or enforcing guided client selection [36]; To tackle the scarce and heterogeneous device resource, FedAvg [42] reduces communication cost by performing mult”
- How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy (2023, methodology): “For more discussion of differential privacy in federated learning, including local differential privacy and other system considerations, we refer readers to Section 5 of Kairouz et al. (2021d) and Section 7.1 of Wang et al. (2021).”
- Federated Learning with Buffered Asynchronous Aggregation (2021, methodology): “We defer to Section B.1 in (Kairouz et al., 2021) for more in-depth descriptions of the functions.”