Bolt-on Differential Privacy for Scalable Stochastic Gradient Descent-based Analytics

By Xi Wu, Fengan Li, Arun Kumar, Kamalika Chaudhuri, Somesh Jha, Jeffrey F. Naughton. 2016

239 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Optimization & Training, Privacy & Security

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

In Privacy & Security

Citations in the corpus

Practical and Private (Deep) Learning without Sampling or Shuffling (2021, background): “Differentially private empirical risk minimization (ERM) and private online learning are well-studied areas in the privacy literature [1, 2, 3, 6, 7, 14, 29, 35, 37, 38, 43, 51, 57, 58, 65, 67, 68, 74]7.”
Privacy Amplification via Random Check-Ins (2020, background): “We formally deﬁne the random check-ins procedure, describe a fully distributed DP-SGD protocol with random check-ins, and analyze its privacy and utility guarantees.”
Generative Models for Effective ML on Private, Decentralized Datasets (2019, background): “Most prior work on differentially private ML (e.g. Chaudhuri et al. (2011); Bassily et al. (2014); Abadi et al. (2016); Wu et al. (2017); Papernot et al. (2017)) deals with example-level privacy: two datasets d and d′ are defined to be adjacent if d′ can be formed by adding or removing a single…”
Privacy-Preserving Tensor Factorization for Collaborative Health Data Analysis (2019, methodology): “,τ , β is the Lipschitz constant of the gradient of (15) regarding B[t ] or C[t ], see Appendix in [23] for β calculation), the L2-sensitivity of this optimization problem in (15) is calculated as ∆2(f ) = 2τLη [30].”
Differentially Private Learning with Adaptive Clipping (2019, neutral): “In example-level DP, datasets are considered neighbors when they differ by the addition or removal of a single example (Chaudhuri et al., 2011; Bassily et al., 2014; Abadi et al., 2016b; Papernot et al., 2017; Wu et al., 2017; Papernot et al., 2018; Iyengar et al., 2019).”
A General Approach to Adding Differential Privacy to Iterative Training Procedures (2018, positive): “…on integrating differential privacy (DP) techniques into iterative training procedures like stochastic gradient descent [Chaudhuri et al., 2011, Bassily et al., 2014, Abadi et al., 2016, Wu et al., 2017, Papernot et al., 2017]; for completeness we provide a formal definition of DP in Appendix A.”
Privacy Amplification by Iteration (2018, background): “Output perturbation∗ [16], [26] Õ (√ d ε2n ) Õ (√ dk ε2n ) ∞”
Differentially-Private "Draw and Discard" Machine Learning (2018, background): “The optimism for the first claim stems from experience with other differentially private applications and the very recent work of [19]; for the second – from the similarities between DDML and the shuffling strategy of [6] and privacy amplification by sampling exploited by [32], [47], [1]; for the th”
cpSGD: Communication-efficient and differentially-private distributed SGD (2018, background): “Providing rigorous privacy guarantees for machine learning applications has been an area of active recent interest [6, 36, 32].”
Federated Learning: Challenges, Methods, and Future Directions (2019, methodology): “For gradient-based learning methods, a popular approach is to apply differential privacy by randomly perturbing the intermediate output at each iteration [e.g., 2, 7, 126].”
Orthogonal Gradient Descent for Continual Learning (2019, methodology): “The way we treat previous knowledge (i.e. through the model prediction gradients not the actual data) is also related diﬀerential private learning (Wu et al., 2017; Li et al., 2018; Pihur et al., 2018; Han et al., 2018) and federated learning (Bonawitz et al., 2019; Smith et al., 2017; Vepakomma et…”
Learning Differentially Private Language Models Without Losing Accuracy (2017, background): “The main goal of this work is to provide a strong guarantee that the trained model protects the privacy of individuals’ data without undue sacriﬁce in model quality, building upon recent work in applying differential privacy to machine learning [11, 10, 16, 8, 6, 5, 18, 24, 22, 1, 20, 26].”
Towards Practical Differentially Private Convex Optimization (2019, methodology): “Here, for small values of γ (for example, γ = O ( 1 n2 ) ), the excess empirical risk of Approximate Minima Perturbation is asymptotically the same as that of objective perturbation [9], and has a better dependence on n than that of Private Permutationbased SGD [12].”
Does learning require memorization? a short tale about a long tail (2019, background): “The best results (that we are aware of) on modern benchmarks that are achieved without interpolation are those for differentially private (DP) training algorithms [Aba+16; PAEGT16; PAEGT17; MRTZ18].”
Private Stochastic Convex Optimization with Optimal Rates (2019, background): “Related work: Differentially private empirical risk minimization (ERM) is a well-studied area spanning over a decade [CM08, CMS11, JKT12, KST12, ST13, SCS13, DJW13, Ull15, JT14, BST14, TTZ15, STU17, WLK + 17, WYX17, INS + 19].”
Privacy-Preserving Stochastic Gradual Learning (2018, background): “bridged differentially-private SGD with a practical RDBMS system [35].”
Automatic Discovery of Privacy–Utility Pareto Fronts (2019, methodology): “[49] provide a practical analysis-backed approach to privately training utility-maximizing models (again, for the case of SGD with a fixed privacy constraint), but hyperparameter optimization is naïvely performed using grid-search.”