Stochastic gradient descent with differentially private updates

By Shuang Song, Kamalika Chaudhuri, Anand D. Sarwate. 2013

762 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Optimization & Training, Privacy & Security

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

In Privacy & Security

Citations in the corpus

Deep Models Under the GAN: Information Leakage from Collaborative Deep Learning (2017, background): “Differential privacy plays an important role in deep learning as shown in [1, 38, 75, 77].”
Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data (2016, neutral): “Attempts to provide differential privacy for machine learning models led to a series of efforts on shallow machine learning models, including work by Bassily et al. (2014); Chaudhuri & Monteleoni (2009); Pathak et al. (2011); Song et al. (2013), and Wainwright et al. (2012).”
Practical and Private (Deep) Learning without Sampling or Shuffling (2021, background): “Differentially private stochastic gradient descent (DPSGD) (Song et al., 2013; Bassily et al., 2014; Abadi et al., 2016) has become state-of-the-art in training private (deep) learning models (Abadi et al.”
The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregation (2021, methodology): “, 2006b), and in particular, DP-SGD can be employed (Song et al., 2013; Bassily et al., 2014; Abadi et al., 2016; Tramèr & Boneh, 2020).”
Tempered Sigmoid Activations for Deep Learning with Differential Privacy (2020, methodology): “…al. (2016) introduced DP-SGD as a method for training deep neural networks with differential-privacy guarantees that was able to achieve better privacy and utility than previous efforts (Chaudhuri, Monteleoni, and Sarwate 2011; Song, Chaudhuri, and Sarwate 2013; Bassily, Smith, and Thakurta 2014).”
Privacy Amplification via Random Check-Ins (2020, methodology): “We formally deﬁne the random check-ins procedure, describe a fully distributed DP-SGD protocol with random check-ins, and analyze its privacy and utility guarantees.”
Federated Machine Learning (2019, neutral): “s require participants’ data to be secretly-shared among non-colluding servers. DifferentialPrivacy.AnotherlineofworkusetechniquesDifferentialPrivacy[18]ork-Anonymity [63] for data privacy protection [1, 12, 42, 61]. The methods of differential privacy, k-anonymity, and diversification [3] involve i”
Subsampled Rényi Differential Privacy and Analytical Moments Accountant (2018, methodology): “…private machine learning models is to use Stochastic Gradient Descent (SGD) with diﬀerentially private releases of (sometimes clipped) gradients evaluated on mini-batches of a dataset (Song et al., 2013; Bassily et al., 2014; Wang et al., 2015; Foulds et al., 2016; Abadi et al., 2016).”
Differentially-Private "Draw and Discard" Machine Learning (2018, background): “ML in the Trusted-Curator Model: Most prior work for differentially private machine learning assumes a trusted-curator model, where the data is ﬁrst collected by the company and only then a privacy-preserving computation is run on it [1], [48], [42], [9].”
Personalized and Private Peer-to-Peer Machine Learning (2017, methodology): “…a private global model, either by aggregating models trained locally by each party (Pathak et al., 2010; Hamm et al., 2016) or by perturbing the gradients and/or the objective in a distributed gradient descent algorithm (Song et al., 2013; Rajkumar and Agarwal, 2012; Shokri and Shmatikov, 2015).”
Scalable Private Learning with PATE (2018, background): “dler et al., 2017). The ﬁrst learning algorithms adapted to provide differential privacy with respect to their training data were often linear and convex (Pathak et al., 2010; Chaudhuri et al., 2011; Song et al., 2013; Bassily et al., 2014; Hamm et al., 2016). More recently, successful developments ”
To Drop or Not to Drop: Robustness, Consistency and Differential Privacy Properties of Dropout (2015, background): “…private optimization (learning) algorithms have received signiﬁcant attention Chaudhuri and Monteleoni [2008], Dwork and Lei [2009], Chaudhuri et al. [2011], Jain et al. [2012], Kifer et al. [2012], Duchi et al. [2013], Song et al. [2013], Jain and Thakurta [2014], Bassily et al. [2014].”
Differentially Private Empirical Risk Minimization: Efficient Algorithms and Tight Error Bounds (2014, background): “The main idea in [47] is to run stochastic gradient descent with gradients computed over small batches of disjoint samples from the data set (as opposed to one single sample used in Algorithm ANoise−GD).”
Learning Differentially Private Language Models Without Losing Accuracy (2017, background): “The main goal of this work is to provide a strong guarantee that the trained model protects the privacy of individuals’ data without undue sacriﬁce in model quality, building upon recent work in applying differential privacy to machine learning [11, 10, 16, 8, 6, 5, 18, 24, 22, 1, 20, 26].”
Encode, Shuffle, Analyze Privacy Revisited: Formalizations and Empirical Evaluation (2020, background): “At a high level, LDP-SGD follows the following template of noisy stochastic gradient descent [38], [39], [40].”
Towards Practical Differentially Private Convex Optimization (2019, methodology): “Here, for small values of γ (for example, γ = O ( 1 n2 ) ), the excess empirical risk of Approximate Minima Perturbation is asymptotically the same as that of objective perturbation [9], and has a better dependence on n than that of Private Permutationbased SGD [12].”
Differentially Private Model Publishing for Deep Learning (2019, background): “Because it is difficult to characterize the maximum difference of the model parameters over any two neighboring datasets for neural networks, differentially private deep learning [7], [28], [37] relies on differentially private stochastic gradient descent (DP-SGD) to control the influence of trainin”
Privacy for Free: Posterior Sampling and Stochastic Gradient Monte Carlo (2015, methodology): “The idea is to simply privately release an estimate of the gradient (as in Song et al. (2013); Bassily et al. (2014)) and leverage upon the following two celebrated lemmas in diﬀerential privacy in the same way as Bassily et al. (2014) does in deriving the near-optimal ( (cid:15), δ )-diﬀerentially…”
Private Stochastic Convex Optimization with Optimal Rates (2019, background): “The problem of ERM with differential privacy (DP-ERM) has been well-studied and asymptotically tight upper and lower bounds on excess loss 1 are known [CM08, CMS11, JKT12, KST12, ST13, SCS13, DJW13, Ull15, JT14, BST14, TTZ15, STU17, WLK + 17, WYX17, INS + 19].”