Learning Differentially Private Recurrent Language Models

By H. Brendan McMahan, Daniel Ramage, Kunal Talwar, Li Zhang. 2017

1,510 citations recorded by the source corpus.

Subfields: Large Language Models, Privacy & Security

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

In Privacy & Security

Citations in the corpus

Advances and Open Problems in Federated Learning (2019, methodology): “(3) In the context of FL, D and D′ correspond to decentralized datasets that are adjacent if D′ can be obtained from D by adding or subtracting all the records of a single client (user) [290].”
How To Backdoor Federated Learning (2018, methodology): “The attacker always knows this bound because it is sent to all participants [44].”
Federated Learning With Differential Privacy: Algorithms and Performance Analysis (2019, methodology): “The work in [32] introduced an algorithm for user-level differentially private training of large neural networks, in particular a complex sequence model for next-word prediction.”
Federated Learning: Collaborative Machine Learning without Centralized Training Data (2022, neutral): “It has already been applied to the distributed setting, but with a slightly diﬀerent goal: One does not try to hide the individual training samples, but whether a given client has participated in the training of the machine learning model or not [17, 11].”
Practical and Private (Deep) Learning without Sampling or Shuffling (2021, background): “This assumption was introduced and validated by [51], which showed that keeping the clipping norm bound, training rounds, and the scale of the noise added to the model update constant, increasing the report goal does not change the ﬁnal model accuracy.”
The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregation (2021, methodology): “We adopt user-level privacy in this work – i.e., each entry in the input corresponds to all the records associated with a single user (McMahan et al., 2018), and thus the privacy guarantee holds with respect to all data belonging to that user.”
Privacy Amplification by Decentralization (2020, neutral): “We denote by D = D 1 ∪· et al., 2018).”
Differentially Private Learning Needs Better Features (or Much More Data) (2020, methodology): “Following McMahan et al. (2018), we first consider the data complexity of private end-to-end learning.”
Tempered Sigmoid Activations for Deep Learning with Differential Privacy (2020, background): “Yet, its advantages have not been without cost: empirically, the test accuracy of differentially private ML is consistently lower than that of nonprivate learning unless the dataset is very large (McMahan et al. 2017).”
Privacy Amplification via Random Check-Ins (2020, background): “One can view this line of reasoning to be analogous to privacypreserving deep learning research, in that the theoretical guarantees for convex models acted as guiding principles for non-convex (deep learning) models [9, 1, 29].”
Attack of the Tails: Yes, You Really Can Backdoor Federated Learning (2020, background): “We further show that these attacks are robust to defense mechanisms based on differential privacy (DP) [27, 35], norm clipping [27], and robust aggregators such as Krum and Multi-Krum [17].”
Federated Learning and Differential Privacy: Software tools analysis, the Sherpa.ai FL framework and methodological guidelines for preserving data privacy (2020, methodology): “parameters through model-inversion adversarial attacks [16], resulting in a great risk of private data leakage. For this reason, DP techniques [12] are commonly used in order to share model parameters[17]. 8 2.3. The deﬁnition of Differential Privacy DP is the property of an algorithm whose input is”
On the Outsized Importance of Learning Rates in Local Update Methods (2020, background): “, 2020), and has seen a large volume of work on the intersection of federated learning with topics including differential privacy (McMahan et al., 2018; Augenstein et al., 2020), fairness (Mohri et al.”
Privacy in Deep Learning: A Survey (2020, background): “A large body of Differentially private mechanisms has been proposed for various applications [73, 24, 74].”
Personalized Federated Learning: A Meta-Learning Approach (2020, methodology): “In particular, we elaborate on its connections with the original FedAvg algorithm (McMahan et al., 2017a), and also, discuss a number of considerations that one need to take into account for implementing Per-FedAvg.”
Salvaging Federated Learning by Local Adaptation (2020, methodology): “To protect privacy, differentially private federated learning (McMahan et al., 2018) bounds how much the federated model can reveal about the input from any individual participant.”