Differentially Private Meta-Learning

By Jeffrey Li, Mikhail Khodak, Sebastian Caldas, Ameet Talwalkar. 2019

127 citations recorded by the source corpus.

Parameter-transfer is a well-known and versatile approach for meta-learning, with applications including few-shot learning, federated learning, and reinforcement learning. However, parameter-transfer algorithms often require sharing models that have been trained on the samples from specific tasks, thus leaving the task-owners susceptible to breaches of privacy. We conduct the first formal study of privacy in this setting and formalize the notion of task-global differential privacy as a practical relaxation of more commonly studied threat models. We then propose a new differentially private algorithm for gradient-based parameter transfer that not only satisfies this privacy requirement but also retains provable transfer learning guarantees in convex settings. Empirically, we apply our analysis to the problems of federated learning with personalization and few-shot classification, showing that allowing the relaxation to task-global privacy from the more commonly studied notion of local privacy leads to dramatically increased performance in recurrent neural language modeling and image classification.

Subfields: Artificial Intelligence, Meta & Continual Learning, Privacy & Security

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Meta & Continual Learning

In Privacy & Security

Citations in the corpus

Advances and Open Problems in Federated Learning (2019, background): “Additional connections with differential privacy were studied in [260].”
Federated Learning With Differential Privacy: Algorithms and Performance Analysis (2019, background): “Also, the work in [31] focused on the class of gradient-based parameter-transfer methods and developed a DP based meta learning algorithm that not only satisfies the privacy requirement but also retains provable learning performance in convex settings.”
Backdoor Attacks on Federated Meta-Learning (2020, background): “While the use of meta-learning in a federated setting and its privacy concerns were explored by previous work [10, 11], the influence of backdoor attacks on federated meta-learning has not been investigated.”
A Comprehensive Survey of Privacy-preserving Federated Learning (2021, methodology): “The global differential privacy scheme has been widely used in many FL methods [3, 31, 40, 63, 69, 70, 78, 94, 128, 149, 172, 193].”
A Survey on Federated Learning for Resource-Constrained IoT Devices (2021, unknown): “Another interesting approach to a DP mechanism based on metalearning is proposed in [136] that can be used in FL through personalization.”
Privacy and Robustness in Federated Learning: Attacks and Defenses (2020, background): “[109] proposed locally differentially-private algorithms in the context of meta-learning, which might be applicable to federated learning with personalization, while also providing provable learning guarantees in convex settings.”
Personalized Federated Learning with Theoretical Guarantees: A Model-Agnostic Meta-Learning Approach (2020, methodology): “A similar idea is later used in [37] to design differentially private algorithms with application in FL.”