Generative Models for Effective ML on Private, Decentralized Datasets

By Sean Augenstein, H. Brendan McMahan, Daniel Ramage, Swaroop Indra Ramaswamy, Peter Kairouz, Mingqing Chen, Rajiv Mathews, Blaise Agüera y Arcas. 2019

211 citations recorded by the source corpus.

To improve real-world applications of machine learning, experienced modelers develop intuition about their datasets, their models, and how the two interact. Manual inspection of raw data - of representative samples, of outliers, of misclassifications - is an essential tool in a) identifying and fixing problems in the data, b) generating new modeling hypotheses, and c) assigning or refining human-provided labels. However, manual data inspection is problematic for privacy sensitive datasets, such as those representing the behavior of real-world individuals. Furthermore, manual data inspection is impossible in the increasingly important setting of federated learning, where raw examples are stored at the edge and the modeler may only access aggregated outputs such as metrics or model parameters. This paper demonstrates that generative models - trained using federated methods and with formal differential privacy guarantees - can be used effectively to debug many commonly occurring data issues even when the data cannot be directly inspected. We explore these methods in applications to text with differentially private federated RNNs and to images using a novel algorithm for differentially private federated GANs.

Subfields: Artificial Intelligence, Generative Models, Federated & Distributed Learning, Privacy & Security

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Generative Models

In Federated & Distributed Learning

In Privacy & Security

Citations in the corpus

Advances and Open Problems in Federated Learning (2019, background): “[32] explore demonstrate the generation of synthetic data in a federated fashion.”
Privacy Amplification via Random Check-Ins (2020, methodology): “Attempts to combine FL and the above lines of DP research have been made previously; notably, [28, 3] extended the approach of [1] to FL and user-level DP.”
On the Outsized Importance of Learning Rates in Local Update Methods (2020, background): “…and has seen a large volume of work on the intersection of federated learning with topics including differential privacy (McMahan et al., 2018; Augenstein et al., 2020), fairness (Mohri et al., 2019; Li et al., 2020b), robustness (Ghosh et al., 2019; Bagdasaryan et al., 2018; Sun et al.,…”
Three Approaches for Personalization with Applications to Federated Learning (2020, background): “For other applications, one can train a GAN and send the GAN to the clients and the clients can sample from the GAN to create the dataset C (Augenstein et al., 2019).”
Generative Adversarial Networks in Time Series: A Systematic Literature Review (2022, background): “Decentralised/distributed approaches to GAN algorithms require large communication bandwidth to ensure convergence [5, 46] and are also subject to strict privacy constraints.”
Differentially Private Federated Learning: An Information-Theoretic Perspective (2021, methodology): “Augenstein et al. [15] recently studied the same setting where, after T iterations, the last model parameters are used to generate synthetic data for data inspection purposes.”
Unleashing the Power of Edge-Cloud Generative AI in Mobile Networks: A Survey of AIGC Services (2023, unknown): “Therefore, in [213], the authors propose a differential private”