Federated Model Distillation with Noise-Free Differential Privacy

By Lichao Sun, Lingjuan Lyu. 2020

133 citations recorded by the source corpus.

Conventional federated learning directly averages model weights, which is only possible for collaboration between models with homogeneous architectures. Sharing prediction instead of weight removes this obstacle and eliminates the risk of white-box inference attacks in conventional federated learning. However, the predictions from local models are sensitive and would leak training data privacy to the public. To address this issue, one naive approach is adding the differentially private random noise to the predictions, which however brings a substantial trade-off between privacy budget and model performance. In this paper, we propose a novel framework called FEDMD-NFDP, which applies a Noise-FreeDifferential Privacy (NFDP) mechanism into a federated model distillation framework. Our extensive experimental results on various datasets validate that FEDMD-NFDP can deliver not only comparable utility and communication efficiency but also provide a noise-free differential privacy guarantee. We also demonstrate the feasibility of our FEDMD-NFDP by considering both IID and Non-IID settings, heterogeneous model architectures, and unlabelled public datasets from a different distribution.

Subfields: Artificial Intelligence, Federated & Distributed Learning, Privacy & Security

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Lichao Sun
2. Qiang Yang
3. Lingjuan Lyu
4. Philip S. Yu
5. Dacheng Tao
6. Bo Du
7. Han Yu
8. Mang Ye

In Federated & Distributed Learning

1. Murali Annavaram
2. Qiang Yang
3. S. Avestimehr
4. Chaoyang He
5. Dacheng Tao
6. Bo Du
7. Mang Ye
8. Philip S. Yu

In Privacy & Security

1. Bo Du
2. Mang Ye
3. Philip S. Yu
4. Chaoyang He
5. S. Avestimehr
6. Murali Annavaram
7. Jiayu Zhou
8. Junyuan Hong

Citations in the corpus

• Heterogeneous Federated Learning: State-of-the-art and Research Challenges (2023, background): “Therefore, FedMD-NFDP adds a noise-free DP mechanism to FedMD [115], which protects data privacy without introducing noise.”
• Learn from Others and Be Yourself in Heterogeneous Federated Learning (2022, methodology): “To tackle this challenge, one type of methods typically performs fine-tuning for several rounds [38, 50, 58, 74, 88].”
• LDP-FL: Practical Private Aggregation in Federated Learning with Local Differential Privacy (2020, methodology): “In order to solve the problem, recent works start to use differential privacy to protect the private data in federated learning [Nguyên et al., 2016; Lyu et al., 2020; Sun and Lyu, 2020], but most of them cannot give a practical solution for deep learning on complex datasets due to the trade-off bet”
• Secure Deep Graph Generation with Link Differential Privacy (2020, background): “In dealing with such tension between the need to release utilizable data and the concern of data owners’ privacy, quite a few secure deep generative models have been proposed recently, focusing on grid-based data like images, texts and gene sequences [Papernot et al., 2018; Sun et al., 2020a; Sun an”
• Privacy and Robustness in Federated Learning: Attacks and Defenses (2020, methodology): “DP CDP [7], [45] LDP [18], [83], [84], [85], [86], [87], [88] DDP+Cryptography [21], [89], [90] Secure Multiparty Computation [5], [91]”
• Data-Free Knowledge Distillation for Heterogeneous Federated Learning (2021, methodology): “Most existing work is data-dependent (Lin et al., 2020; Sun & Lyu, 2020; Guha et al., 2019; Chen & Chao, 2021).”
• Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge (2020, methodology): “We use KD [19] in a different manner from existing and concurrent works [38, 39, 40, 41, 42, 43, 44, 45].”