Differentially-private Federated Neural Architecture Search

By Ishika Singh, Haoyi Zhou, Kunlin Yang, Mengxiao Ding, Bill Lin, Pengtao Xie. 2020

25 citations recorded by the source corpus.

Neural architecture search, which aims to automatically search for architectures (e.g., convolution, max pooling) of neural networks that maximize validation performance, has achieved remarkable progress recently. In many application scenarios, several parties would like to collaboratively search for a shared neural architecture by leveraging data from all parties. However, due to privacy concerns, no party wants its data to be seen by other parties. To address this problem, we propose federated neural architecture search (FNAS), where different parties collectively search for a differentiable architecture by exchanging gradients of architecture variables without exposing their data to other parties. To further preserve privacy, we study differentially-private FNAS (DP-FNAS), which adds random noise to the gradients of architecture variables. We provide theoretical guarantees of DP-FNAS in achieving differential privacy. Experiments show that DP-FNAS can search highly-performant neural architectures while protecting the privacy of individual parties. The code is available at https://github.com/UCSD-AI4H/DP-FNAS

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Qiang Yang
2. Yang Liu
3. Murali Annavaram
4. Chaoyang He
5. Guodong Long
6. Jing Jiang
7. Chengqi Zhang
8. Tianyi Zhou

In Computer Vision and Pattern Recognition

1. Murali Annavaram
2. S. Avestimehr
3. Chaoyang He
4. Ramesh Raskar
5. Qiang Yang
6. Yang Liu
7. Praneeth Vepakomma
8. Peilin Zhao

Citations in the corpus

• FedML: A Research Library and Benchmark for Federated Machine Learning (2020, background): “In terms of exchanged information, aside from exchanging gradients and models, recent FL algorithms propose to exchange information such as pseudo labels in semi-supervised FL [56] and architecture parameters in neural architecture search-based FL [5, 57, 58].”
• FedProto: Federated Prototype Learning across Heterogeneous Clients (2021, background): “…have also attempted to combine the neural architecture search with federated learning (Zhu, Zhang, and Jin 2020; He, Annavaram, and Avestimehr 2020; Singh et al. 2020), which can be applied to discover a customized model architecture for each group of clients with different hardware capabilities…”
• Federated Learning on Non-IID Data: A Survey (2021, background): “However, only a limited amount of research on federated NAS has been reported [145, 121, 48, 169] and little work has considered the influence of Non-IID distributions.”
• Self-supervised Cross-silo Federated Neural Architecture Search (2021, background): “[22] studied DP-FNAS algorithm which aggregates the gradients from the local network, and uses differential privacy to protect the communication contents.”