FedNAS: Federated Deep Learning via Neural Architecture Search

By Chaoyang He, Murali Annavaram, Amir Salman Avestimehr. 2020

98 citations recorded by the source corpus.

Federated Learning (FL) has been proved to be an effective learning framework when data cannot be centralized due to privacy, communication costs, and regulatory restrictions. When training deep learning models under an FL setting, people employ the predefined model architecture discovered in the centralized environment. However, this predefined architecture may not be the optimal choice because it may not fit data with non-identical and independent distribution (non-IID). Thus, we advocate automating federated learning (AutoFL) to improve model accuracy and reduce the manual design effort. We specifically study AutoFL via Neural Architecture Search (NAS), which can automate the design process. We propose a Federated NAS (FedNAS) algorithm to help scattered workers collaboratively searching for a better architecture with higher accuracy. We also build a system based on FedNAS. Our experiments on non-IID dataset show that the architecture searched by FedNAS can outperform the manually predefined architecture.

Subfields: Artificial Intelligence, Neural Architecture Search & AutoML, Federated & Distributed Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Neural Architecture Search & AutoML

In Federated & Distributed Learning

Citations in the corpus

FedML: A Research Library and Benchmark for Federated Machine Learning (2020, methodology): “For example, [5] has studied large DNN for FL in the hospital scenario.”
Attack of the Tails: Yes, You Really Can Backdoor Federated Learning (2020, background): “Related Work Due to its emphasis on preserving privacy, FL has gained significant attention in the community [12, 39, 40, 41, 42].”
Federated Mutual Learning (2020, background): “Clients may have different hardware capabilities (Wu et al., 2019; He et al., 2020), different representation of local data (Liang et al.”
Self-supervised Cross-silo Federated Neural Architecture Search (2021, methodology): “In the past, NAS has been applied to HFL scenarios with privacy considerations [21]– [24].”
Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge (2020, background): “Recent works FedNAS [27, 3] and [28] work on large CNNs, but they rely on GPU training to complete the evaluations.”
Federated Neural Architecture Search (2020, methodology): “Federated AutoML The initial concept of automating machine DecNAS A concurrent work with DecNAS , FedNAS [17], also GPU-equipped edge server located in an organization, while DecNAS”
Learn from Others and Be Yourself in Heterogeneous Federated Learning (2022, background): “On the other hand, due to different design criteria, distinct hardware capabilities [20, 86] and intellectual property rights [56], participants require to customize models, which poses a practical challenge: model heterogeneity.”
FedProto: Federated Prototype Learning across Heterogeneous Clients (2021, methodology): “Some recent studies have also attempted to combine the neural architecture search with federated learning (Zhu, Zhang, and Jin 2020; He, Annavaram, and Avestimehr 2020; Singh et al. 2020), which can be applied to discover a customized model architecture for each group of clients with different…”
Federated learning for 6G communications: Challenges, methods, and future directions (2020, background): “For example, a study in [38] proposed a federated NAS (FedNAS) algorithm to help distributed devices collaborate to find a better architecture with higher accuracy.”
The future of digital health with federated learning (2020, background): “However, not only does this severely limit data availability, it is only practical for cases in which the consent granted by the data owners or patients is unconditional, since recalling data from those who may have had access to the data is practically unenforceable.”
Differentially-private Federated Neural Architecture Search (2020, unknown): “In (He et al., 2020), each client locally performs neural architecture search.”