Group Knowledge Transfer: Collaborative Training of Large CNNs on the Edge

By Chaoyang He, Murali Annavaram, Amir Salman Avestimehr. 2020

24 citations recorded by the source corpus.

Scaling up the convolutional neural network (CNN) size (e.g., width, depth, etc.) is known to effectively improve model accuracy. However, the large model size impedes training on resource-constrained edge devices. For instance, federated learning (FL) on edge devices cannot tackle large CNN training demands, even though there is a strong practical need for FL due to its privacy and confidentiality properties. To address the resource-constrained reality, we reformulate FL as a group knowledge transfer (GKT) training algorithm. GKT designs a variant of the alternating minimization approach to train small CNNs on edge nodes and periodically transfer their knowledge by knowledge distillation to a large server-side CNN. GKT consolidates several advantages in a single framework: reduced demand for edge computation, lower communication cost for large CNNs, and asynchronous training, all while maintaining model accuracy comparable to FL. To simplify the edge training, we also develop a distributed training system based on our GKT. We train CNNs designed based on ResNet-56 and ResNet-110 using three distinct datasets (CIFAR-10, CIFAR-100, and CINIC-10) and their non-IID variants. Our results show that GKT can obtain comparable or even slightly higher accuracy. More importantly, GKT makes edge training affordable. Compared to the edge training using FedAvg, GKT demands 9 to 17 times less computational power (FLOPs) on edge devices and requires 54 to 105 times fewer parameters in the edge CNN.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Federated & Distributed Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Murali Annavaram
2. Chaoyang He
3. Qiang Yang
4. Yang Liu
5. S. Avestimehr
6. Praneeth Vepakomma
7. Abhishek Singh
8. Ramesh Raskar

In Computer Vision and Pattern Recognition

1. Murali Annavaram
2. Ramesh Raskar
3. Qiang Yang
4. S. Avestimehr
5. Chaoyang He
6. Xu Chen
7. Yang Liu
8. Praneeth Vepakomma

In Efficiency & Compression

1. Murali Annavaram
2. S. Avestimehr
3. Chaoyang He
4. Zhi Zhou
5. Ramesh Raskar
6. Li Shen
7. Qiang Yang
8. Xu Chen

In Federated & Distributed Learning

1. Murali Annavaram
2. Chaoyang He
3. Qiang Yang
4. Yang Liu
5. S. Avestimehr
6. Praneeth Vepakomma
7. Abhishek Singh
8. Yan Kang

Citations in the corpus

• FedML: A Research Library and Benchmark for Federated Machine Learning (2020, methodology): “For example, [127] has proposed an efficient training algorithm for large CNN models on edge devices.”
• From distributed machine learning to federated learning: a survey (2021, background): “In addition, knowledge distillation can also be exploited to aggregate the models, while requiring that there is data in the centralized server [60].”
• FedHome: Cloud-Edge Based Personalized Federated Learning for In-Home Health Monitoring (2020, methodology): “[41], we also compare our FedHome model with FL-CNNLarge, a CNN-based model whose parameter number is 1.”
• Distributed Distillation for On-Device Learning (2020, unknown): “To improve the computational efficiency, [38] integrates distillation as a mediator between FL and Split Learning [37].”