Learning Student Networks with Few Data

By Shumin Kong, Tianyu Guo, Shan You. 2020

12 citations recorded by the source corpus.

Recently, the teacher-student learning paradigm has drawn much attention in compressing neural networks on low-end edge devices, such as mobile phones and wearable watches. Current algorithms mainly assume the complete dataset for the teacher network is also available for the training of the student network. However, for real-world scenarios, users may only have access to part of training examples due to commercial profits or data privacy, and severe over-fitting issues would happen as a result. In this paper, we tackle the challenge of learning student networks with few data by investigating the ground-truth data-generating distribution underlying these few data. Taking Wasserstein distance as the measurement, we assume this ideal data distribution lies in a neighborhood of the discrete empirical distribution induced by the training examples. Thus we propose to safely optimize the worst-case cost within this neighborhood to boost the generalization. Furthermore, with theoretical analysis, we derive a novel and easy-to-implement loss for training the student network in an end-to-end fashion. Experimental results on benchmark datasets validate the effectiveness of our proposed method.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Chen Qian
2. Fei Wang
3. Changshui Zhang
4. Shan You
5. Tao Huang
6. Mingmin Yang
7. Xiaotong Zhang
8. Xianchao Zhang

In Statistical and Nonlinear Physics

1. Changshui Zhang
2. Fei Wang

Citations in the corpus

• GreedyNAS: Towards Fast One-Shot NAS With Greedy Supernet (2020, background): “By dint of automatic feature engineering, deep neural networks (DNNs) have achieved remarkable success in various computer vision tasks, such as image classification [37, 36, 41, 33, 32, 15, 39], visual generation [34, 35], image retrieval [40, 7, 8, 12] and semantic comprehension [18, 17].”
• Agree to Disagree: Adaptive Ensemble Knowledge Distillation in Gradient Space (2020, unknown): “Besides, some approaches also implement distillation on the intermediate features via various manners, such as hints [22], distribution of features [10] and local correlations [14].”