Bringing Giant Neural Networks Down to Earth with Unlabeled Data

By Yehui Tang, Shan You, Chang Xu, Boxin Shi, Chao Xu. 2019

11 citations recorded by the source corpus.

Compressing giant neural networks has gained much attention for their extensive applications on edge devices such as cellphones. During the compressing process, one of the most important procedures is to retrain the pre-trained models using the original training dataset. However, due to the consideration of security, privacy or commercial profits, in practice, only a fraction of sample training data are made available, which makes the retraining infeasible. To solve this issue, this paper proposes to resort to unlabeled data in hand that can be cheaper to acquire. Specifically, we exploit the unlabeled data to mimic the classification characteristics of giant networks, so that the original capacity can be preserved nicely. Nevertheless, there exists a dataset bias between the labeled and unlabeled data, which may disturb the training and degrade the performance. We thus fix this bias by an adversarial loss to make an alignment on the distributions of their low-level feature representations. We further provide theoretical discussions about how the unlabeled data help compressed networks to generalize better. Experimental results demonstrate that the unlabeled data can significantly improve the performance of the compressed networks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Chen Qian
2. Fei Wang
3. Jianlong Wu
4. Xiaojie Li
5. Changshui Zhang
6. Shan You
7. Shangchen Du
8. Tao Huang

In Computer Vision and Pattern Recognition

1. Fei Wang
2. Chen Qian
3. Jianlong Wu
4. Changshui Zhang
5. Xiaojie Li
6. Shan You
7. Shangchen Du
8. Tao Huang

Citations in the corpus

• GreedyNAS: Towards Fast One-Shot NAS With Greedy Supernet (2020, background): “By dint of automatic feature engineering, deep neural networks (DNNs) have achieved remarkable success in various computer vision tasks, such as image classification [37, 36, 41, 33, 32, 15, 39], visual generation [34, 35], image retrieval [40, 7, 8, 12] and semantic comprehension [18, 17].”
• Learning Student Networks with Few Data (2020, methodology): “…required by deep neural networks, several techniques to directly compress existing trained networks are investigated, such as vector quantization (Gong et al. 2014), hash encoding (Chen et al. 2015), weight matrices decomposition (Denton et al. 2014) and us-ing unlabeled data (Tang et al. 2019).”
• Agree to Disagree: Adaptive Ensemble Knowledge Distillation in Gradient Space (2020, background): “With this aim, efforts have been made to design more compact networks, compress or prune a pretrained model [28, 29] or even search for a tiny backbone network [37, 39].”