Private Model Compression via Knowledge Distillation

By Ji Wang, Weidong Bao, Lichao Sun, Xiaomin Zhu, Bokai Cao, Philip S. Yu. 2018

133 citations recorded by the source corpus.

The soaring demand for intelligent mobile applications calls for deploying powerful deep neural networks (DNNs) on mobile devices. However, the outstanding performance of DNNs notoriously relies on increasingly complex models, which in turn is associated with an increase in computational expense far surpassing mobile devices’ capacity. What is worse, app service providers need to collect and utilize a large volume of users’ data, which contain sensitive information, to build the sophisticated DNN models. Directly deploying these models on public mobile devices presents prohibitive privacy risk. To benefit from the on-device deep learning without the capacity and privacy concerns, we design a private model compression framework RONA. Following the knowledge distillation paradigm, we jointly use hint learning, distillation learning, and self learning to train a compact and fast neural network. The knowledge distilled from the cumbersome model is adaptively bounded and carefully perturbed to enforce differential privacy. We further propose an elegant query sample selection method to reduce the number of queries and control the privacy loss. A series of empirical evaluations as well as the implementation on an Android mobile device show that RONA can not only compress cumbersome models efficiently but also provide a strong privacy guarantee. For example, on SVHN, when a meaningful (9.83,10−6)-differential privacy is guaranteed, the compact model trained by RONA can obtain 20× compression ratio and 19× speed-up with merely 0.97% accuracy loss.

Subfields: Artificial Intelligence, Computer Networks and Communications, Transportation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Transportation

Citations in the corpus

Federated Model Distillation with Noise-Free Differential Privacy (2020, methodology): “In order to protect training data privacy in FL, various privacy protection techniques have been proposed in the literature [ Geyer et al. , 2017; McMahan et al. , 2018; Bonawitz et al. , 2017; Wang et al. , 2019b; Zhao et al. , 2020; Sun et al. , 2020a; Lyu et al. , 2020].”
Improved Knowledge Distillation via Teacher Assistant (2019, methodology): “Adversarial methods also have been utilized for modeling knowledge transfer between teacher and student (Heo et al. 2018; Xu, Hsu, and Huang 2018; Wang et al. 2018b; 2018c).”
Communication-Efficient and Distributed Learning Over Wireless Networks: Principles and Applications (2020, unknown): “In KD, the output of a pretrained model may leak the raw data used during the pretraining process [67].”
Improved Knowledge Distillation via Teacher Assistant: Bridging the Gap Between Student and Teacher (2019, methodology): “Another line of research focuses on distilling the original (large) network to a smaller network (Polino et al., 2018; Wang et al., 2018a), in which case the smaller network is more flexible in its architecture design does not have to be a sub-graph of the original network.”
Highlight Every Step: Knowledge Distillation via Collaborative Teaching (2019, background): “During the past few years, many researchers have been studying how to deploy these deep neural networks in practice [17], [43], [44].”