Teacher-Student Compression with Generative Adversarial Networks

By Ruishan Liu, Nicoló Fusi, Lester W. Mackey. 2018

19 citations recorded by the source corpus.

More accurate machine learning models often demand more computation and memory at test time, making them difficult to deploy on CPU- or memory-constrained devices. Teacher-student compression (TSC), also known as distillation, alleviates this burden by training a less expensive student model to mimic the expensive teacher model while maintaining most of the original accuracy. However, when fresh data is unavailable for the compression task, the teacher's training data is typically reused, leading to suboptimal compression. In this work, we propose to augment the compression dataset with synthetic data from a generative adversarial network (GAN) designed to approximate the training data distribution. Our GAN-assisted TSC (GAN-TSC) significantly improves student accuracy for expensive models such as large random forests and deep neural networks on both tabular and image datasets. Building on these results, we propose a comprehensive metric---the TSC Score---to evaluate the quality of synthetic datasets based on their induced TSC performance. The TSC Score captures both data diversity and class affinity, and we illustrate its benefits over the popular Inception Score in the context of image classification.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Jianping Gou
3. S. Maybank
4. Lin Wang
5. Kuk-Jin Yoon
6. Baosheng Yu
7. Luc Van Gool
8. Trevor Darrell

Citations in the corpus

• Knowledge Distillation: A Survey (2020, methodology): “Furthermore, the parameters of the teacher model (or the connections between layers) also contain another knowledge (Liu et al. 2019c).”
• Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, background): “Method GAN type Purpose Inputs of D Number of D Online KD Chen [31] First type Classification Logits One No Belagiannis [19] First type Classification Features One No Liu [135] First type ClassificationObject detection Features One No Hong [89] First type Object detection Features Six No Wang [229] ”