Compressing GANs using Knowledge Distillation

By Angeline Aguinaldo, Ping-yeh Chiang, Alex Gain, Ameya D. Patil, Kolten Pearson, Soheil Feizi. 2019

87 citations recorded by the source corpus.

Generative Adversarial Networks (GANs) have been used in several machine learning tasks such as domain transfer, super resolution, and synthetic data generation. State-of-the-art GANs often use tens of millions of parameters, making them expensive to deploy for applications in low SWAP (size, weight, and power) hardware, such as mobile devices, and for applications with real time capabilities. There has been no work found to reduce the number of parameters used in GANs. Therefore, we propose a method to compress GANs using knowledge distillation techniques, in which a smaller "student" GAN learns to mimic a larger "teacher" GAN. We show that the distillation methods used on MNIST, CIFAR-10, and Celeb-A datasets can compress teacher GANs at ratios of 1669:1, 58:1, and 87:1, respectively, while retaining the quality of the generated image. From our experiments, we observe a qualitative limit for GAN's compression. Moreover, we observe that, with a fixed parameter budget, compressed GANs outperform GANs trained using standard training methods. We conjecture that this is partially owing to the optimization landscape of over-parameterized GANs which allows efficient training using alternating gradient descent. Thus, training an over-parameterized GAN followed by our proposed compression scheme provides a high quality generative model with a small number of parameters.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Knowledge Distillation: A Survey (2020, methodology): “Besides, using knowledge distillation to compress GANs, a learned small GAN student network mimics a larger GAN teacher network via knowledge transfer (Aguinaldo et al. 2019; Li et al. 2020c).”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, background): “Method GAN type Purpose Inputs of D Number of D Online KD Chen [31] First type Classification Logits One No Belagiannis [19] First type Classification Features One No Liu [135] First type ClassificationObject detection Features One No Hong [89] First type Object detection Features Six No Wang [229] ”
Distilling portable Generative Adversarial Networks for Image Translation (2020, unknown): “Denote GT as the pretrained teacher generator and GS as the portable student generator, a straightforward method, which was proposed in (Aguinaldo et al. 2019), to adopt knowledge distillation to the student generator could be formulated as:”
TinyGAN: Distilling BigGAN for Conditional Image Generation (2020, background): “Perhaps the work most related to ours is [20], which to our knowledge is the first to apply knowledge distillation on GANs.”
Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillation (2021, methodology): “In the generative models, a generator can be compressed by distilling the latent features from a cumbersome generator [Aguinaldo et al. , 2019].”
Exclusivity-Consistency Regularized Knowledge Distillation for Face Recognition (2020, background): “Some studies [23, 2, 40, 4, 1, 54] extended knowledge distillation to other applications.”
GAN Compression: Efficient Architectures for Interactive Conditional GANs (2020, methodology): “[2] adopts this method to accelerate unconditional GANs.”
StyleGAN2 Distillation for Feed-forward Image Manipulation (2020, background): “[3] show that knowledge distillation is successfully applicable for generative models.”