Knowledge distillation via adaptive instance normalization

By Jing Yang, Brais Martínez, Adrian Bulat, Georgios Tzimiropoulos. 2020

29 citations recorded by the source corpus.

This paper addresses the problem of model compression via knowledge distillation. To this end, we propose a new knowledge distillation method based on transferring feature statistics, specifically the channel-wise mean and variance, from the teacher to the student. Our method goes beyond the standard way of enforcing the mean and variance of the student to be similar to those of the teacher through an $L_2$ loss, which we found it to be of limited effectiveness. Specifically, we propose a new loss based on adaptive instance normalization to effectively transfer the feature statistics. The main idea is to transfer the learned statistics back to the teacher via adaptive instance normalization (conditioned on the student) and let the teacher network "evaluate" via a loss whether the statistics learned by the student are reliably transferred. We show that our distillation method outperforms other state-of-the-art distillation methods over a large set of experimental settings including different (a) network architectures, (b) teacher-student capacities, (c) datasets, and (d) domains.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Knowledge Distillation: A Survey (2020, methodology): “…of them belong to natural language understanding (NLU), and many of these KD methods for NLU are designed as the task-speciﬁc distillation (Tang et al. 2019; Turc et al. 2019; Mou et al. 2016) and multi-task distillation (Liu et al. 2019f; Yang et al. 2020c; Sanh et al. 2019; Clark et al. 2019).”