Similarity-Preserving Knowledge Distillation

By Frederick Tung, Greg Mori. 2019

1,259 citations recorded by the source corpus.

Knowledge distillation is a widely applicable technique for training a student neural network under the guidance of a trained teacher network. For example, in neural network compression, a high-capacity teacher is distilled to train a compact student; in privileged learning, a teacher trained with privileged data is distilled to train a student without access to that data. The distillation loss determines how a teacher's knowledge is captured and transferred to the student. In this paper, we propose a new form of knowledge distillation loss that is inspired by the observation that semantically similar inputs tend to elicit similar activation patterns in a trained network. Similarity-preserving knowledge distillation guides the training of a student network such that input pairs that produce similar (dissimilar) activations in the teacher network produce similar (dissimilar) activations in the student network. In contrast to previous distillation methods, the student is not required to mimic the representation space of the teacher, but rather to preserve the pairwise similarities in its own representation space. Experiments on three public datasets demonstrate the potential of our approach.

Subfields: Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

Citations in the corpus

Class-Incremental Learning with Cross-Space Clustering and Controlled Transfer (2022, neutral): “Tung and Mori [49] propose a distillation objective that enforces an L2 loss in the student that constraints the similarities between activation maps of two examples to be consistent with those of the teacher.”
Piggyback GAN: Efficient Lifelong Learning for Image Conditioned Generation (2021, background): “Parameter efficiency can also be encouraged during training via architecture design [23,29], sparsity inducing priors [33], resource re-allocation [26], or knowledge distillation from a teacher network [34,41].”
Knowledge Distillation: A Survey (2020, methodology): “ping Gou1 et al. embedding, the similarities of the data in the teacher network are transferred into the student network. Tung and Mori proposed a similarity-preserving knowledge distillation method (Tung and Mori, 2019). In particular, similarity-preserving knowledge, which arises from the similar ”
Generative Feature Replay For Class-Incremental Learning (2020, methodology): “Note that feature distillation has also been used in other applications [32, 47, 55].”
Knowledge Distillation for Incremental Learning in Semantic Segmentation (2019, methodology): “6 L′′′′∗ D ([14]) 92.”
Knowledge Distillation Beyond Model Compression (2020, methodology): “SP [16] encourages the student to preserve the pairwise similarities in the teacher in such a way that data pairs that produce similar/dissimilar activations in the teacher, also produce similar/dissimilar activations in the student.”
Collaborative Multi-Robot Search and Rescue: Planning, Coordination, Perception, and Active Vision (2020, background): “Common approaches include knowledge distillation [206], where a compact student network is trained to mimic a larger network, e.g., by guiding the network to produce similar activations for similar inputs, and advanced network models, such as Operational Neural Networks [207], where the linear…”
Patch-based Knowledge Distillation for Lifelong Person Re-Identification (2022, methodology): “SPD is an advanced feature distillation method.”
Relationship Matters: Relation Guided Knowledge Transfer for Incremental Learning of Object Detectors (2020, background): “Leveraging feature similarity [24, 28, 38] is also another recent trend for knowledge distillation.”
Focal and Global Knowledge Distillation for Detectors (2021, methodology): “There have been many works [10, 27, 33, 34] that improve the student classifiers significantly.”
Hyper-LifelongGAN: Scalable Lifelong Learning for Image Conditioned Generation (2021, methodology): “Regularization-based approaches were proposed addressing catastrophic forgetting by regularizing the network parameters when learning new tasks [21, 39, 7] or regularizing the discrepancy between the output of the old and new network using a distillation loss [22, 30, 29, 6].”
Anomaly Detection via Reverse Distillation from One-Class Embedding (2022, methodology): “For knowledge transfer in the T-S model, the cosine similarity is taken as the KD loss, as it is more precisely to capture the relation in both highand low-dimensional information [37, 49].”
Distilling Object Detectors via Decoupled Features (2021, methodology): “Since then, KD has been widely adopted in classification tasks [22, 63, 3, 54, 7, 65, 10].”
Quality-Agnostic Image Recognition via Invertible Decoder (2021, methodology): “For the knowledge distillation methods [21, 37, 36, 1, 43], the reported hyperparameters are used.”
Inter-Region Affinity Distillation for Road Marking Segmentation (2020, positive): “[23] take the similarity scores of features of different samples as distillation targets.”