Learning Deep Representations with Probabilistic Knowledge Transfer

By Nikolaos Passalis, Anastasios Tefas. 2018

487 citations recorded by the source corpus.

Knowledge Transfer (KT) techniques tackle the problem of transferring the knowledge from a large and complex neural network into a smaller and faster one. However, existing KT methods are tailored towards classification tasks and they cannot be used efficiently for other representation learning tasks. In this paper we propose a novel probabilistic knowledge transfer method that works by matching the probability distribution of the data in the feature space instead of their actual representation. Apart from outperforming existing KT techniques, the proposed method allows for overcoming several of their limitations providing new insight into KT as well as novel KT applications, ranging from KT from handcrafted feature extractors to cross-modal KT from the textual modality into the representation extracted from the visual modality of the data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Knowledge Distillation: A Survey (2020, methodology): “g it important to transfer knowledge between diﬀerent modalities. In view of these facts, there are some cross-modal knowledge distillation methods recently (Roheda et al., 2018; Garcia et al., 2018; Passalis and Tefas, 2018; Do et al., 2019 ;Gupta et al. ,2016 Albanie et al. 2018 Zhao et al. 2018;K”
Learning Metrics From Teachers: Compact Networks for Image Embedding (2019, methodology): “The experiments show that the relative teacher network significantly outperforms DarkRank and PKT , obtaining 1.8% and 4.9% more on CUB200-2011 and 2.3% and 29.7% on Cars-196.”
Block-Wisely Supervised Neural Architecture Search With Knowledge Distillation (2019, background): “To address this problem, the second category of knowledge distillation proposes to employ the internal representation of the teacher to guide the training of the student [26, 37, 36, 33, 25].”
Probabilistic Knowledge Transfer for Lightweight Deep Representation Learning (2020, methodology): “This article is an extended version of our previous work presented in [23].”
ISD: Self-Supervised Learning by Iterative Similarity Distillation (2020, methodology): “Similarity based knowledge distillation: While the above methods [16, 3, 30, 36, 38] only extract the information about a single data point from the teacher, similarity based distillation methods [27, 29, 26, 33, 2, 32, 1] represent the knowledge from the teacher in terms of similarities between dat”
R-DFCIL: Relation-Guided Representation Learning for Data-Free Class Incremental Learning (2022, methodology): “Since then, various KD methods [22,13,26,18] have been developed.”
Distilled Person Re-Identification: Towards a More Scalable System (2019, methodology): “PKT [40] used probability distribution for knowledge distillation, which is not so effective as using similarity in our method for Re-ID.”
Knowledge Distillation Meets Self-Supervision (2020, methodology): “We compare our method with representative knowledge distillation methods, including: KD [16], FitNet [37], AT [46], SP [43], VID [2], RKD [33], PKT [34], AB [15], FT [20], CRD [42].”
Fully Dynamic Inference With Deep Neural Networks (2020, background): “enhance the knowledge distillation process by introducing a concept called feature space probability distribution loss [24].”
Heterogeneous Knowledge Distillation Using Information Flow Modeling (2020, methodology): “The proposed scheme indeed leads to faster convergence over both single layer KD using the PKT method, as well as over the multi-layer PKT-H method.”
Factorizing Knowledge in Neural Networks (2022, background): “Since the seminal work of [20], a series of KD approaches have been proposed to strengthen the performances of student models [52,67,48].”
ExpandNets: Linear Over-parameterization to Train Compact Convolutional Networks (2018, methodology): “For CIFAR-10 and CIFAR-100 [27], we use the same compact network as in [37] (architecture and training setting in the supplementary material).”
Learning an Evolutionary Embedding via Massive Knowledge Distillation (2020, methodology): “Besides, in Table 2, EEL also outperforms other state-of-the-art methods, including KD [16], Logits [1], Fitnet [50], Attention Transfer (AT) [69], Flow of Solution Procedure (FSP) [67], Neural Selective Transfer (NST) [21], Probabilistic Knowledge Transfer (PKT) [46], Factor Transfer (FT) [24], Dee”
Subclass Distillation (2020, methodology): “Several distillation methods have been proposed in the last few years (Zagoruyko & Komodakis, 2016; Tung & Mori, 2019; Peng et al., 2019; Ahn et al., 2019; Park et al., 2019; Passalis & Tefas, 2018; Heo et al., 2018; Kim et al., 2018; Yim et al., 2017; Huang & Wang, 2017).”
Quality-Agnostic Image Recognition via Invertible Decoder (2021, methodology): “For the knowledge distillation methods [21, 37, 36, 1, 43], the reported hyperparameters are used.”
Distilling Object Detectors via Decoupled Features (2021, background): “FitNets [44] shows that activations [23] and features of intermediate layers [39] can also be treated as knowledge to guide the student network.”
Curriculum Temperature for Knowledge Distillation (2022, methodology): “As an easy-to-use plugin technique, we applied our CTKD to the existing distillation frameworks including vanilla KD (Hinton, Vinyals, and Dean 2015), PKT (Passalis and Tefas 2018), SP (Tung and Mori 2019), VID (Ahn et al.”