Probabilistic Knowledge Transfer for Lightweight Deep Representation Learning

By Nikolaos Passalis, Maria Tzelepi, Anastasios Tefas. 2020

135 citations recorded by the source corpus.

Knowledge-transfer (KT) methods allow for transferring the knowledge contained in a large deep learning model into a more lightweight and faster model. However, the vast majority of existing KT approaches are designed to handle mainly classification and detection tasks. This limits their performance on other tasks, such as representation/metric learning. To overcome this limitation, a novel probabilistic KT (PKT) method is proposed in this article. PKT is capable of transferring the knowledge into a smaller student model by keeping as much information as possible, as expressed through the teacher model. The ability of the proposed method to use different kernels for estimating the probability distribution of the teacher and student models, along with the different divergence metrics that can be used for transferring the knowledge, allows for easily adapting the proposed method to different applications. PKT outperforms several existing state-of-the-art KT techniques, while it is capable of providing new insights into KT by enabling several novel applications, as it is demonstrated through extensive experiments on several challenging data sets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Jianping Gou
3. S. Maybank
4. Baosheng Yu
5. Chixiang Zhang
6. Feng Zhou
7. Qiang Meng
8. Xiaoqiang Xu

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Jianping Gou
3. S. Maybank
4. Baosheng Yu
5. Chixiang Zhang
6. Feng Zhou
7. Qiang Meng
8. Xiaoqiang Xu

Citations in the corpus

• Knowledge Distillation: A Survey (2020, methodology): “To explore the pairwise hint information, the student model also mimics the mutual information flow from pairs of hint layers of the teacher model (Passalis et al. 2020b).”
• Learning Compatible Embeddings (2021, background): “PKT [31] proposes to align probability distributions in the feature spaces for representation/metric learning tasks.”