Distilling Knowledge From a Deep Pose Regressor Network

By Muhamad Risqi U. Saputra, Pedro Porto Buarque de Gusmão, Yasin Almalioglu, Andrew Markham, Agathoniki Trigoni. 2019

118 citations recorded by the source corpus.

This paper presents a novel method to distill knowledge from a deep pose regressor network for efficient Visual Odometry (VO). Standard distillation relies on ''dark knowledge'' for successful knowledge transfer. As this knowledge is not available in pose regression and the teacher prediction is not always accurate, we propose to emphasize the knowledge transfer only when we trust the teacher. We achieve this by using teacher loss as a confidence score which places variable relative importance on the teacher prediction. We inject this confidence score to the main training task via Attentive Imitation Loss (AIL) and when learning the intermediate representation of the teacher through Attentive Hint Training (AHT) approach. To the best of our knowledge, this is the first work which successfully distill the knowledge from a deep pose regression network. Our evaluation on the KITTI and Malaga dataset shows that we can keep the student prediction close to the teacher with up to 92.95% parameter reduction and 2.12x faster in computation time.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

Knowledge Distillation: A Survey (2020, background): “, 2019), visual odometry (Saputra et al., 2019) and visual question answering (Mun et al.”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “To tackle these challenges, many impressive KD methods [29], [32], [33], [53], [62], [77], [89], [98], [100], [110], [122], [146], [189], [209], [225], [237] have been proposed for compressing visual detection networks.”
FrugalScore: Learning Cheaper, Lighter and Faster Evaluation Metrics for Automatic Text Generation (2021, background): “A few attempts have been made on regression problems (Chen et al., 2017; Saputra et al., 2019; Takamoto et al., 2020), in which special losses were proposed to train the student with respect to both the teacher’s regression outputs and ground truth scores.”
Extracurricular Learning: Knowledge Transfer Beyond Empirical Distribution (2020, methodology): “[50] explored some variations of KD for regression.”
Few-Shot Learning With a Strong Teacher (2021, background): “Equation 5) by a weighted square loss [104], ∑”