Cross-Modal Knowledge Distillation for Action Recognition

By Fida Mohammad Thoker, Juergen Gall. 2019

105 citations recorded by the source corpus.

In this work, we address the problem how a network for action recognition that has been trained on a modality like RGB videos can be adapted to recognize actions for another modality like sequences of 3D human poses. To this end, we extract the knowledge of the trained teacher network for the source modality and transfer it to a small ensemble of student networks for the target modality. For the cross-modal knowledge distillation, we do not require any annotated data. Instead we use pairs of sequences of both modalities as supervision, which are straightforward to acquire. In contrast to previous works for knowledge distillation that use a KL-loss, we show that the cross-entropy loss together with mutual learning of a small ensemble of student networks performs better. In fact, the proposed approach for cross-modal knowledge distillation nearly achieves the accuracy of a student network trained with full supervision.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

1. Jun Liu
2. Qiuhong Ke
3. Hossein Rahmani
4. Zehua Sun
5. Dacheng Tao
6. Yu Qiao
7. S. Maybank
8. Junjie Yan

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Jun Liu
3. Luc Van Gool
4. Hossein Rahmani
5. Xiaogang Wang
6. Yu Qiao
7. Qiuhong Ke
8. Ming-Hsuan Yang

In Human-Computer Interaction

1. Jun Liu
2. Dacheng Tao
3. Qiuhong Ke
4. Hossein Rahmani
5. Zehua Sun
6. Yu Qiao
7. S. Maybank
8. Junjie Yan

Citations in the corpus

• Knowledge Distillation: A Survey (2020, background): “The idea behind the paired samples is to transfer the annotation or label information via pair-wise sample registration and has been widely used for cross-modal applications (Albanie et al. 2018; Zhao et al. 2018; Thoker and Gall 2019).”
• Collaborative Distillation in the Parameter and Spectrum Domains for Video Action Recognition (2020, background): “Knowledge distillation has been studied widely and shown feasibility in the image-level task (Thoker and Gall 2019; Hinton, Vinyals, and Dean 2015; Heo et al. 2019; Park et al. 2019), which aims to train small and efficient video networks (student) with comparable performance as large ones (teacher).”
• Human Action Recognition From Various Data Modalities: A Review (2020, methodology): “investigated in existing works [24], [26], [360].”