Multi-teacher Knowledge Distillation for Compressed Video Action Recognition on Deep Neural Networks

By Meng-Chieh Wu, Ching-Te Chiu, Kun-Hsuan Wu. 2019

84 citations recorded by the source corpus.

Recently, convolutional neural networks (CNNs) have seen great progress in classifying images. Action recognition is different from still image classification; video data contains temporal information that plays an important role in video understanding. Currently, most CNN-based approaches for action recognition have excessive computational costs, with an explosion of parameters and computation time. The currently most efficient method trains a deep network directly on compressed video containing the motion information. However, this method has a large number of parameters. We propose a multi-teacher knowledge distillation framework for compressed video action recognition to compress this model. With this framework, the model is compressed by transferring the knowledge from multiple teachers to a single small student model. With multi-teacher knowledge distillation, students learn better than with single-teacher knowledge distillation. Experiments show that we can reach a 2.4× compression rate in a number of parameters and a 1.2× computation reduction with 1.79% loss of accuracy on the UCF-101 dataset and 0.35% loss of accuracy on the HMDB51 dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jian Pei
2. Yan Fu
3. Ze Yang
4. Linjun Shou
5. Fei Yuan
6. Ming Gong
7. Wutao Lin
8. Daxin Jiang

In Computer Vision and Pattern Recognition

1. Jian Pei
2. Ze Yang
3. Linjun Shou
4. Ming Gong
5. Wutao Lin
6. Daxin Jiang

In Human-Computer Interaction

1. Linjun Shou
2. Ming Gong
3. Wutao Lin
4. Daxin Jiang

Citations in the corpus

• Model Compression with Two-stage Multi-teacher Knowledge Distillation for Web Question Answering System (2019, methodology): “formance of different ensemble methods under the framework of multi-task learning. You et al. [29] presented a method to train a thin deep network by about the dissimilarity among examples. Wu et al. [27] propose a multi-teacher knowledge distillation framework for compressed video action recognitio”
• Reinforced Multi-Teacher Selection for Knowledge Distillation (2020, methodology): “Wu, Chiu, and Wu (2019) also assign a fixed weight to each teacher model and use the weighted average of the probability distributions from multiple teacher models to train a student model.”
• A Full Dive Into Realizing the Edge-Enabled Metaverse: Visions, Enabling Technologies, and Challenges (2022, unknown): “Instead of using only one large model, the authors in [197] propose to use multiple teachers to transfer the knowledge to a single student.”