Knowledge Integration Networks for Action Recognition

By Shiwen Zhang, Sheng Guo, Limin Wang, Weilin Huang, Matthew R. Scott. 2020

22 citations recorded by the source corpus.

In this work, we propose Knowledge Integration Networks (referred as KINet) for video action recognition. KINet is capable of aggregating meaningful context features which are of great importance to identifying an action, such as human information and scene context. We design a three-branch architecture consisting of a main branch for action recognition, and two auxiliary branches for human parsing and scene recognition which allow the model to encode the knowledge of human and scene for action recognition. We explore two pre-trained models as teacher networks to distill the knowledge of human and scene for training the auxiliary tasks of KINet. Furthermore, we propose a two-level knowledge encoding mechanism which contains a Cross Branch Integration (CBI) module for encoding the auxiliary knowledge into medium-level convolutional features, and an Action Knowledge Graph (AKG) for effectively fusing high-level context information. This results in an end-to-end trainable framework where the three tasks can be trained collaboratively, allowing the model to compute strong context knowledge efficiently. The proposed KINet achieves the state-of-the-art performance on a large-scale action recognition benchmark Kinetics-400, with a top-1 accuracy of 77.8%. We further demonstrate that our KINet has strong capability by transferring the Kinetics-trained model to UCF-101, where it obtains 97.8% top-1 accuracy.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

Knowledge Distillation: A Survey (2020, methodology): “…mutual teacher–student networks (Thoker and Gall 2019), multiple stream networks (Garcia et al. 2018), spatiotemporal distilled dense-connectivity network (Hao and Zhang 2019), graph distillation (Luo et al. 2018) and multi-teacher to multi-student networks (Wu et al. 2019b; Zhang et al. 2020c).”