Exploiting Spatial-Temporal Modelling and Multi-Modal Fusion for Human Action Recognition

By Dongliang He, Fu Li, Qijie Zhao, Xiang Long, Yi Fu, Shilei Wen. 2018

19 citations recorded by the source corpus.

In this report, our approach to tackling the task of ActivityNet 2018 Kinetics-600 challenge is described in detail. Though spatial-temporal modelling methods, which adopt either such end-to-end framework as I3D \cite{i3d} or two-stage frameworks (i.e., CNN+RNN), have been proposed in existing state-of-the-arts for this task, video modelling is far from being well solved. In this challenge, we propose spatial-temporal network (StNet) for better joint spatial-temporal modelling and comprehensively video understanding. Besides, given that multi-modal information is contained in video source, we manage to integrate both early-fusion and later-fusion strategy of multi-modal information via our proposed improved temporal Xception network (iTXN) for video understanding. Our StNet RGB single model achieves 78.99\% top-1 precision in the Kinetics-600 validation set and that of our improved temporal Xception network which integrates RGB, flow and audio modalities is up to 82.35\%. After model ensemble, we achieve top-1 precision as high as 85.0\% on the validation set and rank No.1 among all submissions.

Subfields: Artificial Intelligence, Biomedical Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

A Short Note on the Kinetics-700 Human Action Dataset (2019, background): “There was an improvement between the winning systems of the two challenges, with error getting down from 12.4% (in 2017) to 11.0% (in 2018) [1, 6].”
SlowFast Networks for Video Recognition (2018, background): “In particular, the gray-scale version of the Fast pathway is nearly as good as the RGB variant, but reduces FLOPs by ∼ 5%.”
A Short Note about Kinetics-600 (2018, background): “There was an improvement between the winning systems of the two challenges, with error going down from 12.4% (in 2017) to 11.0% (in 2018) [1, 4].”
Learning Spatio-Temporal Representation With Local and Global Diffusion (2019, methodology): “Since this dataset is recently made available for ActivityNet 2018 challenge, we show the performance of different approaches reported by the challenge winner [10] and challenge runner-up [48].”