Temporal Pyramid Network for Action Recognition

By Ceyuan Yang, Yinghao Xu, Jianping Shi, Bo Dai, Bolei Zhou. Computer Vision and Pattern Recognition, 2020

445 citations recorded by the source corpus.

Visual tempo characterizes the dynamics and the temporal scale of an action. Modeling such visual tempos of different actions facilitates their recognition. Previous works often capture the visual tempo through sampling raw videos at multiple rates and constructing an input-level frame pyramid, which usually requires a costly multi-branch network to handle. In this work we propose a generic Temporal Pyramid Network (TPN) at the feature-level, which can be flexibly integrated into 2D or 3D backbone networks in a plug-and-play manner. Two essential components of TPN, the source of features and the fusion of features, form a feature hierarchy for the backbone so that it can capture action instances at various tempos. TPN also shows consistent improvements over other challenging baselines on several action recognition datasets. Specifically, when equipped with TPN, the 3D ResNet-50 with dense sampling obtains a 2\% gain on the validation set of Kinetics-400. A further analysis also reveals that TPN gains most of its improvements on action classes that have large variances in their visual tempos, validating the effectiveness of TPN.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/a30477966d4b96331f704f876cc23ea1ff09ea5b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

VA-RED2: Video Adaptive Redundancy Reduction (2021, methodology): “We further test our VA-RED2 with all of the three model architectures: R(2+1)D-18, I3D-InceptionV2, and X3D-M (Table 2) including the very recent temporal pyramid module (Yang et al., 2020) and correlation module (Wang et al., 2020) on Mini-Kinetics-200 dataset.”
FineGym: A Hierarchical Video Dataset for Fine-Grained Action Understanding (2020, background): “The remarkable progress in action recognition [39, 42, 40, 25, 37, 49], particularly the development of many new recognition models, e.”
2D or not 2D? Adaptive 3D Convolution Selection for Efficient Video Recognition (2020, background): “Towards this goal, extensive studies have been conducted with 3D convolutional networks by extending 2D convolutions over time [33, 4, 35, 34, 9, 8, 45].”
Generative Hierarchical Features from Synthesizing Images (2020, methodology): “In addition, we introduce a lightweight Spatial Alignment Module (SAM) [56, 38] into the encoder structure to better capture the spatial information from the input image.”
Human Action Recognition From Various Data Modalities: A Review (2020, background): “[119] proposed a generic featurelevel Temporal Pyramid Network (TPN) to model speed variations in performing actions.”
Video Representation Learning with Visual Tempo Consistency (2020, methodology): “These results verify the effectiveness of utilizing the rich hierarchy inside a deep network, which correlate well with previous studies [50].”