StNet: Local and Global Spatial-Temporal Modeling for Action Recognition

By Dongliang He, Zhichao Zhou, Chuang Gan, Fu Li, Xiao Liu, Yandong Li, Limin Wang, Shilei Wen. 2018

145 citations recorded by the source corpus.

Despite the success of deep learning for static image understanding, it remains unclear what are the most effective network architectures for spatial-temporal modeling in videos. In this paper, in contrast to the existing CNN+RNN or pure 3D convolution based approaches, we explore a novel spatialtemporal network (StNet) architecture for both local and global modeling in videos. Particularly, StNet stacks N successive video frames into a super-image which has 3N channels and applies 2D convolution on super-images to capture local spatial-temporal relationship. To model global spatialtemporal structure, we apply temporal convolution on the local spatial-temporal feature maps. Specifically, a novel temporal Xception block is proposed in StNet, which employs a separate channel-wise and temporal-wise convolution over the feature sequence of a video. Extensive experiments on the Kinetics dataset demonstrate that our framework outperforms several state-of-the-art approaches in action recognition and can strike a satisfying trade-off between recognition accuracy and model complexity. We further demonstrate the generalization performance of the leaned video representations on the UCF101 dataset.

Subfields: Artificial Intelligence, Biomedical Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Shiji Song
2. Gao Huang
3. Limin Wang
4. Yi Yang
5. Lorenzo Torresani
6. Yulin Wang
7. Le Yang
8. Zhan Tong

In Biomedical Engineering

1. Limin Wang
2. Matt Feiszli
3. Yi Yang
4. Laura Sevilla-Lara
5. Lorenzo Torresani
6. Chen Qian
7. Wayne Wu
8. Tong Lu

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Ming-Hsuan Yang
3. Yi Yang
4. Limin Wang
5. Yu Qiao
6. Dacheng Tao
7. Jitendra Malik
8. Xiaogang Wang

Citations in the corpus

• Dynamic Neural Networks: A Survey (2021, background): “, 3D CNNs [263], [264], [265], most of them still follow a static inference scheme.”
• TAM: Temporal Adaptive Module for Video Recognition (2020, methodology): “Since the deep learning method has been wildly used in the image tasks, there are many attempts [18, 31, 40, 46, 10, 23, 39] based on 2D CNNs devoted to modeling the video clips.”
• Only Time Can Tell: Discovering Temporal Data for Temporal Modeling (2019, background): “Temporal convolution methods [35, 36, 38, 25, 14] extend the success of 2D convolutions in image understanding to include convolutions across frames that are adjacent in time.”
• STM: SpatioTemporal and Motion Encoding for Action Recognition (2019, unknown): “StNet [12] ResNet50 ImageNet+Kinetics 93.”
• Multi-Agent Reinforcement Learning Based Frame Sampling for Effective Untrimmed Video Recognition (2019, methodology): “Given the heavy computation overhead of I3D, a series of works [52, 33, 5, 56, 42, 44, 20] are done to strike good effectiveness-efficiency trade-off.”
• TDN: Temporal Difference Networks for Efficient Action Recognition (2020, methodology): “StNet [9] proposed a local and global module to model temporal information hierarchically.”
• TEA: Temporal Excitation and Aggregation for Action Recognition (2020, methodology): “Nonlocal module [43] and stnet [13] applied self-attention mechanism to model long-range temporal relationship.”
• AttentionNAS: Spatiotemporal Attention Cell Search for Video Classification (2020, unknown): “22 - 411 StNet-IRv2 [9] 78.”
• Dynamic Inference: A New Approach Toward Efficient Video Action Recognition (2020, methodology): “Following the I3D [1] paradigm for spatialtemporal modeling, S3D [34], P3D [22], R(2+1)D [27], MFNet [2] and StNet [12] are proposed to reduce computation overhead of 3D convolution while remaining the spatial-temporal modeling property.”