STM: SpatioTemporal and Motion Encoding for Action Recognition

By Boyuan Jiang, Mengmeng Wang, Weihao Gan, Wei Wu, Junjie Yan. IEEE International Conference on Computer Vision, 2019

450 citations recorded by the source corpus.

Spatiotemporal and motion features are two complementary and crucial information for video action recognition. Recent state-of-the-art methods adopt a 3D CNN stream to learn spatiotemporal features and another flow stream to learn motion features. In this work, we aim to efficiently encode these two features in a unified 2D framework. To this end, we first propose a STM block, which contains a Channel-wise SpatioTemporal Module (CSTM) to present the spatiotemporal features and a Channel-wise Motion Module (CMM) to efficiently encode motion features. We then replace original residual blocks in the ResNet architecture with STM blcoks to form a simple yet effective STM network by introducing very limited extra computation cost. Extensive experiments demonstrate that the proposed STM network outperforms the state-of-the-art methods on both temporal-related datasets (i.e., Something-Something v1 & v2 and Jester) and scene-related datasets (i.e., Kinetics-400, UCF-101, and HMDB-51) with the help of encoding spatiotemporal and motion features together.

Subfields: Artificial Intelligence, Biomedical Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/b812b20192ffac37b03bde0261934a2a8c7fdf47

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Multiscale Vision Transformers (2021, background): “Incorporating downsampling, shift invariance, and shared weights, ConvNets are de-facto standard backbones for computer vision tasks for image [65, 62, 88, 90, 46, 12, 15, 34, 93, 81, 41] and video [87, 31, 11, 79, 69, 106, 96, 30, 105, 35, 29, 117, 57].”
A Multigrid Method for Efficiently Training Video Models (2019, background): “Some recent studies propose lighter weight models that use efficient temporal modules [2,5,19,21,28–30,32,34,40,44,46] and/or exploit temporal redundancy [9,51].”
UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learning (2022, methodology): “3D Convolution Neural Networks (CNNs) have been dominant in video understanding (Tran et al., 2015; Feichtenhofer et al., 2019).”
Multi-Modal Domain Adaptation for Fine-Grained Action Recognition (2019, background): “The latest architectures have focused on reducing the high computational cost of 3D convolutions [12, 21, 61], yet still show improvements when reporting results of two-stream fusion [61].”
TDN: Temporal Difference Networks for Efficient Action Recognition (2020, methodology): “The work of TEINet [20] and STM [14] We present a video-level framework for learning action models from the entire video, coined as TDN. Based on the sparse sampling from multiple segments, our TDN aims to model both short-term and long-term motion information in our framework.”
Collaborative Distillation in the Parameter and Spectrum Domains for Video Action Recognition (2020, background): “Existing video action classiﬁers (i.e., TSN(Wang et al. 2016), STM(Jiang et al. 2019), I3D(Carreira and Zisserman 2017), etc.) learn unstructural features of scene and motion information, which are not conducive to transfer from teacher to student networks for mimic learning.”
TEA: Temporal Excitation and Aggregation for Action Recognition (2020, methodology): “The result of STM using efficient inference protocol is cited from Table 9 in [20].”
VidTr: Video Transformer Without Convolutions (2021, methodology): “The I3D was used as backbone for many following work including two-stream network [19, 55], the networks with focus on temporal modeling [31, 32, 59], and the 3D networks with reﬁned 3D convolution kernels [27, 33, 39, 44].”
MoViNets: Mobile Video Networks for Efficient Video Recognition (2021, background): “They extend 2D image models with a temporal dimension, most notably incorporating 3D convolution [28, 62, 63, 72, 23, 49, 29, 52].”
AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognition (2021, background): “• LSTM: Update per-frame predictions by hidden states in LSTM and averages all predictions as the video-level prediction.”
ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning for Action Recognition (2022, background): “As temporal information is important for modeling the dynamics, a variety of motion learning techniques has been introduced [75, 30, 49].”
When Video Classification Meets Incremental Classes (2021, background): “Recently, many works perform temporal modeling with an extra module instead of the simple 1D temporal convolution [22, 28, 30, 32, 39].”
Is Space-Time Attention All You Need for Video Understanding? (2021, background): “…performance, since it is able to span Method SSv2 Diving-48 ∗∗ SlowFast (Feichtenhofer et al., 2019b) 61.7 77.6 TSM (Lin et al., 2019) 63.4 N/A STM (Jiang et al., 2019) 64.2 N/A MSNet (Kwon et al., 2020) 64.7 N/A TEA (Li et al., 2020b) 65.1 N/A bLVNet (Fan et al., 2019) 65 about 12 seconds of a…”
UniFormer: Unifying Convolution and Self-Attention for Visual Recognition (2022, background): “Besides, many recent studies of video understanding [43], [52], [55], [58] focus on adapting vanilla 2D CNNs with elaborated temporal modeling modules, such as temporal shift [58], [66], motion enhancement [43], [55], [62], and spatiotemporal excitation [51], [52], etc.”
Multi-Modal Domain Adaptation for Fine-Grained Action Recognition (2020, background): “The latest architectures have focused on reducing the high computational cost of 3D convolutions [12, 21, 61], yet still show improvements when reporting results of two-stream fusion [61].”
Attention Bottlenecks for Multimodal Fusion (2021, background): “[34] Boyuan Jiang, MengMeng Wang, Weihao Gan, Wei Wu, and Junjie Yan.”
TAM: Temporal Adaptive Module for Video Recognition (2020, background): “, Nonlocal Net [41], ARTNet [38], STM [17], TDN [39]).”