Human Action Recognition Using Factorized Spatio-Temporal Convolutional Networks

By Lin Sun, K. Jia, Dit-Yan Yeung, Bertram E. Shi. 2015

551 citations recorded by the source corpus.

Human actions in video sequences are three-dimensional (3D) spatio-temporal signals characterizing both the visual appearance and motion dynamics of the involved humans and objects. Inspired by the success of convolutional neural networks (CNN) for image classification, recent attempts have been made to learn 3D CNNs for recognizing human actions in videos. However, partly due to the high complexity of training 3D convolution kernels and the need for large quantities of training videos, only limited success has been reported. This has triggered us to investigate in this paper a new deep architecture which can handle 3D signals more effectively. Specifically, we propose factorized spatio-temporal convolutional networks (FstCN) that factorize the original 3D convolution kernel learning as a sequential process of learning 2D spatial kernels in the lower layers (called spatial convolutional layers), followed by learning 1D temporal kernels in the upper layers (called temporal convolutional layers). We introduce a novel transformation and permutation operator to make factorization in FstCN possible. Moreover, to address the issue of sequence alignment, we propose an effective training and inference strategy based on sampling multiple video clips from a given action video sequence. We have tested FstCN on two commonly used benchmark datasets (UCF-101 and HMDB-51). Without using auxiliary training videos to boost the performance, FstCN outperforms existing CNN based methods and achieves comparable performance with a recent method that benefits from using auxiliary training videos.

Subfields: Artificial Intelligence, Biomedical Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Multiview Transformers for Video Recognition (2022, neutral): “Since the release of the Kinetics dataset [33], 3D CNNs [10, 24, 69] have gained popularity, and many variants [22, 63, 70, 71, 79] have been developed to improve the speed and accuracy.”
Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification (2017, positive): “This idea has been used in a variety of recent papers, including [11] (who call it “R(2+1)D”), [15] (who call it “Pseudo-3D network”), [16] (who call it “factorized spatio-temporal convolutional networks”), etc.”
UvA-DARE (Digital Academic Repository) Self-Supervised Video Representation Learning With Odd-One-Out Networks (positive): “Inspired by the recently proposed family of motion representations [5, 37, 43] that compress arbitrary length video sequences into fixed-dimensional tensors while maintaining their spatio-temporal structure, we propose a new video-segment level representation learning strategy for videos in self-sup”
Temporal Segment Networks: Towards Good Practices for Deep Action Recognition (2016, methodology): “Inspired by [28], We experiment with adding stacked RGB diﬀerence as another input modality and investigate its performance in action recognition.”
RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, methodology): “[125] factorized a 3D filter into a combination of 2D and 1D filters.”
Convolutional Two-Stream Network Fusion for Video Action Recognition (2016, background): “Another way of learning spatiotemporal relationships is proposed in [26], where the authors factorize 3D convolution into a 2D spatial and a 1D temporal convolution.”
Actions ~ Transformations (2015, methodology): “Recurrent Neural Networks have also been used to encode temporal information for learning video representations [43, 4, 30, 45, 57, 46].”
A Closer Look at Spatiotemporal Convolutions for Action Recognition (2017, methodology): “Our architecture, called R(2+1)D, is related to Factorized Spatio-Temporal Convolutional Networks [33] (FSTCN ) in the way of factorizing spatiotemporal convolutions into spatial and temporal ones.”
Temporal Convolutional Networks for Action Segmentation and Detection (2016, background): “Large-scale Recognition: There has been substantial work on spatiotemporal models for large scale video classification and detection [31, 11, 12, 27, 33, 23, 19].”
TSM: Temporal Shift Module for Efficient Video Understanding (2018, methodology): “Another way to save computation is to decompose the 3D convolution into a 2D spatial convolution and a 1D temporal convolution [46, 33, 42].”
ViViT: A Video Vision Transformer (2021, background): “As 3D convolutional networks require signiﬁcantly more computation than their image counterparts, many architectures factorise convolutions across spatial and temporal dimensions and/or use grouped convo-lutions [59, 66, 67, 81, 20].”
Temporal Segment Networks for Action Recognition in Videos (2017, methodology): “FSTCN[54] 59:1% FSTCN [54] 88:1% TDD+FV [5] 63:2% TDD+FV [5] 90:3%”
Asynchronous Temporal Fields for Action Recognition (2016, methodology): “Moving towards more expressive deep networks such as LSTM has become a popular method for encoding such temporal information [48, 4, 65, 50, 58, 41, 64].”
Segmental Spatiotemporal CNNs for Fine-Grained Action Segmentation (2016, background): “This is much larger than those used other related work (e.g. [2,28]).”
Grouped Spatial-Temporal Aggregation for Efficient Action Recognition (2019, methodology): “One widely used way is to decouple the spatial and temporal part [23, 17, 26, 33].”
Lattice Long Short-Term Memory for Human Action Recognition (2017, methodology): “Inspired by [30], we apply temporal data augmentation for training the network.”
Spatio-Temporal Channel Correlation Networks for Action Classification (2018, background): “Sun et al. [18] decomposed the 3D convolutions into 2D spatial and 1D temporal convolutions.”
Learning Long-Term Dependencies for Action Recognition with a Biologically-Inspired Deep Network (2016, unknown): “0% F ST CN [33] 59 .”