Spatio-Temporal Channel Correlation Networks for Action Classification

By Ali Diba, Vivek Sharma, Mohammad Mahdi Arzani, Rahman Yousefzadeh, Juergen Gall, Luc Van Gool. 2018

197 citations recorded by the source corpus.

The work in this paper is driven by the question if spatio-temporal correlations are enough for 3D convolutional neural networks (CNN)? Most of the traditional 3D networks use local spatio-temporal features. We introduce a new block that models correlations between channels of a 3D CNN with respect to temporal and spatial features. This new block can be added as a residual unit to different parts of 3D CNNs. We name our novel block ‘Spatio-Temporal Channel Correlation’ (STC). By embedding this block to the current state-of-the-art architectures such as ResNext and ResNet, we improve the performance by 2–3% on the Kinetics dataset. Our experiments show that adding STC blocks to current state-of-the-art architectures outperforms the state-of-the-art methods on the HMDB51, UCF101 and Kinetics datasets. The other issue in training 3D CNNs is about training them from scratch with a huge labeled dataset to get a reasonable performance. So the knowledge learned in 2D CNNs is completely ignored. Another contribution in this work is a simple and effective technique to transfer knowledge from a pre-trained 2D CNN to a randomly initialized 3D CNN for a stable weight initialization. This allows us to significantly reduce the number of training samples for 3D CNNs. Thus, by fine-tuning this network, we beat the performance of generic and recent methods in 3D CNNs, which were trained on large video datasets, e.g. Sports-1M, and fine-tuned on the target datasets, e.g. HMDB51/UCF101.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

DynamoNet: Dynamic Action and Motion Network (2019, methodology): “Further, we utilize these T d-dimensional dynamic motion features along with STC-ResNext [6, 20] 3D-ConvNet features for video classification.”
Holistic Large Scale Video Understanding (2019, methodology): “In this work, we chose 3D-ResNet [41] and STCnet [7] as our 3D CNNs baseline which have competitive results on Kinetics and UCF101 datasets.”
TAM: Temporal Adaptive Module for Video Recognition (2020, methodology): “We notice that the structure of local branch is similar to the SENet [13] and STC [5].”
Attention mechanisms in computer vision: A survey (2021, background): “[5], [37], [53], [54], [55], [56], [57], [58] [25], [59], [60] Spatial attention Generate attention mask across spatial domains and use it to select important spatial regions (e.”
STM: SpatioTemporal and Motion Encoding for Action Recognition (2019, methodology): “Another type of methods tries to learn spatiotemporal features from RGB frames directly with 3D CNN [27, 2, 4, 7, 24].”
Human Action Recognition From Various Data Modalities: A Review (2020, methodology): “In their subsequent study [104], they introduced a new block embedded in some architectures such as ResNext and ResNet, which can model the inter-channel correlations of a 3D CNN with respect to the temporal and spatial features.”
DistInit: Learning Video Representations Without a Single Labeled Video (2019, methodology): “5 STC [5] 3D-ResNet 16 Knowledge Tx 82.”
Temporal Pyramid Network for Action Recognition (2020, unknown): “STC-X101 [4] 32 68.”
3D CNNs with Adaptive Temporal Feature Resolutions (2020, unknown): “Recently, meth-ods like HATNet [6], STC [5], and DynamoNet [7] focus on exploiting spatial-temporal correlations in a more efﬁ-cient way or on learning more accurate motion representations for videos.”
X3D: Expanding Architectures for Efficient Video Recognition (2020, unknown): “[2,5,9, 11, 13, 18, 28, 35, 38, 42, 44, 55, 56, 63, 65, 66, 72, 76, 84–86].”
Cross-Modal Knowledge Distillation for Action Recognition (2019, methodology): “In [4], a 3D convolutional neural network is initialized by transferring the knowledge of a pre-trained 2D CNN.”