Revisiting the Effectiveness of Off-the-shelf Temporal Modeling Approaches for Large-scale Video Classification

By Yunlong Bian, Chuang Gan, Xiao Liu, Fu Li, Xiang Long, Yandong Li, Heng Qi, Jie Zhou, Shilei Wen, Yuanqing Lin. 2017

49 citations recorded by the source corpus.

This paper describes our solution for the video recognition task of ActivityNet Kinetics challenge that ranked the 1st place. Most of existing state-of-the-art video recognition approaches are in favor of an end-to-end pipeline. One exception is the framework of DevNet. The merit of DevNet is that they first use the video data to learn a network (i.e. fine-tuning or training from scratch). Instead of directly using the end-to-end classification scores (e.g. softmax scores), they extract the features from the learned network and then fed them into the off-the-shelf machine learning models to conduct video classification. However, the effectiveness of this line work has long-term been ignored and underestimated. In this submission, we extensively use this strategy. Particularly, we investigate four temporal modeling approaches using the learned features: Multi-group Shifting Attention Network, Temporal Xception Network, Multi-stream sequence Model and Fast-Forward Sequence Model. Experiment results on the challenging Kinetics dataset demonstrate that our proposed temporal modeling approaches can significantly improve existing approaches in the large-scale video recognition tasks. Most remarkably, our best single Multi-group Shifting Attention Network can achieve 77.7% in term of top-1 accuracy and 93.2% in term of top-5 accuracy on the validation set.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Billion-scale semi-supervised learning for image classification (2019, neutral): “3 3 stream SATT [44] RGB+flow+audio 77.”
Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification (2017, background): “We note that even though we focus on the speed-accuracy trade-offs in action classification network design, the performance is competitive compared with recent Kinetics Challenge winners and concurrent works; notably [41] and [42] use heavier backbone architectures (e.”
Large-Scale Weakly-Supervised Pre-Training for Video Action Recognition (2019, methodology): “9 RGB + flow 3-stream SATT [69]; ImageNet 77.”
MoViNets: Mobile Video Networks for Efficient Video Recognition (2021, background): “This has a number of advantages, including the ability to reduce a subclip xclipi down to a single frame without affecting activations or predictions, and enables 3D CNNs to work on streaming video for online inference.”
Audiovisual SlowFast Networks for Video Recognition (2020, methodology): “When comparing to other methods that also utilize audio [78, 8], despite building upon a stronger baseline (our SlowFast baseline is as good as GBlend’s [78] ﬁnal AV model: 77.9% vs .”
Attention Bottlenecks for Multimodal Fusion (2021, unknown): “This gap is widened, however, for the [34] 73.7 91.6 - - TEA [44] 76.1 92.5 - - TS S3D-G [65] 77.2 93.0 - - 3-stream SATT [9] 77.7 93.2 - - AVSlowFast, R101 [64] 78.8 93.6 85.0 † - LGD-3D R101 [52] 79.4 94.4 - - SlowFast R101-NL [22] 79.8 93.9 - - ViViT-Base [6] 80 [35] and Kinetics Sound [4].”
Rethinking Spatiotemporal Feature Learning For Video Understanding (2017, unknown): “Shifting Attention Net [1] RGB+Flow+Audio Inception-ResNet-v2 77.”
What Makes Training Multi-Modal Classification Networks Hard? (2019, methodology): “Second, G-Blend outperforms state-of-the-arts multi-modal base-line Shift-Attention Network [10] by 1.4% while using less modalities (not using optical ﬂow) and no pre-training.”
Geometry Guided Convolutional Neural Networks for Self-Supervised Video Representation Learning (2018, background): “To recognize actions and events happening in videos, recent approaches that employ deep convolutional neural networks (CNNs) [12, 17, 31, 34, 35], recurrent networks [15, 33, 4], and attention networks [23, 22] have achieved state-of-the-art results.”
Exploiting Spatial-Temporal Modelling and Multi-Modal Fusion for Human Action Recognition (2018, unknown): “We improve our formerly proposed temporal Xception network [12] and enable it to integrate both early-fusion and later-fusion features of multi-modal information.”
Trusted Multi-View Classification With Dynamic Evidential Fusion (2022, methodology): “used in various applications [47], [48], [49], [50].”
Learning Spatio-Temporal Representation With Local and Global Diffusion (2019, background): “This accuracy is also higher than 2D CNN with a deeper backbone reported by the ActivityNet 2017 challenge winner [1].”