Temporal Segment Networks for Action Recognition in Videos

By Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, Luc Van Gool. 2017

969 citations recorded by the source corpus.

We present a general and flexible video-level framework for learning action models in videos. This method, called temporal segment network (TSN), aims to model long-range temporal structure with a new segment-based sampling and aggregation scheme. This unique design enables the TSN framework to efficiently learn action models by using the whole video. The learned models could be easily deployed for action recognition in both trimmed and untrimmed videos with simple average pooling and multi-scale temporal window integration, respectively. We also study a series of good practices for the implementation of the TSN framework given limited training samples. Our approach obtains the state-the-of-art performance on five challenging action recognition benchmarks: HMDB51 (71.0 percent), UCF101 (94.9 percent), THUMOS14 (80.1 percent), ActivityNet v1.2 (89.6 percent), and Kinetics400 (75.7 percent). In addition, using the proposed RGB difference as a simple motion representation, our method can still achieve competitive accuracy on UCF101 (91.0 percent) while running at 340 FPS. Furthermore, based on the proposed TSN framework, we won the video classification track at the ActivityNet challenge 2016 among 24 teams.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

Late Temporal Modeling in 3D CNN Architectures with BERT for Action Recognition (2020, neutral): “TSN [33] X ImageNet 71.”
All in One: Exploring Unified Video-Language Pre-Training (2022, background): “Sparse sampling along the temporal dimension, a type of data augmentation proposed in TSN [37], has been widely adopted to train video backbones.”
vCLIMB: A Novel Video Class Incremental Learning Benchmark (2022, methodology): “In fact, the 91.42% accuracy obtained by the naive rehearsal baseline on the UCF101 10- task CIL is almost on par with the 94.9% accuracy reported for training TSN [35], which is the the backbone in our experiments, on all UCF101 classes simultaneously.”
Cross-Modal and Hierarchical Modeling of Video and Text (2018, background): “Note tsn [39] is fully supervised thus not directly comparable.”
MIST: Multiple Instance Self-Training Framework for Video Anomaly Detection (2021, methodology): “2D Feature Encoder vs 3D Feature Encoder We also conduct experiment on 2D feature encoder the RGB branch of TSN [10] but fail.”
Graph Convolutional Label Noise Cleaner: Train a Plug-And-Play Action Classifier for Anomaly Detection (2019, methodology): “The action classifiers are both implemented upon the Caffe [25] platform with the same settings of video sampling and data augment as [62].”
HMS: Hierarchical Modality Selection for Efficient Video Recognition (2021, methodology): “Two-stream ConvNets [33] use pre-computed optical flow images as extra inputs to deep networks, which then becomes a common paradigm for video recognition [6, 18, 41, 56] especially in approaches using 2D networks.”
Dynamic Sampling Networks for Efficient Action Recognition in Videos (2020, methodology): “Meanwhile, several works [5], [29]–[33] attempted to uti-”
Omni-sourced Webly-supervised Learning for Video Recognition (2020, methodology): “2D TSN Different from the original setting in [46], we choose ResNet-50 [14] to be the backbone, unless otherwise speciﬁed.”
Why Can't I Dance in the Mall? Learning to Mitigate Scene Bias in Action Recognition (2019, background): “Recent advances in this field focus on capturing longer-range temporal dependencies [55, 56, 63].”
Exploring Video Quality Assessment on User Generated Contents from Aesthetic and Technical Perspectives (2022, background): “5(b)) through spatial downsampling [63] and temporal sparse frame sampling [64] which preserves the semantics and composition of original videos.”
ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning for Action Recognition (2022, methodology): “As a key component, the model architectures adopted by existing video methods has expanded from CNNs [32, 68, 19, 18, 77, 69, 72, 48, 41, 45] to Transformers [17, 40, 38, 44, 2, 6].”
SimAM: A Simple, Parameter-Free Attention Module for Convolutional Neural Networks (2021, background): “…convolution (Wang et al., 2020), adding spatial-wise attention (Woo et al., 2018), incorporating long-range dependencies (Cao et al., 2020; Wang et al., 2018b), unifying attention and normalization process (Li et al., 2019a), or utilizing style cues of that features (Lee et al., 2019).”