Temporal Segment Networks: Towards Good Practices for Deep Action Recognition

By Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, Luc Van Gool. 2016

4,303 citations recorded by the source corpus.

Deep convolutional networks have achieved great success for visual recognition in still images. However, for action recognition in videos, the advantage over traditional methods is not so evident. This paper aims to discover the principles to design effective ConvNet architectures for action recognition in videos and learn these models given limited training samples. Our first contribution is temporal segment network (TSN), a novel framework for video-based action recognition. which is based on the idea of long-range temporal structure modeling. It combines a sparse temporal sampling strategy and video-level supervision to enable efficient and effective learning using the whole action video. The other contribution is our study on a series of good practices in learning ConvNets on video data with the help of temporal segment network. Our approach obtains the state-the-of-art performance on the datasets of HMDB51 (\( 69.4\,\% \)) and UCF101 (\( 94.2\,\% \)). We also visualize the learned ConvNet models, which qualitatively demonstrates the effectiveness of temporal segment network and the proposed good practices (Models and code at https://github.com/yjxiong/temporal-segment-networks).

Subfields: Computer Vision and Pattern Recognition, Endocrinology, Diabetes and Metabolism

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Endocrinology, Diabetes and Metabolism

Citations in the corpus

Learning Energy-Based Spatial-Temporal Generative ConvNets for Dynamic Patterns (2019, background): “, [3], [4], [5], [6], [7], [8], [9], and [10], however, the progress in developing generative models of dynamic patterns for synthesis purpose has been lagging behind.”
Towards Automatic Learning of Procedures From Web Instructional Videos (2017, background): “Action understanding remains an intensely studied problem-space, e.g., action recognition (Donahue et al. 2015; Wang et al. 2016), action detection (Singh et al. 2016; Yeung et al. 2016; Shou, Wang, and Chang 2016) and action labeling (Kuehne, Richard, and Gall 2017; Huang, Fei-Fei, and Niebles…”
Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “The task is similar to action recognition (e.g., [100,110,139,221,243]) because in some cases a specific action can correspond to a specific state change.”
Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, methodology): “Both are trained end-toend using a TSN backbone [66] and results can be seen in Table 5.”
Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification (2017, background): “Since then, many video classiﬁcation methods follow the same multi-stream 2D CNN design, and have made improvements in terms of new representations [25,26], different backbone architecture [27,28,29,17], fusion of the streams [30,31,32,33] and exploiting richer temporal structures [34,35,36].”
TVQA: Localized, Compositional Video Question Answering (2018, methodology): “When multiple streams are used, we simply sum up the scores from each stream as the final score (Wang et al., 2016a).”
Less is More: CLIPBERT for Video-and-Language Learning via Sparse Sampling (2021, methodology): “Standard approaches [23, 74, 17, 80, 33, 12, 31, 32] leverage offline extracted video and text features from action recognition models [26, 66, 4, 73], image recognition models [10, 21], and language models [47, 50, 11, 42].”
Localizing Moments in Video with Natural Language (2017, methodology): “Likewise, we extract optical flow features from the penultimate layer from a competitive activity recognition model [46].”
SlowFast Networks for Video Recognition (2018, methodology): “This method has been a foundation of many competitive results in the literature [12, 13, 55].”
UvA-DARE (Digital Academic Repository) Self-Supervised Video Representation Learning With Odd-One-Out Networks (positive): “Stack of differences of frames video-clip encoder: Inspired by [37, 43], we also stack the difference of frames instead of summing them.”
Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognition (2017, methodology): “Many methods based on the two-stream CNNs are proposed to improve action recognition performance [5, 6, 20, 22].”
Video Action Recognition Via Neural Architecture Searching (2019, methodology): “Other architectures, such as I3D [22] and TSN, which are either two-stream methods with extra input modality or pre-trained on another larger datasets, are not taken into consideration, owing to the limitation of computational resources.”
AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures (2019, neutral): “68 TSN (Wang et al., 2016) RGB 24.11 49.10 Ioffe & Szegedy (2015) Flow 11.”
Class-Incremental Learning for Action Recognition in Videos (2021, positive): “With the great success of deep learning, various action recognition methods based on convolutional neural networks have been studied [2, 17, 21, 36, 40].”
Video Representation Learning Using Discriminative Pooling (2018, methodology): “HMDB-51 (accuracy over 3 splits) Method Accuracy Temporal segment networks[54] 69.”
RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, background): “[147] devised a temporal segment network (TSN) that uses a sparse sampling scheme to extract short snippets over a long video sequence.”
Chained Multi-stream Networks Exploiting Pose, Motion, and Appearance for Action Classification and Detection (2017, methodology): “The flow and magnitude values in the image are multiplied by 16 and quantized into the [0,255] interval [20, 31, 46, 47].”
PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understanding (2017, background): “In practice, we adopt features derived from convolution networks of TSN network that have been trained for action recognition as a robust RGB-based deep network feature.”