TSM: Temporal Shift Module for Efficient Video Understanding

By Ji Lin, Chuang Gan, Song Han. 2018

2,128 citations recorded by the source corpus.

The explosive growth in video streaming gives rise to challenges on performing video understanding at high accuracy and low computation cost. Conventional 2D CNNs are computationally cheap but cannot capture temporal relationships; 3D CNN based methods can achieve good performance but are computationally intensive, making it expensive to deploy. In this paper, we propose a generic and effective Temporal Shift Module (TSM) that enjoys both high efficiency and high performance. Specifically, it can achieve the performance of 3D CNN but maintain 2D CNN’s complexity. TSM shifts part of the channels along the temporal dimension; thus facilitate information exchanged among neighboring frames. It can be inserted into 2D CNNs to achieve temporal modeling at zero computation and zero parameters. We also extended TSM to online setting, which enables real-time low-latency online video recognition and video object detection. TSM is accurate and efficient: it ranks the first place on the Something-Something leaderboard upon publication; on Jetson Nano and Galaxy Note8, it achieves a low latency of 13ms and 35ms for online video recognition. The code is available at: https://github. com/mit-han-lab/temporal-shift-module.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Late Temporal Modeling in 3D CNN Architectures with BERT for Action Recognition (2020, neutral): “TSM [18] Kinetics-400 73.”
Imitating Interactive Intelligence (2020, methodology): “For the vision stack, we had a hyperparameter controlling whether to use a standard ResNet-50 (He et al., 2016) or a TSM ResNet, which adds a temporal shift module inside the residual block (Lin et al., 2019).”
NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions (2021, background): “Recognizing the objects “toddler”, “lady” and describing the independent action contents like “a toddler is crying” and “a lady picks the toddler up” in a video are now possible with advanced neural network models [14, 28, 63].”
Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, background): “89 TSM [50] 68.”
Class-Incremental Learning for Action Recognition in Videos (2021, positive): “With the great success of deep learning, various action recognition methods based on convolutional neural networks have been studied [2, 17, 21, 36, 40].”
Is Space-Time Attention All You Need for Video Understanding? (2021, background): “We note that whereas most previous Method Top-1 Top-5 TFLOPs R(2+1)D (Tran et al., 2018) 72.0 90.0 17.5 bLVNet (Fan et al., 2019) 73.5 91.2 0.84 TSM (Lin et al., 2019) 74.7 N/A N/A S3D-G (Xie et al., 2018) 74.7 93.4 N/A Oct-I3D+NL (Chen et al., 2019) 75.7 N/A 0.84 D3D (Stroud et al., 2020) 75.9 N/A…”
ViViT: A Video Vision Transformer (2021, methodology): “In addition, our model obtains substantial improvements for Top-1 accuracy of “noun” classes, and the only method which achieves higher “verb” accuracy used optical ﬂow as an additional input modality [43, 50].”
Action Genome: Actions As Compositions of Spatio-Temporal Scene Graphs (2019, background): “To assist more complicated action understanding, another growing set of research explores structural information in videos including temporal ordering [51, 88], object localization [4, 25, 32, 53, 74, 76], and implicit interactions between objects [4, 53].”
When Video Classification Meets Incremental Classes (2021, methodology): “Thus, we choose TSM-ResNet50 [26] which extracts 3D features to model the spatial and temporal information as our backbone.”
MoViNets: Mobile Video Networks for Efficient Video Recognition (2021, methodology): “Improving the efficiency of video models has gained increased attention [19, 64, 20, 18, 40, 17, 3, 11, 37, 48].”
FineGym: A Hierarchical Video Dataset for Fine-Grained Action Understanding (2020, methodology): “TSM [30] RGB 34.”
Cross-Domain First Person Audio-Visual Action Recognition through Relative Norm Alignment (2021, background): “Since egocentric action recognition suffers from the motion of the camera and sudden changes of view, the main approaches proposed so far are based on multistream architectures [7, 46, 39, 34, 8, 27, 38], many of which are inherited from the third-person action recognition literature.”
Listen to Look: Action Recognition by Previewing Audio (2019, background): “Some work balances the accuracy-efﬁciency trade-off by using compressed video representations [74, 54] or designing efﬁcient network architectures [78, 89, 11, 62, 39].”