TDN: Temporal Difference Networks for Efficient Action Recognition

By Limin Wang, Zhan Tong, Gangshan Wu. Computer Vision and Pattern Recognition, 2020

493 citations recorded by the source corpus.

Temporal modeling still remains challenging for action recognition in videos. To mitigate this issue, this paper presents a new video architecture, termed as Temporal Difference Network (TDN), with a focus on capturing multi-scale temporal information for efficient action recognition. The core of our TDN is to devise an efficient temporal module (TDM) by explicitly leveraging a temporal difference operator, and systematically assess its effect on short-term and long-term motion modeling. To fully capture temporal information over the entire video, our TDN is established with a two-level difference modeling paradigm. Specifically, for local motion modeling, temporal difference over consecutive frames is used to supply 2D CNNs with finer motion pattern, while for global motion modeling, temporal difference across segments is incorporated to capture long-range structure for motion feature excitation. TDN provides a simple and principled temporal modeling framework and could be instantiated with the existing CNNs at a small extra computational cost. Our TDN presents a new state of the art on the Something-Something V1 & V2 datasets and is on par with the best performance on the Kinetics-400 dataset. In addition, we conduct in-depth ablation studies and plot the visualization results of our TDN, hopefully providing insightful analysis on temporal difference modeling. We release the code at https://github.com/MCG-NJU/TDN.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/2ad2981a53393dc5987419a22cbe1ee3d7fa6e42

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Yu Qiao
2. Jun Liu
3. Hongsheng Li
4. Luc Van Gool
5. Xiaogang Wang
6. Dacheng Tao
7. Hossein Rahmani
8. Qiuhong Ke

In Control and Systems Engineering

1. Yali Wang
2. Yu Qiao
3. Limin Wang
4. Jun Liu
5. Qiuhong Ke
6. Hossein Rahmani
7. Zehua Sun
8. Wayne Wu

Citations in the corpus

• VideoComposer: Compositional Video Synthesis with Motion Controllability (2023, background): “Motion cues play a crucial role in video understanding fields, such as action recognition [2, 5, 7, 39, 55, 57, 58], action detection [10, 62, 70, 74], human video generation [35, 37, 61], etc .”
• UniFormer: Unifying Convolution and Self-Attention for Visual Recognition (2022, unknown): “In fact, our best model achieves the new state-of-the-art results: 61.0% top-1 accuracy on SthSth V1 (4.2% higher than TDNEN ) [96] and 71.2% top-1 accuracy on SthSth V2 (1.6% higher than Swin-B [61]).”
• Human Action Recognition From Various Data Modalities: A Review (2020, methodology): “[136] designed a two-level Temporal Difference Module (TDM) to capture both finer local and long-range global motion information.”
• UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learning (2022, methodology): “Other prior works (Tran et al., 2018; Qiu et al., 2017; Tran et al., 2019; Feichtenhofer, 2020; Wang et al., 2020a) try to factorize 3D convolution kernel in different dimensions to reduce complexity.”
• TAM: Temporal Adaptive Module for Video Recognition (2020, methodology): “Since the deep learning method has been wildly used in the image tasks, there are many attempts [18, 31, 40, 46, 10, 23, 39] based on 2D CNNs devoted to modeling the video clips.”