TAM: Temporal Adaptive Module for Video Recognition

By Zhaoyang Liu, Limin Wang, Wayne Wu, Chen Qian, Tong Lu. IEEE International Conference on Computer Vision, 2020

374 citations recorded by the source corpus.

Video data is with complex temporal dynamics due to various factors such as camera motion, speed variation, and different activities. To effectively capture this diverse motion pattern, this paper presents a new temporal adaptive module (TAM) to generate video-specific temporal kernels based on its own feature map. TAM proposes a unique two-level adaptive modeling scheme by decoupling the dynamic kernel into a location sensitive importance map and a location invariant aggregation weight. The importance map is learned in a local temporal window to capture short-term information, while the aggregation weight is generated from a global view with a focus on long-term structure. TAM is a modular block and could be integrated into 2D CNNs to yield a powerful video architecture (TANet) with a very small extra computational cost. The extensive experiments on Kinetics-400 and Something-Something datasets demonstrate that our TAM outperforms other temporal modeling methods consistently, and achieves the state-of-the-art performance under the similar complexity. The code is available at https://github.com/liu-zhy/temporal-adaptive-module.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/7f31f46193b0b02e19ecdb8e20c9221659ea79b2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

TDN: Temporal Difference Networks for Efficient Action Recognition (2020, methodology): “In addition, we try to use RGB difference representation to learn the channel attention weight just as TEINet [21], and its performance is also worse than our proposed S-TDM (47.”
Expanding Language-Image Pretrained Models for General Video Recognition (2022, background): “For efficiency purposes, some studies [46,36,51] factorize convolutions across spatial and temporal dimensions, while others insert the specific temporal modules into 2D CNNs [28,26,31].”
VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking (2023, methodology): “Since the introduction of Kinetics benchmarks [28], some supervised pre-trained models on it have been transferred to small-scale datasets for action recognition, such as 2D CNNs (TSN [71], TSM [40], TANet [48], TDN [70]), 3D CNNs (I3D [7], R(2+1)D [64], ARTNet [69], SlowFast [19]), Transformer (Tim”
ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning for Action Recognition (2022, methodology): “As a key component, the model architectures adopted by existing video methods has expanded from CNNs [32, 68, 19, 18, 77, 69, 72, 48, 41, 45] to Transformers [17, 40, 38, 44, 2, 6].”
MoViNets: Mobile Video Networks for Efficient Video Recognition (2021, background): “to improve 2D CNNs’ accuracy using temporal aggregation [40, 17, 70, 43, 16], however their limited inter-frame interactions reduce these models’ abilities to adequately model long-range temporal dependencies like 3D CNNs.”
Attention mechanisms in computer vision: A survey (2021, background): “Combine local attention and global attention TAM [173] Arxiv2020 Action a) local: global spatial average pooling → 1D Convs, b) global: global spatial average pooling → MLP → adaptive convolution (A) (0, 1) S (II), (III)”