TEA: Temporal Excitation and Aggregation for Action Recognition

By Yan Li, Bin Ji, Xintian Shi, Jianguo Zhang, Bin Kang, Limin Wang. 2020

566 citations recorded by the source corpus.

Temporal modeling is key for action recognition in videos. It normally considers both short-range motions and long-range aggregations. In this paper, we propose a Temporal Excitation and Aggregation (TEA) block, including a motion excitation (ME) module and a multiple temporal aggregation (MTA) module, specifically designed to capture both short- and long-range temporal evolution. In particular, for short-range motion modeling, the ME module calculates the feature-level temporal differences from spatiotemporal features. It then utilizes the differences to excite the motion-sensitive channels of the features. The long-range temporal aggregations in previous works are typically achieved by stacking a large number of local temporal convolutions. Each convolution processes a local temporal window at a time. In contrast, the MTA module proposes to deform the local convolution to a group of sub-convolutions, forming a hierarchical residual architecture. Without introducing additional parameters, the features will be processed with a series of sub-convolutions, and each frame could complete multiple temporal aggregations with neighborhoods. The final equivalent receptive field of temporal dimension is accordingly enlarged, which is capable of modeling the long-range temporal relationship over distant frames. The two components of the TEA block are complementary in temporal modeling. Finally, our approach achieves impressive results at low FLOPs on several action recognition benchmarks, such as Kinetics, Something-Something, HMDB51, and UCF101, which confirms its effectiveness and efficiency.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Is Space-Time Attention All You Need for Video Understanding? (2021, methodology): “Finally, multimodal video-text transformers (Sun et al., 2019; Li et al., 2020a) have also been trained or pretrained in unsupervised fashion by adopting masked-token pretext tasks adapted from the language domain (Devlin et al., 2018; Radford et al., 2018).”
ViViT: A Video Vision Transformer (2021, background): “However, common practice on datasets such as Kinetics [21, 75, 42] is to average results over multiple, shorter “views” of the same video clip.”
When Video Classification Meets Incremental Classes (2021, background): “Recently, many works perform temporal modeling with an extra module instead of the simple 1D temporal convolution [22, 28, 30, 32, 39].”
Res2Net: A New Multi-Scale Backbone Architecture (2019, background): “segmentation [46], salient object detection [21], interactive image segmentation [41], video recognition [37], concealed object detection [18], and medical segmentation [19], [20], [66].”
Multi-Scale Spatial Temporal Graph Convolutional Network for Skeleton-Based Action Recognition (2021, methodology): “…resources to process pixel-level information in RGB image or temporal optical ﬂow (Tran et al. 2015; Simonyan and Zis-serman 2014; Zhang et al. 2016; Li et al. 2020), skeleton “walking” (b) “hand waving” (c) “jumping up” (d) “cheer-ing up” (e) “wearing on glasses” (f) “taking off glasses” . based…”
UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learning (2022, methodology): “Recent methods propose well-designed modules to enhance the temporal modeling ability for 2D CNNs (Wang et al., 2016; Lin et al., 2019; Luo & Yuille, 2019; Jiang et al., 2019; Liu et al., 2020a; Li et al., 2020b; Kwon et al., 2020; Li et al., 2020a; 2021a; Wang et al., 2020b).”
Expanding Language-Image Pretrained Models for General Video Recognition (2022, background): “For efficiency purposes, some studies [46,36,51] factorize convolutions across spatial and temporal dimensions, while others insert the specific temporal modules into 2D CNNs [28,26,31].”
Attention Bottlenecks for Multimodal Fusion (2021, background): “[44] Yan Li, Bin Ji, Xintian Shi, Jianguo Zhang, Bin Kang, and Limin Wang.”
VidTr: Video Transformer Without Convolutions (2021, methodology): “Compared with previous SOTA compact models [34, 39], our compact VidTr achieves better or sim-Model Table 4: Quantitative analysis on Kinetics-400 dataset.”
AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognition (2021, background): “TEA (Li et al., 2020b) introduces a motion excitation module including multiple temporal aggregation modules to capture both short-and long-range temporal evolution in videos.”