Grouped Spatial-Temporal Aggregation for Efficient Action Recognition

By Chenxu Luo, Alan L. Yuille. 2019

172 citations recorded by the source corpus.

Temporal reasoning is an important aspect of video analysis. 3D CNN shows good performance by exploring spatial-temporal features jointly in an unconstrained way, but it also increases the computational cost a lot. Previous works try to reduce the complexity by decoupling the spatial and temporal filters. In this paper, we propose a novel decomposition method that decomposes the feature channels into spatial and temporal groups in parallel. This decomposition can make two groups focus on static and dynamic cues separately. We call this grouped spatial-temporal aggregation (GST). This decomposition is more parameter-efficient and enables us to quantitatively analyze the contributions of spatial and temporal features in different layers. We verify our model on several action recognition tasks that require temporal reasoning and show its effectiveness.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval (2021, methodology): “During training, we sample a single frame uniformly from each segment (in a similar manner to TSN [67] and GST [39]).”
UniFormer: Unifying Convolution and Self-Attention for Visual Recognition (2022, background): “Additionally, other studies [41], [49], [50], [52], [54] focus on enhancing the temporal modeling ability for 2D CNNs via well-designed plug-and-play modules, such as temporal shift [54], [62], motion enhancement [41], [52], [58] and spatiotemporal excitation [49], [50].”
In the Eye of the Beholder: Gaze and Actions in First Person Video (2020, background): “Ourdataset is publicly available(1) andhas been considered by several recentworks [22], [23], [24], [25], [26], [27].”
Only Time Can Tell: Discovering Temporal Data for Temporal Modeling (2019, background): “Temporal convolution methods [35, 36, 38, 25, 14] extend the success of 2D convolutions in image understanding to include convolutions across frames that are adjacent in time.”
UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learning (2022, methodology): “Recent methods propose well-designed modules to enhance the temporal modeling ability for 2D CNNs (Wang et al., 2016; Lin et al., 2019; Luo & Yuille, 2019; Jiang et al., 2019; Liu et al., 2020a; Li et al., 2020b; Kwon et al., 2020; Li et al., 2020a; 2021a; Wang et al., 2020b).”