Gaussian Temporal Awareness Networks for Action Localization

By Fuchen Long, Ting Yao, Zhaofan Qiu, Xinmei Tian, Jiebo Luo, Tao Mei. Computer Vision and Pattern Recognition, 2019

361 citations recorded by the source corpus.

Temporally localizing actions in a video is a fundamental challenge in video understanding. Most existing approaches have often drawn inspiration from image object detection and extended the advances, e.g., SSD and Faster R-CNN, to produce temporal locations of an action in a 1D sequence. Nevertheless, the results can suffer from robustness problem due to the design of predetermined temporal scales, which overlooks the temporal structure of an action and limits the utility on detecting actions with complex variations. In this paper, we propose to address the problem by introducing Gaussian kernels to dynamically optimize temporal scale of each action proposal. Specifically, we present Gaussian Temporal Awareness Networks (GTAN) --- a new architecture that novelly integrates the exploitation of temporal structure into an one-stage action localization framework. Technically, GTAN models the temporal structure through learning a set of Gaussian kernels, each for a cell in the feature maps. Each Gaussian kernel corresponds to a particular interval of an action proposal and a mixture of Gaussian kernels could further characterize action proposals with various length. Moreover, the values in each Gaussian curve reflect the contextual contributions to the localization of an action proposal. Extensive experiments are conducted on both THUMOS14 and ActivityNet v1.3 datasets, and superior results are reported when comparing to state-of-the-art approaches. More remarkably, GTAN achieves 1.9% and 1.1% improvements in mAP on testing set of the two datasets.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/10cb36f95b6f56b70c1f01bf8f73bbae11c9b2f1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation (2019, background): “e [32], [33], [34], [35], [36], [37], [38], [39], [40]. Other approaches combine the proposal generation and classiﬁcation in a single-stage architecture which enables end-to-end training [41], [42], [43]. 3 TEMPORAL ACTION SEGMENTATION We introduce a multi-stage temporal convolutional network (MS-T”
G-TAD: Sub-Graph Localization for Temporal Action Detection (2019, background): “Therefore, incorporating context into temporal action detection has become an important strategy to boost detection accuracy in the recent literature [11, 15, 9, 33, 44, 56, 30].”
ActionFormer: Localizing Moments of Actions with Transformers (2022, methodology): “We compare to a strong set of baselines, including both two-stage ( e . g ., G-TAD [75], BC-GNN [4], TAL-MR [84]) and single-stage ( e . g ., A2Net [77], GTAN [50], AFSD [35], TadTR [46]) methods for TAL.”