G-TAD: Sub-Graph Localization for Temporal Action Detection

By Mengmeng Xu, Chen Zhao, D. Rojas, Ali K. Thabet, Bernard Ghanem. 2019

506 citations recorded by the source corpus.

Temporal action detection is a fundamental yet challenging task in video understanding. Video context is a critical cue to effectively detect actions, but current works mainly focus on temporal context, while neglecting semantic context as well as other important context properties. In this work, we propose a graph convolutional network (GCN) model to adaptively incorporate multi-level semantic context into video features and cast temporal action detection as a sub-graph localization problem. Specifically, we formulate video snippets as graph nodes, snippet-snippet correlations as edges, and actions associated with context as target sub-graphs. With graph convolution as the basic operation, we design a GCN block called GCNeXt, which learns the features of each node by aggregating its context and dynamically updates the edges in the graph. To localize each sub-graph, we also design an SGAlign layer to embed each sub-graph into the Euclidean space. Extensive experiments show that G-TAD is capable of finding effective video context without extra supervision and achieves state-of-the-art performance on two detection benchmarks. On ActivityNet-1.3 it obtains an average mAP of 34.09%; on THUMOS14 it reaches 51.6% at IoU@0.5 when combined with a proposal processing method. The code has been made available at https://github.com/frostinassiky/gtad.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Graph Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Graph Learning

Citations in the corpus

Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “[229] Mengmeng Xu, Chen Zhao, David S Rojas, Ali Thabet, and Bernard Ghanem.”
Self-Supervised Learning by Cross-Modal Audio-Video Clustering (2019, methodology): “In particular, we employ the recent G-TAD [80] action localization algorithm, where we replace the clip features (originally extracted from a TSN [73] model pretrained on Kinetics) with our XDC features from the R(2+1)D-18 model pretrained on IG-Kinetics or IG-Random.”
Video Self-Stitching Graph Network for Temporal Action Localization (2020, methodology): “G-TAD [42] breaks the restriction of the temporal location of video snippets and uses a graph to aggregate features from snippets not located in a temporal neighborhood.”
ActionFormer: Localizing Moments of Actions with Transformers (2022, methodology): “Most previous works have considered using action proposals [36] or anchor windows [50], and developed convolutional [55, 85], recurrent [7], and graph [4, 76] neural networks for TAL.”