Graph Convolutional Networks for Temporal Action Localization

By Runhao Zeng, Wenbing Huang, Mingkui Tan, Yu Rong, Peilin Zhao, Junzhou Huang, Chuang Gan. IEEE International Conference on Computer Vision, 2019

542 citations recorded by the source corpus.

Most state-of-the-art action localization systems process each action proposal individually, without explicitly exploiting their relations during learning. However, the relations between proposals actually play an important role in action localization, since a meaningful action always consists of multiple proposals in a video. In this paper, we propose to exploit the proposal-proposal relations using GraphConvolutional Networks (GCNs). First, we construct an action proposal graph, where each proposal is represented as a node and their relations between two proposals as an edge. Here, we use two types of relations, one for capturing the context information for each proposal and the other one for characterizing the correlations between distinct actions. Then we apply the GCNs over the graph to model the relations among different proposals and learn powerful representations for the action classification and localization. Experimental results show that our approach significantly outperforms the state-of-the-art on THUMOS14(49.1% versus 42.8%). Moreover, augmentation experiments on ActivityNet also verify the efficacy of modeling action proposal relationships.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/fd8e725159159ca2169d302f6cb510e3b1cc1a4b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Dense Regression Network for Video Grounding (2020, background): “This task has drawn increasing attention over the past few years due to its vast potential applications in video understanding [38, 3, 45, 44, 6], video retrieval [42, 8], and human-computer interaction [35, 20, 50], etc.”
Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervision (2020, methodology): “ural networks (GCNs) [20,38] over the graph to model relations among dierent nodes and learn powerful representations for computer vision. For instance, GCN are used for temporal action localization [49], video classication [36,40], anomaly detection [50], skeleton-based action recognition [32,44], ”
VideoComposer: Compositional Video Synthesis with Motion Controllability (2023, background): “Motion cues play a crucial role in video understanding ﬁelds, such as action recognition [2, 5, 7, 39, 55, 57, 58], action detection [10, 62, 70, 74], human video generation [35, 37, 61], etc .”
Learning Salient Boundary Feature for Anchor-free Temporal Action Localization (2021, methodology): “(2) Actionness-guided methods can solely provide predictions of temporal boundary, while they have to rely on the extra model such as S-CNN [33] and P-GCN [41] for classiﬁcation.”
ActionFormer: Localizing Moments of Actions with Transformers (2022, methodology): “More recent effort investigates the modeling of temporal context among proposals using graph neural networks [76, 80, 83] or attention and self-attention mechanisms [51, 58, 88].”
NAT: Neural Architecture Transformer for Accurate and Compact Architectures (2019, background): “…networks (DNNs) [25] have been producing state-of-the-art results in many challenging tasks including image classiﬁcation [12, 23, 43, 60, 61, 18, 11, 56], face recognition [39, 44, 59], brain signal processing [34, 35], video analysis [48, 53, 52] and many other areas [58, 57, 24, 3, 10, 9, 4].”
End-to-End Temporal Action Detection With Transformer (2021, methodology): “Those proposal generation methods (BSN, MGG, BMN, G-TAD, MR, BC-GNN, and RTD-Net) are not self-contained, as they rely on an extra classiﬁer (such as P-GCN) to accomplish the TAD task.”
G-TAD: Sub-Graph Localization for Temporal Action Detection (2019, methodology): “[55] consider temporal action proposals as nodes in a graph, and refine their boundaries and classification scores based on the established proposal-proposal dependencies.”
Video Self-Stitching Graph Network for Temporal Action Localization (2020, methodology): “It is shown in the literature [9, 44] that TSN [40] surpasses C3D [34] features, and I3D [6] consistently outperforms TSN with different proposal methods.”
MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation (2019, background): “The first stage is to generate proposals, and then classify and refine the boundaries of these proposals in the second stage [32], [33], [34], [35], [36], [37], [38], [39], [40].”
Location-Aware Graph Convolutional Networks for Video Question Answering (2020, background): “In video QA task, understanding untrimmed videos (Zeng et al. 2019a; Wu et al. 2019) is important.”
Breaking the Curse of Space Explosion: Towards Efficient NAS with Curriculum Search (2020, methodology): “…the candidate subspace, CNAS performs much accurate architecture sampling, thus can significantly improve the search efficiency and find better architectures than standard NAS methods. et al., 2017; Chen et al., 2019; Zeng et al., 2019; Hosseini et al., 2020; Guo et al., 2020b; Cao et al., 2018).”
Local-Global Video-Text Interactions for Temporal Grounding (2020, methodology): “Second, proposal-based techniques [27, 31, 36, 38] extract all action proposals and refine their boundaries for action detection.”