Rethinking the Faster R-CNN Architecture for Temporal Action Localization

By Yu-Wei Chao, Sudheendra Vijayanarasimhan, Bryan Seybold, David A. Ross, Jia Deng, Rahul Sukthankar. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018

688 citations recorded by the source corpus.

We propose TAL-Net, an improved approach to temporal action localization in video that is inspired by the Faster RCNN object detection framework. TAL-Net addresses three key shortcomings of existing approaches: (1) we improve receptive field alignment using a multi-scale architecture that can accommodate extreme variation in action durations; (2) we better exploit the temporal context of actions for both proposal generation and action classification by appropriately extending receptive fields; and (3) we explicitly consider multi-stream feature fusion and demonstrate that fusing motion late is important. We achieve state-of-the-art performance for both action proposal and localization on THUMOS'14 detection benchmark and competitive performance on ActivityNet challenge.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/e85c01ff4979357b428538c9f224fa4259541c1a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Graph Convolutional Networks for Temporal Action Localization (2019, methodology): “As already demonstrated in [8, 6], exploring the contextual information is of great help in refining the detection boundary and increasing the detection accuracy eventually.”
Human Action Recognition and Prediction: A Survey (2018, methodology): “The idea of their action proposals inspired many later research [49, 233,277,331,298,62,22].”
MAN: Moment Alignment Network for Natural Language Moment Retrieval via Iterative Graph Adjustment (2018, background): “To better localize temporal boundaries, a large body of work incorporated deep networks into the detection framework and obtained improved performance [5, 28, 10, 41, 2, 61, 42, 53, 59].”
Completeness Modeling and Context Separation for Weakly Supervised Temporal Action Localization (2019, background): “Fully supervised temporal action localization has witnessed remarkable progress recently [39, 48, 15, 9, 37, 51, 47, 8, 2, 31].”
Counting Out Time: Class Agnostic Video Repetition Counting in the Wild (2020, background): “Repetition counting and periodicity detection are temporally fine-grained tasks like temporal action localization [8, 38], per-frame phase clas-”
Video Self-Stitching Graph Network for Temporal Action Localization (2020, methodology): “The works R-C3D [41], TAL-NET [7], PBRNet [23], belonging to this category, perform pooling / strided convolution to obtain multi-scale features.”
Why Can't I Dance in the Mall? Learning to Mitigate Scene Bias in Action Recognition (2019, neutral): “TAL-Net [9] RGB+Flow I3D 59.”
Video Transformer Network (2021, methodology): “The idea of inflating 2D ConvNets into their 3D counterpart (I3D) was introduced in [3].”
Temporal Action Detection with Structured Segment Networks (2017, background): “TAL-Net (Chao et al. 2018) revisits this two-stage architecture for temporal action detection and proposes multi-tower dilated temporal convolutions to handle actions with varying durations.”
Weakly-Supervised Action Localization With Background Modeling (2019, background): “Interestingly, fullysupervised systems for both objects [22] and actions [4] tend to build explicit models (or classifiers) for background patches and background frames, but this type of reasoning is absent in most weakly-supervised systems.”
Risky Action Recognition in Lane Change Video Clips using Deep Spatiotemporal Networks with Segmentation Mask Transfer (2019, positive): “Closely related to this work, features extracted from deep CNNs have been used as input to temporal networks [23], [24].”
Prompting Visual-Language Models for Efficient Video Understanding (2021, methodology): “In general, there are two popular detection paradigms: the two-stage paradigm [11, 35, 45, 47, 66, 70, 74, 78, 89] first localises class-agnostic action proposals, which covers correct segments with high recall, then classifies and refines each proposal.”
Self-Supervised Spatio-Temporal Representation Learning for Videos by Predicting Motion and Appearance Statistics (2019, background): “Learning powerful spatio-temporal representations is the most fundamental deep learning problem for many video understanding tasks such as action recognition [4, 17, 26], action proposal and localization [5, 33, 34], video captioning [40, 42], etc.”
Central Similarity Quantization for Efficient Image and Video Retrieval (2019, methodology): “Compared with image analysis, video analysis aims to utilize the temporal information [33, 6, 39, 36, 3, 47, 46].”
HACS: Human Action Clips and Segments Dataset for Recognition and Temporal Localization (2017, background): “These datasets have substantially advanced the progress of research on action proposal generation [17, 18, 5, 35] and action localization [59, 48, 61, 34, 9, 2, 4].”
Self-supervised Video Representation Learning by Pace Prediction (2020, background): “Video understanding, especially action recognition, has been extensively studied for decades, where video representation learning serves as the fundamental problem of other video-related tasks, such as complex action recognition [23], action temporal localization [8, 41, 42], video caption [49, 52],”
What Do I Annotate Next? An Empirical Study of Active Learning for Action Localization (2018, positive): “More recently, end-to-end approaches have boosted the performance of stage-wise methods, demonstrating the importance of jointly optimizing classifiers and feature extractors [13,71,75,77].”
3C-Net: Category Count and Center Loss for Weakly-Supervised Action Localization (2019, background): “Most existing action localization approaches are based on strong supervision [15, 5, 21, 33, 23, 31], requiring manually annotated ground-truth temporal boundaries of actions during training.”