Weakly Supervised Action Localization by Sparse Temporal Pooling Network

By Phuc Xuan Nguyen, Ting Liu, Gautam Prasad, Bohyung Han. 2017

380 citations recorded by the source corpus.

We propose a weakly supervised temporal action localization algorithm on untrimmed videos using convolutional neural networks. Our algorithm learns from video-level class labels and predicts temporal intervals of human actions with no requirement of temporal localization annotations. We design our network to identify a sparse subset of key segments associated with target actions in a video using an attention module and fuse the key segments through adaptive temporal pooling. Our loss function is comprised of two terms that minimize the video-level action classification error and enforce the sparsity of the segment selection. At inference time, we extract and score temporal proposals using temporal class activations and class-agnostic attentions to estimate the time intervals that correspond to target actions. The proposed algorithm attains state-of-the-art results on the THUMOS14 dataset and outstanding performance on ActivityNet1.3 even with its weak supervision.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Endocrinology, Diabetes and Metabolism

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Endocrinology, Diabetes and Metabolism

Citations in the corpus

Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, methodology): “ervision. Contrary to Sec. 4.1, this challenge does not use action start/end times during training. Previous weakly-supervised approaches utilised video-level or transcript supervision, where the set [64,65,66,67,68,69] or sequence [70,71,72,73,74,75] of actions in the video are used in training, wi”
3C-Net: Category Count and Center Loss for Weakly-Supervised Action Localization (2019, methodology): “Both UntrimmedNets [28] and Autoloc [22] use TSN [29] as the backbone, whereas STPN [14] and W-TALC [16] use I3D networks similar to our framework.”
Completeness Modeling and Context Separation for Weakly Supervised Temporal Action Localization (2019, methodology): “[45] proposed a framework called UntrimmedNet composed of a classification module and a selection module, based on which a sparsity regularization was later introduced in [33].”
MIST: Multiple Instance Self-Training Framework for Video Anomaly Detection (2021, methodology): “In video-related tasks, MIL takes a video as a bag and clips in the video as instances [23, 19, 8].”
HACS: Human Action Clips and Segments Dataset for Recognition and Temporal Localization (2017, background): “Moreover, videos in HACS Segments have both sparse clip annotation, which is a weak form of supervision for localization [57, 49, 40], and dense segment annotation, which can serve as the ground-truth of localization.”
Action Recognition From Single Timestamp Supervision in Untrimmed Videos (2019, methodology): “We choose Untrimmed Net [32] amongst the aforementioned works [32, 22, 28, 23], which is used to extract features in [23] and is the backbone model of [28], due to the availability of published code.”
W-TALC: Weakly-supervised Temporal Activity Localization and Classification (2018, methodology): “As in literature [57,35], we use the training videos to train our network, and the validation set to test.”
Weakly Supervised Object Localization and Detection: A Survey (2021, neutral): “On the other hand, [105], [129], [133], [162] develop methods to localize temporal actions in the given untrimmed videos, where the main goal is to predict the temporal boundary of each action instance contained in the weakly labeled training videos.”
Temporal Action Detection with Structured Segment Networks (2017, background): “2017), STPN (Nguyen et al. 2018), and AutoLoc (Shou et al.”