UntrimmedNets for Weakly Supervised Action Recognition and Detection

By Limin Wang, Yuanjun Xiong, Dahua Lin, Luc Van Gool. 2017

522 citations recorded by the source corpus.

Current action recognition methods heavily rely on trimmed videos for model training. However, it is expensive and time-consuming to acquire a large-scale trimmed video dataset. This paper presents a new weakly supervised architecture, called UntrimmedNet, which is able to directly learn action recognition models from untrimmed videos without the requirement of temporal annotations of action instances. Our UntrimmedNet couples two important components, the classification module and the selection module, to learn the action models and reason about the temporal duration of action instances, respectively. These two components are implemented with feed-forward networks, and UntrimmedNet is therefore an end-to-end trainable architecture. We exploit the learned models for action recognition (WSR) and detection (WSD) on the untrimmed video datasets of THUMOS14 and ActivityNet. Although our UntrimmedNet only employs weak supervision, our method achieves performance superior or comparable to that of those strongly supervised approaches on these two datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, background): “Previous weakly-supervised approaches utilised video-level or transcript supervision, where the set [64,65,66,67,68,69] or sequence [70,71,72,73,74,75] of actions in the video are used in training, without temporal bounds.”
Skeleton-Based Online Action Prediction Using Scale Selection Network (2019, background): “[64], [65], [66] also copes with untrimmed videos.”
RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, methodology): “Thus, proposal-based methods [95, 97] can be adopted for offline processing.”
NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learning (2018, methodology): “[34] use a purely CNN-based approach to weakly supervised action detection, where they use a different type of supervision, namely short unordered action lists.”
Weakly-Supervised Action Localization With Background Modeling (2019, methodology): “[30] 44.”
BSN: Boundary Sensitive Network for Temporal Action Proposal Generation (2018, methodology): “On THUMOS14, we use top-2 video-level classes generated by UntrimmedNet [43] for proposals generated by BSN and other methods, where multiplication of confidence score and class score is used for retrieving detections.”
Weakly Supervised Object Localization and Detection: A Survey (2021, unknown): “On the other hand, [105], [129], [133], [162] develop methods to localize temporal actions in the given untrimmed videos, where the main goal is to predict the temporal boundary of each action instance contained in the weakly labeled training videos.”
Rethinking the Faster R-CNN Architecture for Temporal Action Localization (2018, background): “Others have also studied temporal action localization in a weakly supervised setting [41, 48], where only video-level action labels are available for training.”
SSNet: Scale Selection Network for Online 3D Action Prediction (2018, background): “Beside the online action prediction task, the problem of temporal action detection [68, 73, 48, 33, 7, 53, 68, 14, 54, 16, 15, 65, 82] also copes with untrimmed videos.”
Weakly Supervised Action Localization by Sparse Temporal Pooling Network (2017, methodology): “Our algorithm outperforms the other two existing approaches based on weakly supervised learning [39, 32].”
Dynamic Sampling Networks for Efficient Action Recognition in Videos (2020, methodology): “They also developed an UntrimmedNet [32] which utilized the attention model”
End-to-End Temporal Action Detection With Transformer (2021, methodology): “There are also some weakly supervised methods [27, 21, 30, 31, 34, 40, 46], which are beyond the scope of this paper.”
Dynamic Inference: A New Approach Toward Efficient Video Action Recognition (2020, background): “Action recognition has drawn great research attention in the community [29, 33, 23, 30, 6, 26, 1, 34, 22, 31, 27, 28, 35, 2, 12, 20].”
ActionFormer: Localizing Moments of Actions with Transformers (2022, methodology): “We also combined external classification scores from UntrimmedNet [65] following previous works [4, 34, 76, 84].”
Graph Convolutional Networks for Temporal Action Localization (2019, methodology): “To achieve this, we assign the top-2 video-level classes predicted by UntrimmedNet to all the proposals in that video.”
What Do I Annotate Next? An Empirical Study of Active Learning for Action Localization (2018, background): “[68] propose a new model that uses video-level annotations combined with an attention mechanism to pinpoint actions temporal bounds.”
G-TAD: Sub-Graph Localization for Temporal Action Detection (2019, methodology): “In inference, following [30] to leverage the global video context, we take the video classification scores from [42] and [51], and multiply them by the fused confidence score pj for evaluation.”
Human Action Recognition and Prediction: A Survey (2018, background): “The idea of their action proposals inspired many later research (Escorci et al., 2016; Shou et al., 2017; Wang et al., 2017;Zhao et al., 2017;Xu et al., 2017;Gao et al., 2017;Chao et al., 2018).”
Part-Activated Deep Reinforcement Learning for Action Prediction (2018, unknown): “Different from action recognition [4, 5, 34, 35, 41, 42, 46, 47],”