Towards Weakly-Supervised Action Localization

By Philippe Weinzaepfel, Xavier Martin, Cordelia Schmid. 2016

47 citations recorded by the source corpus.

This paper presents a novel approach for weakly-supervised action localization, i.e., that does not require per-frame spatial annotations for training. We first introduce an effective method for extracting human tubes by combining a state-of-the-art human detector with a tracking-by-detection approach. Our tube extraction leverages the large amount of annotated humans available today and outperforms the state of the art by an order of magnitude: with less than 5 tubes per video, we obtain a recall of 95% on the UCF-Sports and J-HMDB datasets. Given these human tubes, we perform weakly-supervised selection based on multi-fold Multiple Instance Learning (MIL) with improved dense trajectories and achieve excellent results. We obtain a mAP of 84% on UCF-Sports, 54% on J-HMDB and 45% on UCF-101, which outperforms the state of the art for weakly-supervised action localization and is close to the performance of the best fully-supervised approaches. The second contribution of this paper is a new realistic dataset for action localization, named DALY (Daily Action Localization in YouTube). It contains high quality temporal and spatial annotations for 10 actions in 31 hours of videos (3.3M frames), which is an order of magnitude larger than standard action localization datasets. On the DALY dataset, our tubes have a spatial recall of 82%, but the detection task is extremely challenging, we obtain 10.8% mAP.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, neutral): “Multiple instance learning methods have been employed in many weakly-supervised video understanding problems including: person recognition in movies using scripts [12, 50, 61], anomaly detection [70], weakly supervised action classification [47, 68] and localization [17, 22, 82], co-reference resolu”
AVA: A Video Dataset of Spatio-Temporally Localized Atomic Visual Actions (2017, methodology): “Recent extensions, such as UCF101 [33], DALY [35] and Hollywood2Tubes [23] evaluate spatio-temporal localization in untrimmed videos, which makes the task significantly harder and results in a performance drop.”
Chained Multi-stream Networks Exploiting Pose, Motion, and Appearance for Action Classification and Detection (2017, background): “…the approach is fast: spatial detection runs at a rate of 31 fps and spatio-temporal detection with 10 fps. Compared to the recent works [9, 45, 29, 47], our detection framework has two desirable properties: (1) the pose network directly provides a single detection box per person, which…”
Asynchronous Temporal Fields for Action Recognition (2016, background): “Dataset Recent years have witnessed an emergence of large-scale datasets containing sequences of common daily activities [42, 63, 60].”
Learning from Video and Text via Large-Scale Discriminative Clustering (2017, background): “Another recent work [41] proposes a multi-fold Multiple-Instance Learning (MIL) SVM to localize actions given video-level supervision at training time.”
Online Real-Time Multiple Spatiotemporal Action Localisation and Prediction (2016, background): “Spatio-temporal action localisation can be approached in a supervised [27, 32], semi-supervised [48, 52], or weakly supervised [33, 53, 44] manner.”