Spot On: Action Localization from Pointly-Supervised Proposals

By Pascal Mettes, Jan van Gemert, Cees G. M. Snoek. 2016

127 citations recorded by the source corpus.

We strive for spatio-temporal localization of actions in videos. The state-of-the-art relies on action proposals at test time and selects the best one with a classifier trained on carefully annotated box annotations. Annotating action boxes in video is cumbersome, tedious, and error prone. Rather than annotating boxes, we propose to annotate actions in video with points on a sparse subset of frames only. We introduce an overlap measure between action proposals and points and incorporate them all into the objective of a non-convex Multiple Instance Learning optimization. Experimental evaluation on the UCF Sports and UCF 101 datasets shows that (i) spatio-temporal proposals can be used to train classifiers while retaining the localization performance, (ii) point annotations yield results comparable to box annotations while being significantly faster to annotate, (iii) with a minimum amount of supervision our approach is competitive to the state-of-the-art. Finally, we introduce spatio-temporal action annotations on the train and test videos of Hollywood2, resulting in Hollywood2Tubes, available at http://tinyurl.com/hollywood2tubes.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

VideoLSTM convolves, attends and flows for action recognition (2016, methodology): “Interestingly, however, VideoLSTM is able to slightly outperform recent methods that rely on unsupervised action proposals in combination with box (van Gemert et al., 2015) or point supervision (Mettes et al., 2016), in addition to the action class.”
Asynchronous Temporal Fields for Action Recognition (2016, background): “Most contemporary deep learning based methods have treated the problem of video understanding as that of only appearance and motion (trajectory) modeling [44, 54, 7, 28].”
Human Action Localization with Sparse Spatial Supervision (2017, methodology): “[15] to our human #nframes UCF-Sports J-HMDB UCF-101 1 67.”
Efficient Object Annotation via Speaking and Pointing (2019, background): “2018) and video (Mettes et al. 2016; Manen et al. 2017) object localization tasks.”
Action Search: Spotting Actions in Videos and Its Application to Temporal Action Localization (2017, background): “A large number of works have successfully tackled the task of action recognition [8,9,28,38,42] and spatio-temporal localization [10,18,27,30,33,39].”
A flexible model for training action localization with varying levels of supervision (2018, methodology): “In [27], the authors introduce the idea of working with spatial points instead of bounding boxes for spatial supervision.”
Action Recognition From Single Timestamp Supervision in Untrimmed Videos (2019, methodology): “In [20] a single pixel is used to annotate the action, in a subset of frames, both spatially and temporally.”
CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videos (2017, background): “Recently, temporal action localization has drawn considerable interest in the computer vision community [25, 15, 39, 65, 26, 67, 53, 47, 43, 73, 9, 18, 36].”
3C-Net: Category Count and Center Loss for Weakly-Supervised Action Localization (2019, background): “The work of [25, 14, 28] use action category labels in videos for temporal localization, whereas [13] uses point-level supervision to spatio-temporally localize the actions.”
Pointly-Supervised Instance Segmentation (2021, background): “Point-based supervision has been studied in a variety of tasks, including action localization [37, 38], object detection [42, 46], object counting [27], and semantic segmentation [4, 44].”
Action Sets: Weakly Supervised Action Segmentation Without Ordering Constraints (2017, methodology): “When processing untrimmed videos, actions can either be localized in the temporal domain only [2, 24, 39, 29, 25, 15, 17, 26], or in the spatio-temporal domain [33, 7, 20, 38].”
FineGym: A Hierarchical Video Dataset for Fine-Grained Action Understanding (2020, methodology): “More challenging datasets are proposed subsequently, including UCF101 [40], Kinetics [3], ActivityNet [11], Moments in Time [32], and others [25, 18, 50, 2, 52, 38, 33, 22, 46, 31, 20].”
Y Qualification Main task Y Return N N Y User qualified ? N Feedback Pass ? Pass ? Annotator training Annotating images (2017, background): “[42] proposed to annotate actions in videos with click annotations.”
TURN TAP: Temporal Unit Regression Network for Temporal Action Proposals (2017, methodology): “ges with SelectiveSearch [28] and then apply R-CNN [8] on both RGB and optical ﬂow images for action detection. Weinzaepfel et al. [31] replace SelectiveSearch [28] with EdgeBoxes [36]. Mettes et al. [18] propose to use sparse points Figure 2. Architecture of Temporal Unit Regression Network (TURN).”
Temporal Action Detection with Structured Segment Networks (2017, background): “Note there are also works on spatial-temporal detection [13, 54, 25, 50, 31] and temporal video segmentation [15], which are beyond the scope of this paper.”
Single Shot Temporal Action Detection (2017, background): “And there are some approaches specially proposed for temporal action proposal [3, 5, 7, 22, 43].”