A flexible model for training action localization with varying levels of supervision

By Guilhem Chéron, Jean-Baptiste Alayrac, Ivan Laptev, Cordelia Schmid. 2018

49 citations recorded by the source corpus.

Spatio-temporal action detection in videos is typically addressed in a fully-supervised setup with manual annotation of training videos required at every frame. Since such annotation is extremely tedious and prohibits scalability, there is a clear need to minimize the amount of manual supervision. In this work we propose a unifying framework that can handle and combine varying types of less demanding weak supervision. Our model is based on discriminative clustering and integrates different types of supervision as constraints on the optimization. We investigate applications of such a model to training setups with alternative supervisory signals ranging from video-level class labels over temporal points or sparse action bounding boxes to the full per-frame annotation of action bounding boxes. Experiments on the challenging UCF101-24 and DALY datasets demonstrate competitive performance of our method at a fraction of supervision used by previous methods. The flexibility of our model enables joint learning from data with different levels of annotation. Experimental results demonstrate a significant gain by adding a few fully supervised examples to otherwise weakly labeled videos.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, methodology): “Multiple instance learning methods have been employed in many weakly-supervised video understanding problems including: person recognition in movies using scripts [12, 50, 61], anomaly detection [70], weakly supervised action classification [47, 68] and localization [17, 22, 82], co-reference resolu”
Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, background): “Other forms of weak supervision, including single-pixel and single-timestamp supervision, are gaining popularity due to their scalability and performance balance [63,76,77,78].”
Action Recognition From Single Timestamp Supervision in Untrimmed Videos (2019, methodology): “More recently, several forms of weak supervision, including single temporal points, are evaluated in [7] for the task of spatio-temporal action localisation.”