Weakly Supervised Action Labeling in Videos under Ordering Constraints

By Piotr Bojanowski, Rémi Lajugie, Francis R. Bach, Ivan Laptev, Jean Ponce, Cordelia Schmid, Josef Sivic. 2014

250 citations recorded by the source corpus.

We are given a set of video clips, each one annotated with an ordered list of actions, such as “walk” then “sit” then “answer phone” extracted from, for example, the associated text script. We seek to temporally localize the individual actions in each clip as well as to learn a discriminative classifier for each action. We formulate the problem as a weakly supervised temporal assignment with ordering constraints. Each video clip is divided into small time intervals and each time interval of each video clip is assigned one action label, while respecting the order in which the action labels appear in the given annotations. We show that the action label assignment can be determined together with learning a classifier for each action in a discriminative manner. We evaluate the proposed model on a new and challenging dataset of 937 video clips with a total of 787720 frames containing sequences of 16 different actions from 69 Hollywood movies.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books (2015, background): “[2] localize actions from an ordered list of labels in video clips.”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, methodology): “Therefore, it avoids costly projections and allows the use of complicated constraints such as temporal ordering [5].”
Affine Invariant Analysis of Frank-Wolfe on Strongly Convex Sets (2020, background): “They form a practical family of algorithms [Jaggi 2013; Bojanowski et al. 2014; Alayrac et al. 2016; Seguin et al. 2016; Peyre et al. 2017; Miech x k +1 = (1 − γ t ) x k + γ k v k ⊲ Convex update 5: end for et al. 2018; Lacoste-Julien et al. 2015b; Courty et al. 2016; Paty et al. 2019; Luise et al.…”
Joint Discovery of Object States and Manipulation Actions (2017, methodology): “Previous work has explored “exact one” ordering constraints for time localization problems [6].”
Unsupervised Learning from Narrated Instruction Videos (2015, background): “The values of precision and recall are given at the bottom of Table 1.”
Towards Automatic Learning of Procedures From Web Instructional Videos (2017, methodology): “The proposed ProcNets-LSTM model outperforms all other methods by a huge margin in both Jaccard and mIoU.”
Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, background): “2019) or sequence (Bojanowski et al. 2014; Huang et al. 2016; Ding and Xu 2018; Richard et al. 2018; Chang et al. 2019; Li et al. 2019) of actions in the video are used in training, without temporal bounds.”
Unsupervised Learning by Predicting Noise (2017, methodology): “It has been successfully applied to several computer vision applications, like object discovery (Joulin et al., 2010; Tang et al., 2014) or video/text alignment (Bojanowski et al., 2013; 2014; Ra-manathan et al., 2014).”
A dataset for Movie Description (2015, background): “[11] aims to localize individual short actions in longer clips by exploiting the ordering constrains as weak supervision.”
Describing Videos by Exploiting Temporal Structure (2015, background): “Such structure has demonstrated to be helpful in the context of event and action classification [32, 12, 5], and is explored in this paper.”
Cross-Task Weakly Supervised Learning From Instructional Videos (2019, methodology): “For instance, we make use of ordering constraints to obtain supervision, as was done in [5, 18, 22, 26, 6].”
Weakly-Supervised Alignment of Video with Text (2015, methodology): “Contrary to [7], for example, our work makes use of continuous text representations.”
Weakly-Supervised Learning of Visual Relations (2017, methodology): “Our work uses a discriminative clustering objective [2], that has been successful in several computer vision tasks [4, 23], but has not been so far, to the best of our knowledge, used for modeling relations.”
Connectionist Temporal Modeling for Weakly Supervised Action Labeling (2016, methodology): “For metrics, we follow [5] and use mean average precision for evaluating action detection and average Jaccard measure for evaluating the action alignment.”
Towards Weakly-Supervised Action Localization (2016, background): “[26] assume an ordered list of the actions in each video as input.”
Weakly-Supervised Action Segmentation with Iterative Soft Boundary Assignment (2018, methodology): “[2] formulate the temporal assignment problem and propose Ordering Constrained Discriminative Clustering (OCDC), with the introduction of Hollywood extended dataset.”
NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learning (2018, methodology): “Hollywood Extended has been introduced in [1] for the task of action alignment, which we address with our method in Section 6.”