Weakly supervised learning of actions from transcripts

By Hilde Kuehne, Alexander Richard, Juergen Gall. 2016

131 citations recorded by the source corpus.

Abstract We present an approach for weakly supervised learning of human actions from video transcriptions. Our system is based on the idea that, given a sequence of input data and a transcript, i.e. a list of the order the actions occur in the video, it is possible to infer the actions within the video stream and to learn the related action models without the need for any frame-based annotation. Starting from the transcript information at hand, we split the given data sequences uniformly based on the number of expected actions. We then learn action models for each class by maximizing the probability that the training video sequences are generated by the action models given the sequence order as defined by the transcripts. The learned model can be used to temporally segment an unseen video with or without transcript. Additionally, the inferred segments can be used as a starting point to train high-level fully supervised models. We evaluate our approach on four distinct activity datasets, namely Hollywood Extended, MPII Cooking, Breakfast and CRIM13. It shows that the proposed system is able to align the scripted actions with the video data, that the learned models localize and classify actions in the datasets, and that they outperform any current state-of-the-art approach for aligning transcripts with video data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Towards Automatic Learning of Procedures From Web Instructional Videos (2017, background): “Kuehne et al. (Kuehne, Richard, and Gall 2017) apply Hidden Markov Models (HMM) to learn the likelihood of image features given hidden action states.”
Cross-Task Weakly Supervised Learning From Instructional Videos (2019, methodology): “, like [1, 18, 22, 29]) and jointly learn when steps occur and what they look like.”
A Case Study on Combining ASR and Visual Features for Generating Instructional Video Captions (2019, background): “Several works focus on learning video-instruction alignments, and match a fixed set of instructions to temporal video segments (Regneri et al., 2013; Naim et al., 2015; Malmaud et al., 2015; Hendricks et al., 2017; Kuehne et al., 2017).”
Weakly-Supervised Action Segmentation with Iterative Soft Boundary Assignment (2018, methodology): “Our method outperforms HTK significantly on Breakfast dataset.”
NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learning (2018, methodology): “[16] interpret the task of learning an action segmentation system just given an ordered list of occurring actions as supervision as an instance of a speech recognition problem, where the videos correspond to the audio signal and the action classes correspond to words.”
D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentation (2019, background): “In contrast to fully annotated approaches [20, 31, 39] which require annotations of the exact start and end time of each action, weakly supervised approaches [7, 15, 29, 3, 21] significantly reduce the required annotation effort and improve the applicability to real-world data.”
Pattern Recognition (2015, unknown): “Gemert et al. [27] show dense-trajectories have a powerful ability to perform spatio-temporal action localization, but it fails in detecting small motion.”
Action Sets: Weakly Supervised Action Segmentation Without Ordering Constraints (2017, methodology): “[15] extend their approach from [13] to weak supervision by inferring a linear segmentation from ordered action sequences and training a classical GMM+HMM speech recognition system on iteratively refined segmentations.”
W-TALC: Weakly-supervised Temporal Activity Localization and Classification (2018, methodology): “A similar iterative process is presented by the same authors in [31], but without employing an RNN.”
Coherent Temporal Synthesis for Incremental Action Segmentation (2024, unknown): “Weaker forms of supervision include action transcripts [25], action sets [14,29,40], timestamps [31,38] and activity labels [11].”
Weakly Supervised Action Learning with RNN Based Fine-to-Coarse Modeling (2017, methodology): “In order to limit the amount of action sequences to optimize over, a context-free grammar G is created from the training set as in [16].”
MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation (2019, methodology): “[14] train a model for action segmentation from video transcripts.”
UntrimmedNets for Weakly Supervised Action Recognition and Detection (2017, methodology): “Weakly supervised learning was extensively studied in object recognition and detection [32, 1, 9, 4], and there were several works adapting this method to learn action models from videos [26, 8, 2, 3, 15, 24].”
Weakly Supervised Energy-Based Learning for Action Segmentation (2019, methodology): “The same frame features are used by [8, 12, 20, 22].”
Action Recognition From Single Timestamp Supervision in Untrimmed Videos (2019, background): “Some works [10, 18, 24] assume the transcript includes knowledge of ‘background’, specifying whether the actions occur in succession or with gaps.”