Weakly Supervised Action Learning with RNN Based Fine-to-Coarse Modeling

By Alexander Richard, Hilde Kuehne, Juergen Gall. 2017

222 citations recorded by the source corpus.

We present an approach for weakly supervised learning of human actions. Given a set of videos and an ordered list of the occurring actions, the goal is to infer start and end frames of the related action classes within the video and to train the respective action classifiers without any need for hand labeled frame boundaries. To address this task, we propose a combination of a discriminative representation of subactions, modeled by a recurrent neural network, and a coarse probabilistic model to allow for a temporal alignment and inference over long sequences. While this system alone already generates good results, we show that the performance can be further improved by approximating the number of subactions to the characteristics of the different action classes. To this end, we adapt the number of subaction classes by iterating realignment and reestimation during training. The proposed system is evaluated on two benchmark datasets, the Breakfast and the Hollywood extended dataset, showing a competitive performance on various weak learning tasks such as temporal action segmentation and action alignment.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, background): “Instructional videos are rising in popularity in the context of learning steps of complex tasks [2, 16, 41, 42, 46, 68], visual-linguistic reference resolution [17, 18], action segmentation in long untrimmed videos [66] and joint learning of object states and actions [3].”
Skeleton-Based Action Recognition Using Spatio-Temporal LSTM Network with Trust Gates (2017, neutral): “[25], [26], [27], [28], [29], [30], and RGB-based activity recognition [31], [32], [33], [34], [35], [36].”
Cross-Task Weakly Supervised Learning From Instructional Videos (2019, methodology): “For instance, we make use of ordering constraints to obtain supervision, as was done in [5, 18, 22, 26, 6].”
Weakly-Supervised Action Segmentation with Iterative Soft Boundary Assignment (2018, methodology): “Following a similar pipeline, Richard et al. [16] propose an iterative fine-to-coarse sub-action modeling mechanism with RNN and HMM.”
NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learning (2018, methodology): “1, we briefly summarize the training in a fully supervised setting and the training procedure that is used in [16, 12, 24, 13] for weakly supervised learning.”
MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation (2019, methodology): “Earlier approaches for action segmentation can be grouped into two categories: sliding window approaches [22, 11, 19], that use temporal windows of different scales to detect action segments, and hybrid approaches that apply a coarse temporal modeling using Markov models on top of frame-wise classif”
Multimodal Clustering Networks for Self-supervised Learning from Unlabeled Videos (2021, methodology): “The MiningYoutube [26] dataset considers the task of frame-level temporal action segmentation as used in weakly supervised action learning [37].”
When will you do what? - Anticipating Temporal Occurrences of Activities (2018, methodology): “This is the same grammar as proposed in [22].”
Android in the Wild: A Large-Scale Dataset for Android Device Control (2023, background): “Instead of using application-specific APIs, which are not generally available for any given application or function, these systems directly manipulate user interface (UI) elements on a screen, exactly as a human does [1, 26, 27, 31, 19].”
Weakly Supervised Action Localization by Sparse Temporal Pooling Network (2017, background): “There are some other approaches [3, 16, 25] that learn to localize or segment actions in a weakly supervised setting by exploiting the temporal order of subactions during training.”
3C-Net: Category Count and Center Loss for Weakly-Supervised Action Localization (2019, background): “[17, 2] exploit the order of actions in a video as a weak supervision cue.”
Zero-Shot Anticipation for Instructional Activities (2018, background): “Newer works are weakly-supervised, using cues from nar-rations [34, 48, 6] or receiving ordered sequences of the actions in videos [11, 24, 41], or fully unsupervised [47].”
Action Recognition From Single Timestamp Supervision in Untrimmed Videos (2019, background): “Transcript supervision offers an ordered list of action labels in untrimmed videos, without any temporal annotations [4, 5, 14, 24, 18, 25, 26, 10].”
D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentation (2019, methodology): “First, by explicitly modeling for the context between classes and their temporal progression, both GRU reest [28] and NN-Viterbi [29] are able to outperform ECTC by a large margin [15].”
Action Sets: Weakly Supervised Action Segmentation Without Ordering Constraints (2017, methodology): “Improved dense trajectories, on the contrary, are extracted in an unsupervised manner, making them the features of choice for most weakly supervised approaches [2, 15, 6, 25].”
Temporal Aggregate Representations for Long-Range Video Understanding (2020, methodology): “Temporal models that are built into the architecture [6, 8, 12, 29] are generally favoured because they allow frameworks to be learned end-to-end.”