Connectionist Temporal Modeling for Weakly Supervised Action Labeling

By De-An Huang, Li Fei-Fei, Juan Carlos Niebles. 2016

259 citations recorded by the source corpus.

We propose a weakly-supervised framework for action labeling in video, where only the order of occurring actions is required during training time. The key challenge is that the per-frame alignments between the input (video) and label (action) sequences are unknown during training. We address this by introducing the Extended Connectionist Temporal Classification (ECTC) framework to efficiently evaluate all possible alignments via dynamic programming and explicitly enforce their consistency with frame-to-frame visual similarities. This protects the model from distractions of visually inconsistent or degenerated alignments without the need of temporal supervision. We further extend our framework to the semi-supervised case when a few frames are sparsely annotated in a video. With less than 1 % of labeled frames per video, our method is able to outperform existing semi-supervised approaches and achieve comparable performance to that of fully supervised approaches.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Joint Discovery of Object States and Manipulation Actions (2017, background): “Such methods can incorporate (weak) supervision that helps to steer the clustering towards a preferred solution [5, 13, 21, 24, 42].”
Towards Automatic Learning of Procedures From Web Instructional Videos (2017, background): “Another topic similar to ours is action segmentation or labeling (Kuehne, Gall, and Serre 2016; Kuehne, Richard, and Gall 2017; Huang, Fei-Fei, and Niebles 2016; Bo-janowski et al. 2014).”
Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, neutral): “Previous weakly-supervised approaches utilised videolevel or transcript supervision, where the set [83,84, 85,86,87,88] or sequence [89,90,91,92,93,94] of actions in the video are used in training, without temporal bounds.”
RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, neutral): “Huang et al. (2017) incorporated the Lie group structure into a deep learning architecture to”
Temporal Convolutional Networks for Action Segmentation and Detection (2016, methodology): “In the literature, our task goes by the name action segmentation [7, 8, 6, 15, 29, 16, 10] or action detection [28, 20, 21, 25].”
Cross-Task Weakly Supervised Learning From Instructional Videos (2019, methodology): “As it is often the case in weakly supervised video learning [2, 18, 29], these assumptions constrain the search for when steps occur, helping tackle a combinatorial search space.”
Weakly-Supervised Action Segmentation with Iterative Soft Boundary Assignment (2018, methodology): “However, current methods stick to use Recurrent Neural Networks (RNN) to encode the video data and use different algorithms, such as Extended Connectionist Temporal Classification (ECTC) [6] and Hidden Markov Model (HMM) [16, 10], to go through all the possible action sequences and find the one with”
NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learning (2018, methodology): “In accordance with [16, 24, 11], we extract Fisher vectors of improved dense trajectories [33] over a temporal window of length 20 for each frame and reduce the result to 64 dimensions using PCA.”
SubUNets: End-to-End Hand Shape and Continuous Sign Language Recognition (2017, methodology): “As CTC is an ideal method for tasks where the data is weakly labelled, computer vision researchers have also applied this sequence-to-sequence learning method to sentence-level lip reading [3] and action recognition [21].”
Action Modifiers: Learning From Adverbs in Instructional Videos (2019, methodology): “Narrations are used to learn steps of complex tasks [1, 18, 30, 42, 45, 62], and more recently for video retrieval [33], visual grounding [17, 19], action segmentation [60] and learning actions through object state changes [2, 14].”
Weakly Supervised Energy-Based Learning for Action Segmentation (2019, methodology): “The same frame features are used by [8, 12, 20, 22].”
Neural Sign Language Translation (2018, methodology): “Computer vision researchers adopted CTC and applied it to weakly labeled visual problems, such as lip reading [3], action recognition [30], hand shape recognition [6] and CSLR [6, 17].”
Weakly Supervised Action Learning with RNN Based Fine-to-Coarse Modeling (2017, methodology): “For the Breakfast dataset, we follow the evaluation script of [14, 9] and report results as mean accuracy over frames over four splits.”
Action Sets: Weakly Supervised Action Segmentation Without Ordering Constraints (2017, methodology): “[6], in contrast, rely on connectionist temporal classification (CTC) with LSTMs and extend it by downweighting degenerated alignments and incorporating visual similarity of frames into the decoding algorithm.”
D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentation (2019, methodology): “The use of discriminative modeling and differentiable dynamic programming sets our approach apart from previous work that involves non-differentiable forward-backward algorithms [11, 15, 29] and dramatically alleviates the problem of degenerated alignments that aligns each action label to a single f”
W-TALC: Weakly-supervised Temporal Activity Localization and Classification (2018, background): “In [20], the authors propose a dynamic programming based approach to evaluate and search for possible alignments between video frames and the corresponding labels.”
One-Shot Action Localization by Learning Sequence Matching Network (2018, methodology): “[3, 17] proposed some weakly supervised action labeling methods using only the action order information.”
A Fully Differentiable Beam Search Decoder (2019, background): “, 2016) and other sequence labeling tasks (Liwicki et al., 2007; Huang et al., 2016).”
Weakly Supervised Action Localization by Sparse Temporal Pooling Network (2017, background): “There are some other approaches [3, 16, 25] that learn to localize or segment actions in a weakly supervised setting by exploiting the temporal order of subactions during training.”