NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learning

By Alexander Richard, Hilde Kuehne, Ahsan Iqbal, Juergen Gall. 2018

154 citations recorded by the source corpus.

Video learning is an important task in computer vision and has experienced increasing interest over the recent years. Since even a small amount of videos easily comprises several million frames, methods that do not rely on a frame-level annotation are of special importance. In this work, we propose a novel learning algorithm with a Viterbi-based loss that allows for online and incremental learning of weakly annotated video data. We moreover show that explicit context and length modeling leads to huge improvements in video segmentation and labeling tasks and include these models into our framework. On several action segmentation benchmarks, we obtain an improvement of up to 10% compared to current state-of-the-art methods.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Kristen Grauman
2. Luc Van Gool
3. Jitendra Malik
4. Cordelia Schmid
5. Li Fei-Fei
6. Andrew Zisserman
7. Trevor Darrell
8. Dacheng Tao

In Representation & Self-Supervised Learning

1. Juergen Gall
2. Hilde Kuehne
3. Alexander Richard
4. Sinisa Todorovic
5. Jun Li
6. Peng Lei
7. Giovanni Maria Farinella
8. Antonino Furnari

Citations in the corpus

• ActBERT: Learning Global-Local Video-Text Representations (2020, neutral): “NN-Viterbi [31] 21.”
• Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, background): “Previous weakly-supervised approaches utilised video-level or transcript supervision, where the set [64,65,66,67,68,69] or sequence [70,71,72,73,74,75] of actions in the video are used in training, without temporal bounds.”
• COIN: A Large-Scale Dataset for Comprehensive Instructional Video Analysis (2019, methodology): “We followed the default train and inference pipeline of Action-Sets [1], NN-Viterbi [2] and TCFPN-ISBA [5].”
• VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding (2021, neutral): “Frame Accuracy ↑ S UPERVISED NN-Viterbi (Richard et al., 2018) 21.2 VGG (Simonyan and Zisserman, 2014) 25.8 TCFPN-ISBA (Ding and Xu, 2018) 34.3 CBT (Sun et al., 2019a) 53.9 ActBERT (Zhu and Yang, 2020) 57.0 MIL-NCE (Miech et al., 2020) 61.0 VideoCLIP (Fine-tuned) 68.7 Z ERO - SHOT VideoCLIP…”
• Few-Shot Video Classification via Temporal Alignment (2019, background): “In the vision community, researchers have growing interests in tackling the sequence alignment problem with high dimensional multi-modal data, such as finding the alignment between untrimmed video sequence and the corresponding textual action sequence [5, 8, 31].”
• Procedure Planning in Instructional Videos (2019, background): “Some of the tasks ask the question “What is happening?”, such as action recognition and temporal action segmentation [4, 13, 28], state understanding [2], video summarization/captioning [29, 32, 37], retrieval [32] etc.”
• Mining YouTube - A dataset for learning fine-grained action concepts from webly supervised video data (2019, unknown): “[13, 18, 21, 9, 22].”
• Learning Video Representations using Contrastive Bidirectional Transformer (2019, unknown): “IDEr Zhou et al. (2018c) 4.38 11.55 27.44 0.38 S3D 3.24 9.52 26.09 0.31 VideoBERT 4.33 11.94 28.80 0.55 CBT 5.12( 0.02) 12.97( 0.05) 30.44( 0.08) 0.64( 0.00) Method Frame Acc. Tang et al. (2019) 25.8 Richard et al. (2018) 21.2 Ding & Xu (2018) 34.3 CBT 53.9 Table 4: (Left) Video captioning resul”
• Weakly Supervised Energy-Based Learning for Action Segmentation (2019, methodology): “, with the Viterbi algorithm [22], and then train a classifier on these frame labels by minimizing the standard cross entropy loss.”
• D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentation (2019, methodology): “First, by explicitly modeling for the context between classes and their temporal progression, both GRU reest [28] and NN-Viterbi [29] are able to outperform ECTC by a large margin [15].”
• Action Recognition From Single Timestamp Supervision in Untrimmed Videos (2019, background): “Transcript supervision offers an ordered list of action labels in untrimmed videos, without any temporal annotations [4, 5, 14, 24, 18, 25, 26, 10].”
• Action Sets: Weakly Supervised Action Segmentation Without Ordering Constraints (2017, methodology): “When processing untrimmed videos, actions can either be localized in the temporal domain only [2, 24, 39, 29, 25, 15, 17, 26], or in the spatio-temporal domain [33, 7, 20, 38].”