Unsupervised Learning and Segmentation of Complex Activities from Video

By Fadime Sener, Angela Yao. 2018

126 citations recorded by the source corpus.

This paper presents a new method for unsupervised segmentation of complex activities from video into multiple steps, or sub-activities, without any textual input. We propose an iterative discriminative-generative approach which alternates between discriminatively learning the appearance of sub-activities from the videos' visual features to sub-activity labels and generatively modelling the temporal structure of sub-activities using a Generalized Mallows Model. In addition, we introduce a model for background to account for frames unrelated to the actual activities. Our approach is validated on the challenging Breakfast Actions and Inria Instructional Videos datasets and outperforms both unsupervised and weakly-supervised state of the art.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, background): “Instructional videos are rising in popularity in the context of learning steps of complex tasks [2, 16, 41, 42, 46, 68], visual-linguistic reference resolution [17, 18], action segmentation in long untrimmed videos [66] and joint learning of object states and actions [3].”
Cross-Task Weakly Supervised Learning From Instructional Videos (2019, neutral): “While these scores are slightly lower compared to those obtained by the single-task probabilistic model in Sener [28] (25.4% at K=10 and 23.6% at K=15), we are unable to compare using our full cross-task model on this dataset.”
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos (2020, methodology): “A variety of tasks, focused primarily on text-video modelling, have been applied to these datasets including: task segmentation [19–21], reference resolution [22, 23], action segmentation [5], video clip ordering [24–27], and action recognition [28–30].”
Coherent Temporal Synthesis for Incremental Action Segmentation (2024, unknown): “We use the same set of feature representations as [26,42].”
Zero-Shot Anticipation for Instructional Activities (2018, background): “Newer works are weakly-supervised, using cues from narrations [34, 48, 6] or receiving ordered sequences of the actions in videos [11, 24, 41], or fully unsupervised [47].”
Temporal Aggregate Representations for Long-Range Video Understanding (2020, background): “Various models for complex activity understanding have been addressed before [6, 8, 30]; these approaches are designed to work on instructional videos by explicitly modelling their sequence dynamics.”
Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities (2022, background): “Breakfast [22], GTEA [12], 50Salads [47] are major contributors to the study of multistep activities [9,10,39].”