Action Sets: Weakly Supervised Action Segmentation Without Ordering Constraints

By Alexander Richard, Hilde Kuehne, Juergen Gall. 2017

107 citations recorded by the source corpus.

Action detection and temporal segmentation of actions in videos are topics of increasing interest. While fully supervised systems have gained much attention lately, full annotation of each action within the video is costly and impractical for large amounts of video data. Thus, weakly supervised action detection and temporal segmentation methods are of great importance. While most works in this area assume an ordered sequence of occurring actions to be given, our approach only uses a set of actions. Such action sets provide much less supervision since neither action ordering nor the number of action occurrences are known. In exchange, they can be easily obtained, for instance, from meta-tags, while ordered sequences still require human annotation. We introduce a system that automatically learns to temporally segment and label actions in a video, where the only supervision that is used are action sets. An evaluation on three datasets shows that our method still achieves good results although the amount of supervision is significantly smaller than for other related methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, background): “Instructional videos are rising in popularity in the context of learning steps of complex tasks [2, 16, 41, 42, 46, 68], visual-linguistic reference resolution [17, 18], action segmentation in long untrimmed videos [66] and joint learning of object states and actions [3].”
Cross-Task Weakly Supervised Learning From Instructional Videos (2019, methodology): “To make a fair comparison, we use the trained classifier of the method in [27], and apply the same inference procedure as in our method.”
COIN: A Large-Scale Dataset for Comprehensive Instructional Video Analysis (2019, methodology): “Given the weakest supervision of video transcripts without ordering constraint, Action-Sets [32] achieves the result of 4.”
NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learning (2018, background): “A similar kind of supervision using unordered sets of actions is proposed in [25], who also rely on the model factorization proposed by [23].”
Weakly Supervised Energy-Based Learning for Action Segmentation (2019, background): “Finally, [21] addresses a different weakly supervised setting from ours when the ground truth provides only a set of actions present without their temporal ordering .”
Coherent Temporal Synthesis for Incremental Action Segmentation (2024, unknown): “Weaker forms of supervision include action transcripts [25], action sets [14,29,40], timestamps [31,38] and activity labels [11].”
Action Modifiers: Learning From Adverbs in Instructional Videos (2019, methodology): “Narrations are used to learn steps of complex tasks [1, 18, 30, 42, 45, 62], and more recently for video retrieval [33], visual grounding [17, 19], action segmentation [60] and learning actions through object state changes [2, 14].”
Vision-based human activity recognition: a survey (2020, background): “Hidden Markov Models [82, 83], CRF [113, 179], regressive models [49, 115, 130] and context-free grammars [40, 149] are examples of action grammars.”