Action Recognition by Hierarchical Mid-Level Action Elements

By Tian Lan, Yuke Zhu, Amir Zamir, Silvio Savarese. 2015

82 citations recorded by the source corpus.

Realistic videos of human actions exhibit rich spatiotemporal structures at multiple levels of granularity: an action can always be decomposed into multiple finer-grained elements in both space and time. To capture this intuition, we propose to represent videos by a hierarchy of mid-level action elements (MAEs), where each MAE corresponds to an action-related spatiotemporal segment in the video. We introduce an unsupervised method to generate this representation from videos. Our method is capable of distinguishing action-related segments from background segments and representing actions at multiple spatiotemporal resolutions. Given a set of spatiotemporal segments generated from the training data, we introduce a discriminative clustering algorithm that automatically discovers MAEs at multiple levels of granularity. We develop structured models that capture a rich set of spatial, temporal and hierarchical relations among the segments, where the action label and multiple levels of MAE labels are jointly inferred. The proposed model achieves state-of-the-art performance in multiple action recognition benchmarks. Moreover, we demonstrate the effectiveness of our model in real-world applications such as action recognition in large-scale untrimmed videos and action parsing.

Subfields: Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

Actions ~ Transformations (2015, background): “Mid-to high-level representation are also used to model complex actions [41, 55, 36, 60, 14, 24].”
Connectionist Temporal Modeling for Weakly Supervised Action Labeling (2016, background): “As significant progress has been made on categorizing temporally trimmed video clips, recent research of human activity analysis is shifting towards a higher level understanding in real-world settings [11,22,29,35,43,49].”
Towards Weakly-Supervised Action Localization (2016, background): “[36] leverage a discriminative clustering approach for parsing complex actions into mid-level action elements.”
A Hierarchical Pose-Based Approach to Complex Action Understanding Using Dictionaries of Actionlets and Motion Poselets (2016, background): “Another recent trend aims at finegrained detection of actions performed in sequence such as those in a cooking scenario [24, 16].”
Asynchronous Temporal Fields for Action Recognition (2016, background): “Recently there has been a push towards mid-level representations of video [37, 46, 13, 20], that capture beyond local properties.”
Action Recognition with Image Based CNN Features (2015, methodology): “[18] present an unsupervised method for building hierarchical models to represent human actions using multiple ﬁner-grained elements, called Mid-level Action Elements ( MAE s).”
Temporal Cycle-Consistency Learning (2019, background): “Past work has also explored parsing of fine-grained actions in videos [24, 25, 29] while some others [13, 33, 34, 36] discover sub-activities without explicit supervision of temporal boundaries.”
Discriminative Hierarchical Rank Pooling for Activity Recognition (2016, methodology): “A CRF-based hierarchical sequence summarization method is presented in [26]; a hierarchical recurrent neural network for skeleton based action recognition is presented in [3]; and a hierarchical action proposal based mid-level representation is presented in [16].”