Latent Hierarchical Model of Temporal Structure for Complex Activity Classification

By Limin Wang, Yu Qiao, Xiaoou Tang. 2014

105 citations recorded by the source corpus.

Modeling the temporal structure of sub-activities is an important yet challenging problem in complex activity classification. This paper proposes a latent hierarchical model (LHM) to describe the decomposition of complex activity into sub-activities in a hierarchical way. The LHM has a tree-structure, where each node corresponds to a video segment (sub-activity) at certain temporal scale. The starting and ending time points of each sub-activity are represented by two latent variables, which are automatically determined during the inference process. We formulate the training problem of the LHM in a latent kernelized SVM framework and develop an efficient cascade inference method to speed up classification. The advantages of our methods come from: 1) LHM models the complex activity with a deep structure, which is decomposed into sub-activities in a coarse-to-fine manner and 2) the starting and ending time points of each segment are adaptively determined to deal with the temporal displacement and duration variation of sub-activity. We conduct experiments on three datasets: 1) the KTH; 2) the Hollywood2; and 3) the Olympic Sports. The experimental results show the effectiveness of the LHM in complex activity classification. With dense features, our LHM achieves the state-of-the-art performance on the Hollywood2 dataset and the Olympic Sports dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

A New Representation of Skeleton Sequences for 3D Action Recognition (2017, background): “To recognize a video action, the temporal information of the sequence needs to be exploited to understand the dynamics of human postures [29, 9, 44, 8, 20].”
Action recognition with trajectory-pooled deep-convolutional descriptors (2015, background): “Human action recognition [1, 24, 31, 35, 36] in videos attracts increasing research interests in computer vision community due to its potential applications in video surveillance, human computer interaction, and video content analysis.”
Temporal Segment Networks for Action Recognition in Videos (2017, methodology): “Many research works have been devoted to modeling the temporal structure of video for action recognition [19], [20], [21], [22], [56], [57].”
Mining Motion Atoms and Phrases for Complex Action Recognition (2013, methodology): “For instance, the sport action of highjump can be decomposed into running, jumping, and landing.”
Large Margin Dimensionality Reduction for Action Similarity Labeling (2014, background): “H UMAN action analysis in videos has become a highly active research area due to its wide applications in video surveillance, human-computer interface, and content based video retrieval [1], [2], [3], [4], [5], [6].”
Knowledge Guided Disambiguation for Large-Scale Scene Classification With Multi-Resolution CNNs (2016, background): “object detection [13], [14], event recognition [15], [16], and action recognition [17], [18].”
Video Action Detection with Relational Dynamic-Poselets (2014, methodology): “[28] designed a Latent Hierarchical Model (LHM) to capture the temporal structure among segments in a coarse-to-fine manner.”
Simple to Complex Transfer Learning for Action Recognition (2016, methodology): “methods [4], [7]–[9], [27], [31], [32] have achieved promising performance in recognizing both simple and complex actions.”
Temporal Segment Networks: Towards Good Practices for Deep Action Recognition (2016, methodology): “Wang et al. [17] and Pirsiavash et al. [29] extended the temporal decomposition of complex action into a hierarchical manner using Latent Hierarchical Model (LHM) and Segmental Grammar Model (SGM), respectively.”
Multi-view Super Vector for Action Recognition (2014, background): “Action recognition has been an active research area due to its wide applications [1, 33, 34, 36].”
Two-Stream SR-CNNs for Action Recognition in Videos (2016, background): “Due to its importance in many application areas such as video surveillance, content driven retrieval, robotics and HCI, human action recognition has received a lot of attention recently [20, 24, 25, 26, 27, 28, 29].”
Actionness Estimation Using Hybrid Fully Convolutional Networks (2016, background): “Most of the research works focused on the problem of action classification [7, 9, 22, 37, 38, 41], which aims at predicting an action label given a video.”