Action Recognition by Hierarchical Sequence Summarization

By Yale Song, Louis-philippe Morency, Randall Davis. 2013

101 citations recorded by the source corpus.

Recent progress has shown that learning from hierarchical feature representations leads to improvements in various computer vision tasks. Motivated by the observation that human activity data contains information at various temporal resolutions, we present a hierarchical sequence summarization approach for action recognition that learns multiple layers of discriminative feature representations at different temporal granularities. We build up a hierarchy dynamically and recursively by alternating sequence learning and sequence summarization. For sequence learning we use CRFs with latent variables to learn hidden spatio-temporal dynamics, for sequence summarization we group observations that have similar semantic meaning in the latent space. For each layer we learn an abstract feature representation through non-linear gate functions. This procedure is repeated to obtain a hierarchical sequence summary representation. We develop an efficient learning method to train our model and show that its complexity grows sub linearly with the size of the hierarchy. Experimental results show the effectiveness of our approach, achieving the best published results on the Arm Gesture and Canal9 datasets.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Actions ~ Transformations (2015, background): “Mid-to high-level representation are also used to model complex actions [41, 55, 36, 60, 14, 24].”
Connectionist Temporal Modeling for Weakly Supervised Action Labeling (2016, background): “Our goal of understanding the temporal structure of video is related to [10,31,34,37,39,40].”
Asynchronous Temporal Fields for Action Recognition (2016, background): “Recently there has been a push towards mid-level representations of video [37, 46, 13, 20], that capture beyond local properties.”
Multi-attention Recurrent Network for Human Communication Comprehension (2018, methodology): “…: Extensions of Hidden Markov Models (Baum and Petrie 1966) and Hidden Conditional Random Fields (Quattoni et al. 2007; Morency, Quattoni, and Darrell 2007) have been proposed for learning from multiple different views (modalities) (Song, Morency, and Davis 2012; Song, Morency, and Davis 2013).”
Rank Pooling for Action Recognition (2015, background): “Since the ranking machines act on frame content, they actually capture both the appearance and their evolution over time.”
Minimum uncertainty latent variable models for robot recognition of sequential human activities (2017, methodology): “Given such a clique tree, our implementation of the belief propagation algorithm is present in Algorithm 1, with the computational complexity satisfies the following lemma (similar to previous works [2], [20]): Lemma 3: Algorithm 1 requires O(|E||Y||H|2) to compute the quantity P̃ (y|x;θ) = ∑h P̃ (y”
Spatial, Structural and Temporal Feature Learning for Human Interaction Prediction (2016, methodology): “Traditional sequential models such as Hidden Markov models (HMMs) [15] and Conditional Random Fields (CRFs)[16] have been successfully used for action recognition [17, 18, 19, 20].”
Modeling video evolution for action recognition (2015, methodology): “using HMM [39, 40], CRF-based methods [31] or deep networks [34]), the impact of these on action recognition performance so far has been somewhat disappointing.”
Multimodal Language Analysis with Recurrent Multistage Fusion (2018, background): “, 2012), and multi-view Hierarchical Sequence Summarization models that recursively build up hierarchical representations (Song et al., 2013).”
SSNet: Scale Selection Network for Online 3D Action Prediction (2018, background): “Specifically, previous works [58, 5] suggest that there are often hierarchical structures in the motion patterns over temporal axis, which play an important role in action analysis, hence, we design our convolutional model with a hierarchical structure.”
Discriminative Hierarchical Rank Pooling for Activity Recognition (2016, methodology): “Hierarchical methods have also been used in activity recognition [3, 26].”
Human Action Recognition With Video Data: Research and Evaluation Challenges (2014, background): “HMMs (see [20], [25], [43], [66]–[68], and [79]–[84]), Bayesian networks (see [24], [25], [65]–[67], [69], and [85]–[90]), conditional random fields (see [91]–[93]), and fuzzy-based systems”
Effective Active Skeleton Representation for Low Latency Human Action Recognition (2016, methodology): “[42] proposed a method with hierarchical sequence learning and summarization to classify human actions in latent space.”
Memory Fusion Network for Multi-view Sequential Learning (2018, methodology): “Multi-view variations of these models have been proposed including Multi-view HCRFs where the potentials of the HCRF are changed to facilitate multiple views (Song, Morency, and Davis 2012; Song, Morency, and Davis 2013).”
Learning Factorized Multimodal Representations (2018, methodology): “MVLDHCRF: (Morency et al., 2007) is a variation of the MV-HCRF model that uses LDHCRF instead of HCRF. MV-HSSHCRF: (Song et al., 2013) further extends EF-HSSHCRF by performing Multiview hierarchical sequence summary representation.”