Recognizing actions by shape-motion prototype trees

By Zhe L. Lin, Zhuolin Jiang, Larry S. Davis. IEEE International Conference on Computer Vision, 2009

336 citations recorded by the source corpus.

A prototype-based approach is introduced for action recognition. The approach represents an action as a sequence of prototypes for efficient and flexible action matching in long video sequences. During training, first, an action prototype tree is learned in a joint shape and motion space via hierarchical k-means clustering; then a lookup table of prototype-to-prototype distances is generated. During testing, based on a joint likelihood model of the actor location and action prototype, the actor is tracked while a frame-to-prototype correspondence is established by maximizing the joint likelihood, which is efficiently performed by searching the learned prototype tree; then actions are recognized using dynamic prototype sequence matching. Distance matrices used for sequence matching are rapidly obtained by look-up table indexing, which is an order of magnitude faster than brute-force computation of frame-to-frame distances. Our approach enables robust action matching in very challenging situations (such as moving cameras, dynamic backgrounds) and allows automatic alignment of action sequences. Experimental results demonstrate that our approach achieves recognition rates of 91.07% on a large gesture dataset (with dynamic backgrounds), 100% on the Weizmann action dataset and 95.77% on the KTH action dataset.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/5894aca916753fabd97cda127e8f9ce230c3fb61

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Human-Computer Interaction

Citations in the corpus

Instructing people for training gestural interactive systems (2012, background): “CGD2011* 2011 [1] assorted (M+I) 15 NR 30,000 NR upper body performance HMDB* 2011 [12] natural (M+I) 51 (6849) 6849 actors or none full/upper body performance Keck military* 2009 [14] signalling (M) 14 3 294 Handbook pictograms hands frame-ranges UCF YouTube* 2009 [15] sports (I) 11 (1168) 1168 non”
A Hough transform-based voting framework for action recognition (2010, methodology): “Actions are classified by weighting the n-nearest neighbours and localized either from a foreground segmentation [ 17 ] or from the features’ spatial information either stored during the training stage [21] or extracted at the test stage [29].”
Recent methods and databases in vision-based hand gesture recognition: A review (2015, methodology): “Keck Gesture Dataset The gesture dataset consists of 14 dynamic gestures, which are subsets of military signals (like turn left, go back, and speed up) [136].”
Learning a Deep Model for Human Action Recognition from Novel Viewpoints (2016, background): “Based on this observation, several visual representations have been proposed for discriminative human action recognition such as space-time pattern templates [1], shape matching [2]–[4], spatio-temporal interest points [5]– [10], and motion trajectories based representation [11]– [14].”
Cross-View Action Recognition via Transferable Dictionary Learning (2016, methodology): “We extract the shape descriptor [25] for each video frame.”
Learning spatiotemporal graphs of human activities (2011, methodology): “Related work – Volumetric video representations have been used for exemplar based activity recognition [8] [13].”
Characteristic Kernels on Structured Domains Excel in Robotics and Human Action Recognition (2010, background): “In [15] and [11], the authors benefited from stronger feature vectors as combination of shape and motion and reported high accuracy rates rather than motion feature which is used here.”
Unsupervised learning of event AND-OR grammar and semantics from video (2011, background): “Event analysis has gone a long way from modeling transient or periodic action [10, 3], to longer events [13, 6, 8], achieving promising recognition performance.”
Learning View-Invariant Sparse Representations for Cross-View Action Recognition (2013, methodology): “In order to accurately recognize human actions, most existing approaches focus on developing different discriminative features, such as spatio-temporal interest point (STIP) based features [32, 2, 13, 19], shape [16, 3, 21] and optical flow based features [5, 17, 16].”
Recognizing human actions by attributes (2011, background): “More recently, great progress has been made by introducing more descriptive action representations such as space-time pattern templates[1, 21], 2D shape matching [33, 7], optical flow patterns [6, 33 ], trajectory-based representation [27], and bag-of-video-words [24, 19, 20].”
Visual event recognition in videos by learning from web data (2010, background): “Promising results [1, 13, 19, 22] have been reported on video data sets under controlled settings, such as Weizman [1] and KTH [22] data sets.”
Making Action Recognition Robust to Occlusions and Viewpoint Changes (2010, methodology): “Most state-of-the-art approaches compute image-sequence descriptors based on variants of either sparse interest points [3,11,17,20,24] or dense holistic features [9,13,19,22,23].”
Discovering Motion Primitives for Unsupervised Grouping and One-Shot Learning of Human Actions, Gestures, and Expressions (2013, neutral): “dataset [24] is already partitioned into training and test sets.”
Human gesture recognition on product manifolds (2012, methodology): “One of the key advantages of our method is its direct use of raw pixels while the prototype-tree (Lin et al., 2009), MMI-2+SIFT (Qiu et al.”
Exploiting Image-trained CNN Architectures for Unconstrained Video Classification (2015, background): “[21]), descriptors from spatiotemporal interest points ( e.”
Cross-view action recognition via view knowledge transfer (2011, background): “actions. Some leading representations include space-time pattern templates [3, 27], shape features [ 36 , 24, 7, 19], interest point based representations [15, 26, 29, 20], learned geometrical models of the human body parts [12], and motion/optical flow patterns [1, 25, 36].”