Latent-Dynamic Discriminative Models for Continuous Gesture Recognition

By Louis-philippe Morency, Ariadna Quattoni, Trevor Darrell. 2007 IEEE Conference on Computer Vision and Pattern Recognition, 2007

428 citations recorded by the source corpus.

Many problems in vision involve the prediction of a class label for each frame in an unsegmented sequence. In this paper, we develop a discriminative framework for simultaneous sequence segmentation and labeling which can capture both intrinsic and extrinsic class dynamics. Our approach incorporates hidden state variables which model the sub-structure of a class sequence and learn dynamics between class labels. Each class label has a disjoint set of associated hidden states, which enables efficient training and inference in our model. We evaluated our method on the task of recognizing human gestures from unsegmented video streams and performed experiments on three different datasets of head and eye gestures. Our results demonstrate that our model compares favorably to Support Vector Machines, Hidden Markov Models, and Conditional Random Fields on visual gesture recognition tasks.

Subfields: Computer Vision and Pattern Recognition, Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/e4d628ea1c03b9f70b1a23a61129aeee91e35062

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

The Moving Pose: An Efficient 3D Kinematics Descriptor for Low-Latency Action Recognition and Detection (2013, methodology): “In the context of action detection (discussed later in 5.6), our improvement over LDCRF in terms of per-frame classification rate, is 57.3% over 54%.”
Survey on 3D Hand Gesture Recognition (2016, background): “For 2D continuous hand gesture recognition, both direct [145] and indirect [23], [146], [147] approaches have been well investigated.”
Structured Learning and Prediction in Computer Vision (2011, methodology): “2 Gesture recognition with a chain-structured latent variable conditional random field [106].”
One-shot learning gesture recognition from RGB-D data using bag of features (2013, methodology): “Later, the latent-dynamic conditional field (LDCRF) model (Morency et al., 2007) was proposed, which combines the strengths of CRFs and HCRFs by capturing both extrinsic dynamics and intrinsic sub-structure.”
Factorized Multimodal Transformer for Multimodal Sequential Learning (2019, methodology): “…learning task is treated as a unimodal one and tackled using unimodal sequential models such as Hidden Markov Models (HMMs) (Baum & Petrie, 1966), Hidden Conditional Random Fields (HCRFs) (Quattoni et al., 2007; Morency et al., 2007), and RNNs (e.g. LSTMs Hochreiter & Schmidhuber (1997)).”
Multi-attention Recurrent Network for Human Communication Comprehension (2018, background): “Multi-view Learning : Extensions of Hidden Markov Models (Baum and Petrie 1966) and Hidden Conditional Random Fields (Quattoni et al. 2007; Morency, Quattoni, and Darrell 2007) have been proposed for learning from multiple different views (modalities) (Song, Morency, and Davis 2012; Song, Morency, a”
Dependency Tree-based Sentiment Classification using CRFs with Hidden Variables (2010, methodology): “This parameter estimation method is same to one used for Latent-Dynamic Conditional Random Field (Morency et al., 2007).”
Enhanced Sequence Matching for Action Recognition from 3D Skeletal Data (2014, neutral): “Another methods employ generative model [14, 15].”
Learning Factorized Multimodal Representations (2018, methodology): “EF-LDHCRF (Latent Discriminative HCRFs) (Morency et al., 2007) are a class of models that learn hidden states in a CRF using a latent code between observed concatenated input and hidden output.”
Motion Recognition Employing Multiple Kernel Learning of Fisher Vectors Using Local Skeleton Features (2015, neutral): “Latent-Dynamic CRF [12] 64.”
Minimum uncertainty latent variable models for robot recognition of sequential human activities (2017, methodology): “(8) can be very efficiently computed using belief propagation based on the data structure of a clique tree, which is commonly applied to perform inference in conventional HCRFs [23], [24], [25], [2], [26], Algorithm 1: Sum-Product Belief Propagation Input : HCRF’s graph G=(V, E), graph potentials ψ(”
Hidden Markov Model on a unit hypersphere space for gesture trajectory recognition (2014, methodology): “…using the InteractPlay database (Just and Marcel, 2009) with state-of-the-art techniques such as the Threshold HMM (THMM) (Lee and Kim, 1999), Conditional Random Fields (CRF) (Lafferty et al., 2001), Hidden CRF (HCRF) (Wang et al., 2006), and Latent-Dynamic CRF (LDCRF) (Morency et al., 2007).”
Towards multimodal sentiment analysis: harvesting opinions from the web (2011, background): “Human communication is a dynamic process and explicitly modeling this dynamic has been shown to improve tasks such as gestures recognition [17] and speech recognition [21].”
3D Gestural Interaction: The State of the Field (2013, background): “This approach builds upon the HCRF by providing the ability to model the substructure of a gesture label and learn the dynamics between labels, which helps in recognizing gestures from unsegmented data [82].”
Video-based Sign Language Recognition without Temporal Segmentation (2018, neutral): “Methods Accuracy LSTM (Venugopalan et al. 2014) 0.736 S2VT (Venugopalan et al. 2015) 0.745 LSTM-A (Yao et al. 2015) 0.757 LSTM-E (Pan et al. 2015) 0.768 HAN (Yang et al. 2016) 0.793 CRF (Lafferty, McCallum, and Pereira ) 0.686 LDCRF (Morency, Quattoni, and Darrell 2007) 0.704 DTW-HMM (Zhang, Zhou, a”
Memory Fusion Network for Multi-view Sequential Learning (2018, methodology): “Morency2007 ( # ): Latent Discriminative Hidden Conditional Random Fields (LDHCRFs) are a class of models that learn hidden states in a Conditional Random Field us-ing a latent code between observed input and hidden output (Morency, Quattoni, and Darrell 2007).”