Facial expression recognition from video sequences: temporal and static modeling

By Ira Cohen, Nicu Sebe, Ashutosh Garg, Lawrence S. Chen, Thomas S. Huang. Computer Vision and Image Understanding, 2003

930 citations recorded by the source corpus.

The most expressive way humans display emotions is through facial expressions. In this work we report on several advances we have made in building a system for classification of facial expressions from continuous video input. We introduce and test different Bayesian network classifiers for classifying expressions from video, focusing on changes in distribution assumptions, and feature dependency structures. In particular we use Naive-Bayes classifiers and change the distribution from Gaussian to Cauchy, and use Gaussian Tree-Augmented Naive Bayes (TAN) classifiers to learn the dependencies among different facial motion features. We also introduce a facial expression recognition from live video input using temporal cues. We exploit the existing methods and propose a new architecture of hidden Markov models (HMMs) for automatically segmenting and recognizing human facial expression from video sequences. The architecture performs both segmentation and recognition of the facial expressions automatically using a multi-level architecture composed of an HMM layer and a Markov model layer. We explore both person-dependent and person-independent recognition of expressions and compare the different methods.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/1f1970df3155799ae35fc47e71c3047198f532d9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Xiaogang Wang
3. Shaogang Gong
4. Yi Yang
5. Jitendra Malik
6. Dacheng Tao
7. Cordelia Schmid
8. Yu Qiao

Citations in the corpus

• Multimodal Human Computer Interaction: A Survey (2005, background): “Hierarchical HMM-based systems [32] have been shown to work well for facial expression recognition.”
• Human-Computer Interaction: Overview on State of the Art (2008, neutral): “Facial expression analysis generally deals with recognition of emotions visually [37] [38] [39].”
• Facial Action Unit Recognition by Exploiting Their Dynamic and Semantic Relationships (2007, methodology): “The local methods [4], [9], [10], [3], [11], [12], [13], [14], [15], [16] focus on the facial features locally.”
• Robust facial expression recognition using local binary patterns (2005, methodology): “The best performance of 73.2% was obtained by using Tree-Augmented-Naive Bayes (TAN) classifiers.”
• Automatic facial expression recognition using features of salient facial patches (2015, methodology): “If we collect its histograms in 32 bins, then we are basically grouping the patterns [0, 7], [8, 15], [16, 23] ; .”
• Stretchable, Transparent, Ultrasensitive, and Patchable Strain Sensor for Human-Machine Interfaces Comprising a Nanohybrid of Carbon Nanotubes and Conductive Elastomers. (2015, background): “Eye movements, wrinkling of the forehead, and many other expressions can indicate intentions and the physiological state of humans.(42) To investigate the ability of the sensor to function as an emotion detector, sensors were attached to the forehead and skin near themouth of a subject, and sensor r”
• Short-term emotion assessment in a recall paradigm (2009, background): “Detection of human emotions can be realized by monitoring facial expressions (Cohen et al., 2003; Cowie et al., 2001; Fasel and Luettin, 2003), speech (Cowie, 2000; Ververidis and Kotropoulos, 2006), postures (Coulson, 2004; Kapoor et al.”
• A 3D facial expression database for facial behavior research (2006, methodology): “A number of techniques were successfully developed using 2D static images or video sequences, including machine vision techniques [44, 12, 10, 21, 4, 41, 42] and machine learning techniques [1, 20, 5, 6, 45].”
• Emotion-Induced Engagement in Internet Video Advertisements (2012, methodology): “Hit rates of 86% and 94%, respectively, were assessed using cross-validation for a test subset of the Cohn-Kanade database (Cohen et al. 2003).”
• Real-Time Inference of Complex Mental States from Facial Expressions and Head Gestures (2004, methodology): “al [5] use hierarchical HMMs to automatically segment an arbitrary long video sequence into different expression segments.”
• Facial Expression Recognition in Image Sequences Using Geometric Deformation Features and Support Vector Machines (2007, methodology): “A method for static and dynamic segmentation and classification of facial expressions is proposed in [ 7 ].”
• Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-related Applications (2016, methodology): “Most of these early works used geometric representations, like vectors for describing the motion of the face [134], active contours for describing the shape of the mouth and eyebrows [137], or deformable 2D mesh models [116].”
• Neighborhood repulsed metric learning for kinship verification (2012, neutral): “Representative examples include face recognition [5], [8], [10], [20], [22], [27], [28], [35], [37], [40], [41], [42], [43], [54], [55], [56], [58], [64], [65], [66], [67], [68], facial expression recognition [11], [18], [70], facial age estimation [19], [21], [24], [25], [31], [39], gender classifi”
• Toward Practical Smile Detection (2009, neutral): “Common approaches include static image texture analysis [5], [6], feature point-based expression classifiers [7], [8], [9], [10], 3D face modeling [11], [12], and dynamic analysis of video sequences [13], [14], [15], [16], [17], [18].”