Dynamics of facial expression: recognition of facial actions and their temporal segments from face profile image sequences

By Maja Pantic, Ioannis Patras. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 2006

642 citations recorded by the source corpus.

Automatic analysis of human facial expression is a challenging problem with many applications. Most of the existing automated systems for facial expression analysis attempt to recognize a few prototypic emotional expressions, such as anger and happiness. Instead of representing another approach to machine analysis of prototypic facial expressions of emotion, the method presented in this paper attempts to handle a large range of human facial behavior by recognizing facial muscle actions that produce expressions. Virtually all of the existing vision systems for facial muscle action detection deal only with frontal-view face images and cannot handle temporal dynamics of facial actions. In this paper, we present a system for automatic recognition of facial action units (AUs) and their temporal models from long, profile-view face image sequences. We exploit particle filtering to track 15 facial points in an input face-profile sequence, and we introduce facial-action-dynamics recognition from continuous video input using temporal rules. The algorithm performs both automatic segmentation of an input video into facial expressions pictured and recognition of temporal segments (i.e., onset, apex, offset) of 27 AUs occurring alone or in a combination in the input face-profile video. A recognition rate of 87% is achieved.

Subfields: Computer Vision and Pattern Recognition, Experimental and Cognitive Psychology

View Paper: https://www.semanticscholar.org/paper/36c2849d8c504625f37164b62638584e58a8407d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Experimental and Cognitive Psychology

Citations in the corpus

Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-related Applications (2016, background): “In later years various ways of dynamic representation were also explored like tracking geometrical deformations across frames in RGB [114], [116] and 3D [119], [128] or directly extracting features”
Observer-based measurement of facial expression with the Facial Action Coding System. (2007, background): “Examples from each of three separate groups include Bartlett et al. (1999), Cohn et al. (1999), and Pantic & Patras (2006).”
Facial Expression Analysis (2011, background): “Approaches that use only geometric features (or their derivatives) mostly rely on detecting sets of fiducial facial points [94, 96, 121], a connected face mesh or ac-”
Automatic facial expression recognition using features of salient facial patches (2015, methodology): “, [5], [6], [7], [8]) used shape models based on a set of characteristic points on the face to classify the expressions.”
Multimodal semi-automated affect detection from conversational cues, gross body language, and facial features (2010, background): “There are also some important differences between real and posed affective expressions (Afzal and Robinson 2009; Cohn and Schmidt 2004; Ekman et al. 1990; Pantic and Patras 2006), hence, insights gleaned from studies on acted expressions might not generalize to real contexts.”
Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications (2010, background): “There are also some important differences between real and posed affective expressions [79], [172], [173], [174]; hence, insights gleaned from studies on acted expressions might not generalize to real contexts.”
Facial expression recognition based on Local Binary Patterns: A comprehensive study (2009, methodology): “Facial geometry analysis has been widely exploited in facial representation [9,10,16,17,24], where shapes and locations of facial components are extracted to represent the face geometry.”
Social signal processing: Survey of an emerging domain (2009, methodology): “Yet, it seems that using both geometric and appearance features might be the best choice in the case of certain facial expressions [136].”
Human computing and machine understanding of human behavior: a survey (2006, background): “A number of promising prototype systems have been proposed recently that can recognize 15 to 27 AUs (from a total of 44 AUs) in either (near-) frontal view or profile view face image sequences [31], [43].”
Automatic Recognition of Facial Actions in Spontaneous Expressions (2006, background): “More recent work from this group presented approaches to measuring facial actions in profile views, and recognizing expression dynamics from temporal rules [35].”
Real-time classification of evoked emotions using facial feature tracking and physiological responses (2008, background): “While much of the work analyzes the front view of the face, Pantic and Patras (2006) developed a system for automatic recognition of facial action units and analyzed those units using temporal models from profile-view face image sequences.”
Facial Expression Recognition (2009, background): “eigenfaces, recent studies show that in some cases geometric features can outperform the appearancebased ones [4, 11].”
Effects of Dynamic Aspects of Facial Expressions: A Review (2013, methodology): “The first advances have already been made by incorporating dynamic data into the measurement and modeling of facial actions (e.g., Cosker, Krumhuber, & Hilton, 2011; Pantic & Patras, 2006; see also Calvo & D’Mello, 2010; Kappas, 2010).”