Automatic Analysis of Facial Affect: A Survey of Registration, Representation, and Recognition

By Evangelos Sariyanidi, Hatice Gunes, Andrea Cavallaro. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015

661 citations recorded by the source corpus.

Automatic affect analysis has attracted great interest in various contexts including the recognition of action units and basic or non-basic emotions. In spite of major efforts, there are several open questions on what the important cues to interpret facial expressions are and how to encode them. In this paper, we review the progress across a range of affect recognition applications to shed light on these fundamental questions. We analyse the state-of-the-art solutions by decomposing their pipelines into fundamental components, namely face registration, representation, dimensionality reduction and recognition. We discuss the role of these components and highlight the models and new trends that are followed in their design. Moreover, we provide a comprehensive analysis of facial representations by uncovering their advantages and limitations; we elaborate on the type of information they encode and discuss how they deal with the key challenges of illumination variations, registration errors, head-pose variations, occlusions, and identity bias. This survey allows us to identify open issues and to define future directions for designing real-world affect recognition systems.

Subfields: Computer Vision and Pattern Recognition, Experimental and Cognitive Psychology

View Paper: https://www.semanticscholar.org/paper/24bf94f8090daf9bda56d54e42009067839b20df

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Experimental and Cognitive Psychology

Citations in the corpus

Developing crossmodal expression recognition based on a deep neural model (2016, methodology): “One of the problems with methods that use the FACS is the amount of time necessary for extracting and recognizing the expressions, as debated by Sariyanidi, Gunes, and Cavallaro (2015), and the difficulty of representing spontaneous expressions, as discussed by Zeng, Pantic, Roisman, and Huang…”
The First Facial Landmark Tracking in-the-Wild Challenge: Benchmark and Results (2015, background): “…to achieve state-of-the-art results in a series of important computer vision applications, such as face recognition/veriﬁcation and facial expression analysis, it is important to provide an enhanced representation of the face that contains the locations of several key facial landmarks [47, 52].”
Multi-Factor Authentication: A Survey (2018, background): “A similar problem applies to early facial recognition techniques, which failed to operate without adequate light support, quality camera, etc. [197].”
OpenFace: An open source facial behavior analysis toolkit (2016, background): “A full review of work in facial landmark detection, head pose, eye gaze, and action unit estimation is outside the scope of this paper, we refer the reader to recent reviews of the field [17, 18, 30, 46, 51, 61].”
Deep Affect Prediction in-the-Wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond (2018, background): “Nevertheless, it is now accepted by the community that the facial expressions of naturalistic behaviors can be radically different from the posed ones [10, 47, 65].”
Cross-dataset learning and person-specific normalisation for automatic Action Unit detection (2015, methodology): “We refer the reader to recent surveys and challenges [6], [22], [23], [25].”
A survey of multimodal sentiment analysis (2017, background): “, [24,25], conducting sentiment analysis by computer vision is a relatively recent area of research.”
IExpressNet: Facial Expression Recognition with Incremental Classes (2020, background): “Poster Session D2: Emerging Multimedia Applications & Emotional and Social Signals in Multimedia More specifically, the aim of FER is to encode expression information from facial representations [57].”
Deep Alignment Network: A Convolutional Neural Network for Robust Face Alignment (2017, background): “Face alignment is an important component of many computer vision applications, such as face verification [28], facial emotion recognition [25], humancomputer interaction [6] and facial motion capture [12].”
Factorized Higher-Order CNNs With an Application to Spatio-Temporal Emotion Estimation (2019, background): “Unfortunately, the quality and quantity of available video data and annotation collected in naturalistic conditions is low [40].”
Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-related Applications (2016, background): “In the last years excellent surveys on automatic facial expression analysis have been published [9], [10], [11], [12].”
OpenFace 2.0: Facial Behavior Analysis Toolkit (2018, background): “Recent years have seen an increased interest in machine analysis of faces [58], [45].”
Automatic Analysis of Facial Actions: A Survey (2019, background): “More recent surveys include [154] and [43].”
Facial Expression Analysis under Partial Occlusion (2018, positive): “ Surveys of recent 2D/3D FEA methods [Bettadapura 2012], [Sandbach et al. 2012], [Owusu, et al. 2015], [Sariyanidi, et al. 2015], [Corneanu, et al. 2016].”
Survey on Emotional Body Gesture Recognition (2018, background): “The difference between words people pronounce and our understanding of their content comes from nonverbal communication also commonly called body language [1].”
Aff-Wild: Valence and Arousal ‘In-the-Wild’ Challenge (2017, background): “Nevertheless, it is now accepted by the community that the facial expressions of naturalistic behaviour could be radically different from the posed ones [9, 32, 44].”
Deep Learning for Emotion Recognition on Small Datasets using Transfer Learning (2015, background): “A very detailed and recent review can be found in [21].”
Deep Facial Expression Recognition: A Survey (2018, background): “ited in the ability to represent the complexity and subtlety of our daily affective displays [7], [8], [9], and other emotion description models, such as the facial action coding system (FACS) [10] and the continuous model using affect dimen-”