Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-related Applications

By Ciprian Adrian Corneanu, Marc Oliu, Jeffrey F. Cohn, Sergio Escalera. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016

510 citations recorded by the source corpus.

Facial expressions are an important way through which humans interact socially. Building a system capable of automatically recognizing facial expressions from images and video has been an intense field of study in recent years. Interpreting such expressions remains challenging and much research is needed about the way they relate to human affect. This paper presents a general overview of automatic RGB, 3D, thermal and multimodal facial expression analysis. We define a new taxonomy for the field, encompassing all steps from face detection to facial expression recognition, and describe and classify the state of the art methods accordingly. We also present the important datasets and the bench-marking of most influential methods. We conclude with a general discussion about trends, important questions and future lines of research.

Subfields: Computer Vision and Pattern Recognition, Experimental and Cognitive Psychology

View Paper: https://www.semanticscholar.org/paper/858ddff549ae0a3094c747fb1f26aa72821374ec

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Experimental and Cognitive Psychology

Citations in the corpus

ChaLearn looking at people: A review of events and resources (2017, background): “A total of 17 papers were published withiin this Special Issue at IEEE TPAMI [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41], [42], [43], [44], [45], [46], [47].”
Deep Facial Expression Recognition: A Survey (2018, methodology): “modalities, such as audio and physiological channels, have also been used in multimodal systems [18] to assist in the recognition of expression.”
Deep Affect Prediction in-the-Wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond (2018, background): “Nevertheless, it is now accepted by the community that the facial expressions of naturalistic behaviors can be radically different from the posed ones [10, 47, 65].”
Automatic Analysis of Facial Actions: A Survey (2019, background): “More recent surveys include [154] and [43].”
AVEC 2019 Workshop and Challenge: State-of-Mind, Detecting Depression with AI, and Cross-Cultural Affect Recognition (2019, background): “A common belief in facial expression recognition is that emotional expressions have a large degree of universality across cultures [10, 16].”
Aff-Wild: Valence and Arousal ‘In-the-Wild’ Challenge (2017, background): “Nevertheless, it is now accepted by the community that the facial expressions of naturalistic behaviour could be radically different from the posed ones [9, 32, 44].”
Facial Expression Analysis under Partial Occlusion (2018, unknown): “ Surveys of recent 2D/3D FEA methods [Bettadapura 2012], [Sandbach et al. 2012], [Owusu, et al. 2015], [Sariyanidi, et al. 2015], [Corneanu, et al. 2016].”
Video-Based Facial Micro-Expression Analysis: A Survey of Datasets, Features and Algorithms (2021, background): “5 and 4 seconds [4], and are made using underlying facial movements that cover a large facial area [3]; thus, they can be clearly distinguished from noise such as eye blinks.”
Survey on Emotional Body Gesture Recognition (2018, background): “sions or speech is encouraged to consult dedicated surveys [1], [13], [14].”
Robust Lightweight Facial Expression Recognition Network with Label Distribution Training (2021, background): “In the ﬁeld of human-computer interaction (HCI), the environments adaptive systems, socially aware systems, or robots with social skills can be built by detecting user’s affective states (Maat and Pantic 2007; DeVault et al. 2014; Corneanu et al. 2016).”
IExpressNet: Facial Expression Recognition with Incremental Classes (2020, background): “One is affective gap [12, 26, 43], 𝑖.𝑒. the inconsistency between low-level visual features and high-level expression information.”