Deep Facial Expression Recognition: A Survey

By Shan Li, Weihong Deng. IEEE Transactions on Affective Computing, 2018

1,694 citations recorded by the source corpus.

With the transition of facial expression recognition (FER) from laboratory-controlled to challenging in-the-wild conditions and the recent success of deep learning techniques in various fields, deep neural networks have increasingly been leveraged to learn discriminative representations for automatic FER. Recent deep FER systems generally focus on two important issues: overfitting caused by a lack of sufficient training data and expression-unrelated variations, such as illumination, head pose, and identity bias. In this survey, we provide a comprehensive review of deep FER, including datasets and algorithms that provide insights into these intrinsic problems. First, we introduce the available datasets that are widely used in the literature and provide accepted data selection and evaluation principles for these datasets. We then describe the standard pipeline of a deep FER system with the related background knowledge and suggestions for applicable implementations for each stage. For the state-of-the-art in deep FER, we introduce existing novel deep neural networks and related training strategies that are designed for FER based on both static images and dynamic image sequences and discuss their advantages and limitations. Competitive performances and experimental comparisons on widely used benchmarks are also summarized. We then extend our survey to additional related issues and application scenarios. Finally, we review the remaining challenges and corresponding opportunities in this field as well as future directions for the design of robust deep FER systems.

Subfields: Computer Vision and Pattern Recognition, Experimental and Cognitive Psychology

View Paper: https://www.semanticscholar.org/paper/37fde06dacf825ac8b33d340eb27bfbf65893146

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Experimental and Cognitive Psychology

Citations in the corpus

Deep learning-based facial emotion recognition for human–computer interaction applications (2021, methodology): “To improve the performance of deep learning, researchers are using data augmentation [21], translations, normalizations, cropping, adding noise and, scaling techniques [22] to increase the data size.”
A Systematic Review on Affective Computing: Emotion Models, Databases, and Recent Advances (2022, methodology): “For visual modality, most studies surveyed FER [38,39,41,44] and FMEA [40].”
Facial Expression Recognition With Visual Transformers and Attentional Selective Fusion (2021, background): “Different backgrounds, illuminations and head poses are fairly common under the unconstrained conditions, which are irrelevant to facial expressions[18].”
Deep-Emotion: Facial Expression Recognition Using Attentional Convolutional Network (2019, background): “Some of the other recent works on facial expression recognition includes Multiple attention network for facial expression recognition [42], deep self-attention network for facial emotion recognition [43], and a recent survey on facial expression recognition [44].”
Region Attention Networks for Pose and Occlusion Robust Facial Expression Recognition (2019, background): “With the popularity of data-driven deep learning techniques, several recent efforts on FER have been made on the collection of large-scale datasets [19], [44], [33], and many works [50], [27], [37], [5], [1], [32] exploit deep convolutional neural networks (CNN) to improve the performance of FER.”
EMOCA: Emotion Driven Monocular Face Capture and Animation (2022, methodology): “Over the last decade, the availability of large datasets for single-image expression analysis [7, 57] and audio-visual videos [46–48] shifted the focus from manually designed features to end-to-end trained models [50].”
Generative Adversarial Networks for Face Generation: A Survey (2022, background): “[22, 35, 57, 79, 80, 125, 144, 147, 168] are designed for expression neutralization or augmentation.”
IExpressNet: Facial Expression Recognition with Incremental Classes (2020, background): “The other refers to domain gap [34]: a huge difference between the posed expression acquired in labcontrolled environment and expression in the real scenarios[40], as shown in Figure 2.”
Video-Based Facial Micro-Expression Analysis: A Survey of Datasets, Features and Algorithms (2021, background): “[125] present an extensive and detailed review of state-of-the-art deep learning techniques for facial expression recognition.”
Continual Learning for Affective Robotics: Why, What and How? (2020, background): “Although the current (deep) learning-based approaches provide high performance on affect recognition and classification benchmarks (see [12]–[14] for an overview), they are not able to translate this performance to real-world situations where robots need to dynamically interact with different users.”
Mobile Game Evaluation Method Based on Data Mining of Affective Time Series (2025, unknown): “Currently, facial expression recognition from 2D video recording is one of the most accessible state-of-the-art methods implemented in a number of computer tools (see, e.g., [23], and the survey [24] or [25] for discussion of some limiations).”