Occlusion Aware Facial Expression Recognition Using CNN With Attention Mechanism

By Yong Li, Jiabei Zeng, Shiguang Shan, Xilin Chen. IEEE Transactions on Image Processing, 2019

796 citations recorded by the source corpus.

Facial expression recognition in the wild is challenging due to various unconstrained conditions. Although existing facial expression classifiers have been almost perfect on analyzing constrained frontal faces, they fail to perform well on partially occluded faces that are common in the wild. In this paper, we propose a convolution neutral network (CNN) with attention mechanism (ACNN) that can perceive the occlusion regions of the face and focus on the most discriminative un-occluded regions. ACNN is an end-to-end learning framework. It combines the multiple representations from facial regions of interest (ROIs). Each representation is weighed via a proposed gate unit that computes an adaptive weight from the region itself according to the unobstructedness and importance. Considering different RoIs, we introduce two versions of ACNN: patch-based ACNN (pACNN) and global–local-based ACNN (gACNN). pACNN only pays attention to local facial patches. gACNN integrates local representations at patch-level with global representation at image-level. The proposed ACNNs are evaluated on both real and synthetic occlusions, including a self-collected facial expression dataset with real-world occlusions, the two largest in-the-wild facial expression datasets (RAF-DB and AffectNet) and their modifications with synthesized facial occlusions. Experimental results show that ACNNs improve the recognition accuracy on both the non-occluded faces and occluded faces. Visualization results demonstrate that, compared with the CNN without Gate Unit, ACNNs are capable of shifting the attention from the occluded patches to other related but unobstructed ones. ACNNs also outperform other state-of-the-art methods on several widely used in-the-lab facial expression datasets under the cross-dataset evaluation protocol.

Subfields: Computer Vision and Pattern Recognition, Experimental and Cognitive Psychology

View Paper: https://www.semanticscholar.org/paper/b5bb7e12a15b57b4d307e742da127a74596d0c7c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Feng Zhou
2. Weihong Deng
3. Yu Qiao
4. Shutao Li
5. Xiaojiang Peng
6. Fuyan Ma
7. Bin Sun
8. Qingshan Liu

In Experimental and Cognitive Psychology

1. Fuyan Ma
2. Shutao Li
3. Bin Sun
4. Feng Zhou
5. Zengqun Zhao
6. Qingshan Liu
7. Yu Qiao
8. Xiaojiang Peng

Citations in the corpus

• Deep Facial Expression Recognition: A Survey (2018, background): “Other studies [122], [123], [124], [125], [126], [127] proposed automatically learning the”
• Context-Aware Emotion Recognition Networks (2019, background): “Previous researches for emotion recognition based on handcrafted features [4, 5] or deep networks [6, 7, 8] have mainly focused on the perception of the facial expression, based on the assumption that facial images are one of the most discriminative features of emotional responses.”
• Facial Expression Recognition With Visual Transformers and Attentional Selective Fusion (2021, methodology): “Since gACNN [9], RAN [10] and SCN [11] did not report the specific expression recognition accuracy or the confusion matrices, the corresponding places are marked with ’-’ in Table 1.”
• Region Attention Networks for Pose and Occlusion Robust Facial Expression Recognition (2019, methodology): “We use the RAN with default setting, and train it using the same training data as [34].”
• Suppressing Uncertainties for Large-Scale Facial Expression Recognition (2020, background): “IPA2LT [43] introduces the latent ground-truth idea for training with inconsistent annotations across different FER datasets. gaCNN [24] leverages a patch-based attention network and a global network.”
• Robust Lightweight Facial Expression Recognition Network with Label Distribution Training (2021, methodology): “To verify the effectiveness of the proposed method, we conduct the experiments on three popular in-the-wild facial expression datasets: RAF-DB (Li and Deng 2018), CAER-S (Lee et al. 2019), and AffectNet (Mollahosseini, Hasani, and Mahoor 2017), and five realistic occlusion and pose variation datasets”