Region Attention Networks for Pose and Occlusion Robust Facial Expression Recognition

By Kai-Hong Wang, Xiaojiang Peng, Jianfei Yang, Debin Meng, Yu Qiao. IEEE Transactions on Image Processing, 2019

813 citations recorded by the source corpus.

Occlusion and pose variations, which can change facial appearance significantly, are two major obstacles for automatic Facial Expression Recognition (FER). Though automatic FER has made substantial progresses in the past few decades, occlusion-robust and pose-invariant issues of FER have received relatively less attention, especially in real-world scenarios. This paper addresses the real-world pose and occlusion robust FER problem in the following aspects. First, to stimulate the research of FER under real-world occlusions and variant poses, we annotate several in-the-wild FER datasets with pose and occlusion attributes for the community. Second, we propose a novel Region Attention Network (RAN), to adaptively capture the importance of facial regions for occlusion and pose variant FER. The RAN aggregates and embeds varied number of region features produced by a backbone convolutional neural network into a compact fixed-length representation. Last, inspired by the fact that facial expressions are mainly defined by facial action units, we propose a region biased loss to encourage high attention weights for the most important regions. We validate our RAN and region biased loss on both our built test datasets and four popular datasets: FERPlus, AffectNet, RAF-DB, and SFEW. Extensive experiments show that our RAN and region biased loss largely improve the performance of FER with occlusion and variant pose. Our method also achieves state-of-the-art results on FERPlus, AffectNet, RAF-DB, and SFEW. Code and the collected test data will be publicly available.

Subfields: Computer Vision and Pattern Recognition, Experimental and Cognitive Psychology

View Paper: https://www.semanticscholar.org/paper/a73ea666c92da841c49ab37d6382a5074e063632

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Feng Zhou
2. Yu Qiao
3. Shijian Lu
4. Shutao Li
5. Xiaojiang Peng
6. Luc Van Gool
7. Guo-qing Guo
8. Qingshan Liu

In Experimental and Cognitive Psychology

1. Fuyan Ma
2. Feng Zhou
3. Shutao Li
4. Zengqun Zhao
5. Bin Sun
6. Qingshan Liu
7. Yu Qiao
8. Xiaojiang Peng

Citations in the corpus

• Robust Lightweight Facial Expression Recognition Network with Label Distribution Training (2021, methodology): “To verify the effectiveness of the proposed method, we conduct the experiments on three popular in-the-wild facial expression datasets: RAF-DB (Li and Deng 2018), CAER-S (Lee et al. 2019), and AffectNet (Mollahosseini, Hasani, and Mahoor 2017), and five realistic occlusion and pose variation datasets”
• TransFER: Learning Relation-aware Facial Expression Representations with Transformers (2021, background): “To achieve the aforementioned goal, [19, 28] applied attention mechanisms.”
• Suppressing Uncertainties for Large-Scale Facial Expression Recognition (2020, background): “Recently, both Li et al. [25] and Wang et al. [42] have designed region-based attention networks for pose and occlusion aware FER, where the regions are either cropped from landmark points or fixed positions.”
• Distract Your Attention: Multi-Head Cross Attention Network for Facial Expression Recognition (2021, background): “EfficientFace [51] 1.28 0.15 SCN [49] 11.18 1.82 RAN [48] 11.19 14.55 PSR[50] 20.24 10.12 DACL [7] 103”
• Facial Expression Recognition With Visual Transformers and Attentional Selective Fusion (2021, methodology): “To deal with the imbalance issue, as RAN [10] and SCN [11] do, we adopt the oversampling strategy in our experiments.”