Training deep networks for facial expression recognition with crowd-sourced label distribution

By Emad Barsoum, Cha Zhang, Cristian Canton-Ferrer, Zhengyou Zhang. International Conference on Multimodal Interaction, 2016

869 citations recorded by the source corpus.

Crowd sourcing has become a widely adopted scheme to collect ground truth labels. However, it is a well-known problem that these labels can be very noisy. In this paper, we demonstrate how to learn a deep convolutional neural network (DCNN) from noisy labels, using facial expression recognition as an example. More specifically, we have 10 taggers to label each input image, and compare four different approaches to utilizing the multiple labels: majority voting, multi-label learning, probabilistic label drawing, and cross-entropy loss. We show that the traditional majority voting scheme does not perform as well as the last two approaches that fully leverage the label distribution. An enhanced FER+ data set with multiple labels for each face image will also be shared with the research community.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

View Paper: https://www.semanticscholar.org/paper/33668f2105678d38b2986176058eb54cd4bbd9a6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Facial Expression Recognition With Visual Transformers and Attentional Selective Fusion (2021, methodology): “based methods including CSLD [15], ResNet+VGG [57], SHCNN [58], LDR [59], as well as two recent state-of-the-art methods(RAN [10], SCN [11]).”
Emotion Recognition in Speech using Cross-Modal Transfer in the Wild (2018, methodology): “The resulting model is then finetuned on the FERplus dataset [10] for emotion recognition.”
Deep-Emotion: Facial Expression Recognition Using Attentional Convolutional Network (2019, methodology): “single looped network, citing the two parts’ need for feedback from each other. They used their Boosted Deep Belief Network (BDBN) on CK+ and JAFFE, achieving state-of-the-art accuracy. Barsoum et al [21] worked on using a deep CNN on noisy labels acquired via crowdsourcing for ground truth images. ”
Suppressing Uncertainties for Large-Scale Facial Expression Recognition (2020, methodology): “…have made signiﬁcant progress on facial expression recognition (FER) with algorithms and large-scale datasets, where datasets can be collected in lab-∗ oratory or in the wild, such as CK+ [29], MMI [39], Oulu-CASIA [47], SFEW/AFEW [10], FERPlus [4], AffectNet [32], EmotioNet [11], RAF-DB [22], etc.”
Deep Facial Expression Recognition: A Survey (2018, methodology): “A reasonable approach is to employ crowd-sourcing models [34], [45], [208] under the guidance”
Real-time Convolutional Neural Networks for emotion and gender classification (2017, methodology): “We evaluate our models in three datasets: FER 2013 [7] and FER+ [1] datasets for emotion classification.”
FrugalML: How to Use ML Prediction APIs More Accurately and Cheaply (2020, neutral): “Dataset Size # Classes Dataset Size # Classes Tasks FER+ [20] 6358 7 RAFDB [39] 15339 7 FER EXPW [59] 31510 7 AFFECTNET [42] 287401 7 YELP [18] 20000 2 SHOP [15] 62774 2 SA IMDB [41] 25000 2 WAIMAI [17] 11987 2 DIGIT [5] 2000 10 AUDIOMNIST [21] 30000 10 STT FLUENT [40] 30043 31 COMMAND [52] 64727 31”
Addressing bias in machine learning algorithms: A pilot study on emotion recognition for intelligent systems (2017, methodology): “’ To establish a baseline on the capabilities of learning and inference for a minority class, we evaluate the emotion recognition results associated with children’s facial expressions using the Microsoft Emotion API, a deep learning neural network [18].”
What you Feel, Is what you like Influence of Message Appeals on Customer Engagement on Instagram (2020, methodology): “The Microsoft Face API is based on deep convolutional neural networks, and when tested on a standardized dataset of Facial Action Coding System (FACS) (Ekman 1997) it achieves accuracy for most emotions between 67% and 94% (Barsoum et al. 2016).”
TransFER: Learning Relation-aware Facial Expression Representations with Transformers (2021, background): “FERPlus [3] is extended from FER2013 [13] which is a large-scale dataset collected by APIs in the Google image search.”
Region Attention Networks for Pose and Occlusion Robust Facial Expression Recognition (2019, methodology): “From the test set of FERPlus [4] and the test set of AffectNet [44], we collect the Occlusion-FERPlus, Pose-FERPlus, Occlusion-AffectNet, and Pose-AffectNet for testing.”
MMGCN: Multimodal Fusion via Deep Graph Convolution Network for Emotion Recognition in Conversation (2021, methodology): “The visual facial expression features are extracted using a DenseNet (Huang et al., 2015) pre-traind on the Facial Expression Recognition Plus (FER+) corpus (Barsoum et al., 2016).”
Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities (2021, background): “, 2017) which is trained based on the Facial Expression Recognition Plus (FER+) corpus (Barsoum et al., 2016).”