Deep Affect Prediction in-the-Wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond

By Dimitrios Kollias, Panagiotis Tzirakis, Mihalis A. Nicolaou, Athanasios Papaioannou, Guoying Zhao, Björn Schuller, Irene Kotsia, Stefanos P. Zafeiriou. International Journal of Computer Vision, 2018

518 citations recorded by the source corpus.

Automatic understanding of human affect using visual signals is of great importance in everyday human–machine interactions. Appraising human emotional states, behaviors and reactions displayed in real-world settings, can be accomplished using latent continuous dimensions (e.g., the circumplex model of affect). Valence (i.e., how positive or negative is an emotion) and arousal (i.e., power of the activation of the emotion) constitute popular and effective representations for affect. Nevertheless, the majority of collected datasets this far, although containing naturalistic emotional states, have been captured in highly controlled recording conditions. In this paper, we introduce the Aff-Wild benchmark for training and evaluating affect recognition algorithms. We also report on the results of the First Affect-in-the-wild Challenge (Aff-Wild Challenge) that was recently organized in conjunction with CVPR 2017 on the Aff-Wild database, and was the first ever challenge on the estimation of valence and arousal in-the-wild. Furthermore, we design and extensively train an end-to-end deep neural architecture which performs prediction of continuous emotion dimensions based on visual cues. The proposed deep learning architecture, AffWildNet, includes convolutional and recurrent neural network layers, exploiting the invariant properties of convolutional features, while also modeling temporal dynamics that arise in human behavior via the recurrent layers. The AffWildNet produced state-of-the-art results on the Aff-Wild Challenge. We then exploit the AffWild database for learning features, which can be used as priors for achieving best performances both for dimensional, as well as categorical emotion recognition, using the RECOLA, AFEW-VA and EmotiW 2017 datasets, compared to all other methods designed for the same goal. The database and emotion recognition models are available at http://ibug.doc.ic.ac.uk/resources/first-affect-wild-challenge.

Subfields: Artificial Intelligence, Experimental and Cognitive Psychology

View Paper: https://www.semanticscholar.org/paper/80371062f40dbaf17917ab4cd947fc67466e634f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Experimental and Cognitive Psychology

Citations in the corpus

Privileged Information for Modeling Affect In The Wild (2021, background): “Indicatively, AffWildNet [1], [24] effectively combines CNNs and recurrent neural networks to accurately capture the face dynamics and achieve the best performance in [20].”
AVEC 2019 Workshop and Challenge: State-of-Mind, Detecting Depression with AI, and Cross-Cultural Affect Recognition (2019, methodology): “For these, we employed a VGG-16 [69] network and a ResNet-50 network [24] that are pre-trained with the Affwild dataset [30].”
Factorized Higher-Order CNNs With an Application to Spatio-Temporal Emotion Estimation (2019, background): “This task is the subject of most of the recent research in affect estimation [37, 19, 23, 50] and is the focus of this paper.”
Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace (2019, methodology): “We extend the Aff-Wild database [20, 50], by collecting a new dataset consisting of 260 YouTube videos, with 1,413,000 frames and a total length of 13 hours and 5 minutes.”
Analysing Affective Behavior in the First ABAW 2020 Competition (2020, background): “tated). Aff-Wild2 is currently the largest (and audiovisual) in-the-wild database annotated for valence and arousal. All videos have been collected from YouTube. Aff-Wild2 is an extension of Aff-Wild [16], [29], [15]; 260 more YouTube videos, with 1;413;000 frames, have been added to Aff-Wild. Aff-W”