End-to-End Multimodal Emotion Recognition Using Deep Neural Networks

By Panagiotis Tzirakis, George Trigeorgis, Mihalis A. Nicolaou, Björn Schuller, Stefanos P. Zafeiriou. IEEE Journal on Selected Topics in Signal Processing, 2017

648 citations recorded by the source corpus.

Automatic affect recognition is a challenging task due to the various modalities emotions can be expressed with. Applications can be found in many domains including multimedia retrieval and human–computer interaction. In recent years, deep neural networks have been used with great success in determining emotional states. Inspired by this success, we propose an emotion recognition system using auditory and visual modalities. To capture the emotional content for various styles of speaking, robust features need to be extracted. To this purpose, we utilize a convolutional neural network (CNN) to extract features from the speech, while for the visual modality a deep residual network of 50 layers is used. In addition to the importance of feature extraction, a machine learning algorithm needs also to be insensitive to outliers while being able to model the context. To tackle this problem, long short-term memory networks are utilized. The system is then trained in an end-to-end fashion where—by also taking advantage of the correlations of each of the streams—we manage to significantly outperform, in terms of concordance correlation coefficient, traditional approaches based on auditory and visual handcrafted features for the prediction of spontaneous and natural emotions on the RECOLA database of the AVEC 2016 research challenge on emotion recognition.

Subfields: Computer Vision and Pattern Recognition, Experimental and Cognitive Psychology, Social Psychology

View Paper: https://www.semanticscholar.org/paper/8cf0c072eeaec310aff806bdb7a7931b81e9ed99

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Experimental and Cognitive Psychology

In Social Psychology

Citations in the corpus

ICON: Interactive Conversational Memory Network for Multimodal Emotion Detection (2018, background): “Numerous previous approaches have relied on fusion techniques that leverage multiple modalities for affect recognition (Soleymani et al., 2012; Zadeh et al., 2017; Chen et al., 2017; Tzirakis et al., 2017; Zadeh et al., 2018b).”
Introducing WESAD, a Multimodal Dataset for Wearable Stress and Affect Detection (2018, background): “Using deep neural networks, trained on audio and/or visual data, a high performance in emotion classification is achieved [17, 29].”
Deep-Emotion: Facial Expression Recognition Using Attentional Convolutional Network (2019, background): “Recently, with the use of deep learning and especially convolutional neural networks (CNNs) [16], many features can be extracted and learned for a decent facial expression recognition system [17,18].”
Clustering-Based Speech Emotion Recognition by Incorporating Learned Features and Deep BiLSTM (2020, methodology): “The joint approach of CNN-LSTM is presented in [24] to extract the deep salient high-level features from raw speech data using CNN and passed to the LSTM network for capturing the sequential information similar to [25].”
A Systematic Review on Affective Computing: Emotion Models, Databases, and Recent Advances (2022, methodology): “Feature-level fusion combines features extracted from the multimodal inputs to form one general feature vector, which is then sent into the classifier.”
Speech Emotion Recognition Using Deep Learning Techniques: A Review (2019, methodology): “A hybrid deep learning modality may inherit the underlying properties of RNN with CNN, with convolutional levels implanted with RNN [94], [105], [118].”
EmotionMeter: A Multimodal Framework for Recognizing Human Emotions (2019, background): “However, most studies have focused on combining auditory and visual modalities for multimodal emotion recognition [9].”
An End-to-End Multimodal Voice Activity Detection Using WaveNet Encoder and Residual Networks (2019, neutral): “[39] recently proposed another multimodal end-to-end method for emotion recognition.”
Deep Affect Prediction in-the-Wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond (2018, methodology): “Our loss function was based on the Concordance Correlation Coefficient (CCC) metric that has been shown to provide better insight on whether the prediction follows the structure of the ground truth annotation [54] [53].”
End-to-End Audiovisual Speech Recognition (2018, background): “A similar architecture has been proposed by [18] for audiovisual emotion recognition.”
Conversational Memory Network for Emotion Recognition in Dyadic Dialogue Videos (2018, background): “Recently, this combination of modalities has provided the best performance in affect recognition systems (Poria et al., 2017b; Wang et al., 2017; Tzirakis et al., 2017), thus motivating the use of a multimodal approach.”
Mobile Game Evaluation Method Based on Data Mining of Affective Time Series (2025, unknown): “Classifier construction has been the focus of research in, e.g., [16,17].”