Learning Affective Features With a Hybrid Deep Model for Audio–Visual Emotion Recognition

By Shiqing Zhang, Shiliang Zhang, Tiejun Huang, Wen Gao, Qi Tian. IEEE transactions on circuits and systems for video technology (Print), 2018

305 citations recorded by the source corpus.

Emotion recognition is challenging due to the emotional gap between emotions and audio–visual features. Motivated by the powerful feature learning ability of deep neural networks, this paper proposes to bridge the emotional gap by using a hybrid deep model, which first produces audio–visual segment features with Convolutional Neural Networks (CNNs) and 3D-CNN, then fuses audio–visual segment features in a Deep Belief Networks (DBNs). The proposed method is trained in two stages. First, CNN and 3D-CNN models pre-trained on corresponding large-scale image and video classification tasks are fine-tuned on emotion recognition tasks to learn audio and visual segment features, respectively. Second, the outputs of CNN and 3D-CNN models are combined into a fusion network built with a DBN model. The fusion network is trained to jointly learn a discriminative audio–visual segment feature representation. After average-pooling segment features learned by DBN to form a fixed-length global video feature, a linear Support Vector Machine is used for video emotion classification. Experimental results on three public audio–visual emotional databases, including the acted RML database, the acted eNTERFACE05 database, and the spontaneous BAUM-1s database, demonstrate the promising performance of the proposed method. To the best of our knowledge, this is an early work fusing audio and visual cues with CNN, 3D-CNN, and DBN for audio–visual emotion recognition.

Subfields: Experimental and Cognitive Psychology, Signal Processing

View Paper: https://www.semanticscholar.org/paper/6aeadade814902ddb9d7fd0e1d2d64a70f0e3ff9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Experimental and Cognitive Psychology

1. Ivan Marsic
2. Shuhong Chen
3. Shiyu Fu
4. Xinyu Li
5. Kangning Yang
6. Yue Gu

In Signal Processing

1. Ivan Marsic
2. Shuhong Chen
3. Antonio Liotta
4. Xinyu Li
5. Weifeng Ge
6. Yue Gu

Citations in the corpus

• Multimodal Affective Analysis Using Hierarchical Attention Strategy with Word-Level Alignment (2018, background): “A basic challenge in sentiment analysis and emotion recognition is filling the gap between extracted features and the actual affective states (Zhang et al., 2017).”
• A Systematic Review on Affective Computing: Emotion Models, Databases, and Recent Advances (2022, methodology): “Model-based [33, 37, 279, 353, 369] Feature-level [31, 349, 364-366]”