Multimodal and Temporal Perception of Audio-visual Cues for Emotion Recognition

By Esam Ghaleb, Mirela C. Popa, Stylianos Asteriadis. 2019

87 citations recorded by the source corpus.

In Audio-Video Emotion Recognition (AVER), the idea is to have a human-level understanding of emotions from video clips. There is a need to bring these two modalities into a unified framework, to effectively learn multimodal fusion for AVER. In addition, literature studies lack in-depth analysis and utilization of how emotions vary as a function of time. Psychological and neurological studies show that negative and positive emotions are not recognized at the same speed. In this paper, we propose a novel multimodal temporal deep network framework that embeds video clips using their audio-visual content, onto a metric space, where their gap is reduced and their complementary and supplementary information is explored. We address two research questions, (1) how audio-visual cues contribute to emotion recognition and (2) how temporal information impacts the recognition rate and speed of emotions. The proposed method is evaluated on two datasets, CREMA-D and RAVDESS. The study findings are promising, achieving the state-of-the-artperformance on both datasets, and showing a significant impact of multimodal and temporal emotion perception.

Subfields: Experimental and Cognitive Psychology, Multimodal Learning, Sentiment & Opinion Analysis, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Experimental and Cognitive Psychology

1. Mohammed A. M. Sadeeq
2. S. Abdullah
3. Fei Zhao
4. Chengcui Zhang
5. Baocheng Geng
6. Joel Shor
7. Aren Jansen
8. Subhi R. M. Zeebaree

In Multimodal Learning

1. Yake Wei
2. Marco Tagliasacchi
3. Andong Deng
4. Xiaokang Peng
5. Di Hu
6. Fei Zhao
7. Chengcui Zhang
8. Baocheng Geng

In Sentiment & Opinion Analysis

1. Fei Zhao
2. Chengcui Zhang
3. Baocheng Geng

In Signal Processing

1. Marco Tagliasacchi
2. Joel Shor
3. Felix de Chaumont Quitry
4. Mohammed A. M. Sadeeq
5. Yake Wei
6. Andong Deng
7. S. Abdullah
8. Xiaokang Peng

Citations in the corpus

• Towards Learning a Universal Non-Semantic Representation of Speech (2020, methodology): “The best published result on this dataset uses both audio and visual information (Ghaleb et al., 2019), but our method only uses the audio modality.”
• Deep Multimodal Data Fusion (2024, methodology): “For video-audio-based multimodal data fusion, the works from [35, 37, 51, 163] address the emotion recognition problem by using deep learning techniques, including convolutional neural networks, long short-term memory ( LSTM ) networks, attention mechanisms, and so on.”
• Multimodal Emotion Recognition using Deep Learning (2021, background): “For the last few years, deep learning [23, 26], relying on the most modern system, has achieved great success in many areas, such as signal processing, artificial intelligence, and emotion detection.”
• Balanced Multimodal Learning via On-the-fly Gradient Modulation (2022, methodology): “Besides, OGM-GE is also combined with x-vector [30], i-vector [15], and MWTSM [12] on CREMA-D.”