Analysis of emotion recognition using facial expressions, speech and multimodal information

By Carlos Busso, Zhigang Deng, S. Yıldırım, Murtaza Bulut, Chi-Chun Lee, Ebrahim (Abe) Kazemzadeh, Sungbok Lee, Ulrich Neumann, Shrikanth S. Narayanan. International Conference on Multimodal Interaction, 2004

952 citations recorded by the source corpus.

The interaction between human beings and computers will be more natural if computers are able to perceive and respond to human non-verbal communication such as emotions. Although several approaches have been proposed to recognize human emotions based on facial expressions or speech, relatively limited work has been done to fuse these two, and other, modalities to improve the accuracy and robustness of the emotion recognition system. This paper analyzes the strengths and the limitations of systems based only on facial expressions or acoustic information. It also discusses two approaches used to fuse these two modalities: decision level and feature level integration. Using a database recorded from an actress, four emotions were classified: sadness, anger, happiness, and neutral state. By the use of markers on her face, detailed facial motions were captured with motion capture, in conjunction with simultaneous speech recordings. The results reveal that the system based on facial expression gave better performance than the system based on just acoustic information for the emotions considered. Results also show the complementarily of the two modalities and that when these two modalities are fused, the performance and the robustness of the emotion recognition system improve measurably.

Subfields: Computer Vision and Pattern Recognition, Experimental and Cognitive Psychology, Signal Processing

View Paper: https://www.semanticscholar.org/paper/069443f2bbeb2deedefb600c82ed59cde2137e60

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Experimental and Cognitive Psychology

In Signal Processing

Citations in the corpus

Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-related Applications (2016, methodology): “put, otputting a weighted linear combination of the predictions [161], [162], [167], [200].”
Current State of Text Sentiment Analysis from Opinion to Emotion Mining (2017, background): “There is research on detecting emotions from text, facial expressions, images, speeches, paintings, songs, and other sorts of media [Busso et al. 2004; Wieczorkowska et al. 2006].”
Emotion Recognition From EEG Using Higher Order Crossings (2010, methodology): “in [12], facial expressions, acoustic information and combination of them are examined for their emotion discrimination efficiency among the four aforementioned emotions using SVM;”
A Review and Meta-Analysis of Multimodal Affect Detection Systems (2015, background): “There was also considerable variability (SD = 37.8) in the number of participants used model building, ranging from a single participant [Busso et al. 2004; Haq et al. 2008] to 343 participants [W¨ollmer et al. 2013b].”
Emotion recognition using brain activity (2008, background): “For example, research has been done to make computers recognize emotion from speech [7], facial expressions [12] or a fusion of both methods [4].”
DAGER: Deep Age, Gender and Emotion Recognition Using Convolutional Neural Network (2017, background): “Facial attribute recognition, including age, gender and emotion, [1,2,3,4,5,6,7] has been a topic of interest among computer vision researchers for over a decade.”
Human-Computer Interaction: Overview on State of the Art (2008, neutral): “In [69], the authors attempted to fuse facial and voice data for affect recognition.”
Hierarchical Convolutional Neural Networks for EEG-Based Emotion Recognition (2017, background): “For example, [7] fused expression, speech, and other multimodal information together and discriminated four emotion states.”
Conversational Memory Network for Emotion Recognition in Dyadic Dialogue Videos (2018, background): “Hence, many works used visual and audio features together for detecting affect (Busso et al., 2004; Castellano et al., 2008; Ranganathan et al., 2016).”
Recognising realistic emotions and affect in speech: State of the art and lessons learnt from the first challenge (2011, background): “…such as facial expression or, in a few cases, also body postures and gestures is exploited, followed by fewer studies on audiovisual processing (e.g., Chen et al., 1998; Pantic and Rothkrantz, 2003; Busso et al., 2004; Wang and Guan, 2005; Schuller et al., 2007b, 2009b; Zeng et al., 2007a,b, 2009).”
Deep learning for robust feature generation in audiovisual emotion recognition (2013, background): “Previous research has demonstrated the benefit of using multimodal data in emotion recognition tasks and has identified various techniques for generating robust multimodal features [2] [3] [4] [5] [6].”
Emotion Recognition in Speech using Cross-Modal Transfer in the Wild (2018, background): “Second, it has been observed that certain emotions are easier to detect from speech than faces, and vice versa [15], suggesting that the degree to which there is a redundant emotional signal across modalities may differ across emotions.”