EmoNets: Multimodal deep learning approaches for emotion recognition in video

By Samira Ebrahimi Kahou, Xavier Bouthillier, Pascal Lamblin, Caglar Gulcehre, Vincent Michalski, Kishore Reddy Konda, Sébastien Jean, Pierre Froumenty, Yann Dauphin, Nicolas Boulanger-Lewandowski and 8 more. 2015

412 citations recorded by the source corpus.

The task of the Emotion Recognition in the Wild (EmotiW) Challenge is to assign one of seven emotions to short video clips extracted from Hollywood style movies. The videos depict acted-out emotions under realistic conditions with a large degree of variation in attributes such as pose and illumination, making it worthwhile to explore approaches which consider combinations of features from multiple modalities for label assignment. In this paper we present our approach to learning several specialist models using deep learning techniques, each focusing on one modality. Among these are a convolutional neural network, focusing on capturing visual information in detected faces, a deep belief net focusing on the representation of the audio stream, a K-Means based “bag-of-mouths” model, which extracts visual features around the mouth region and a relational autoencoder, which addresses spatio-temporal aspects of videos. We explore multiple methods for the combination of cues from these modalities into one common classifier. This achieves a considerably greater accuracy than predictions from our strongest single-modality classifier. Our method was the winning submission in the 2013 EmotiW challenge and achieved a test set accuracy of 47.67 % on the 2014 dataset.

Subfields: Computer Vision and Pattern Recognition, Experimental and Cognitive Psychology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Experimental and Cognitive Psychology

Citations in the corpus

Deep Learning for IoT Big Data and Streaming Analytics: A Survey (2017, methodology): “Beyond the physical movements, emotion estimation of humans from video frames has been also investigated in [105] using a model that consists of a CNN, DBN, and AE.”
Cognitive Science in the era of Artificial Intelligence: A roadmap for reverse-engineering the infant language-learner (2016, positive): “sed in Section 5 (speech recognition: Amodei et al., 2016; object recognition: Girshick, Donahue, Darrell, & Malik, 2016; action recognition: Rahmani, Mian, & Shah, 2016; emotion recognition: Kahou et al., 2015) will enable the semi-automatic annotations of large amounts of data. As for ethi”
Deep Multimodal Learning: A Survey on Recent Advances and Trends (2017, background): “…deep-learning methods have been applied to various problems involving human activity such as action recognition (an activity can be composed of two or more sequences of shorter actions), gesture recognition [8], gaze-direction estimation [9], face recognition [10], and emotion recognition [11].”
Recurrent Neural Networks for Emotion Recognition in Video (2015, methodology): “Our approach for activity analysis is based on the action recognition pipeline from [20, 23] which was also used for emotion recognition previously in [16].”
Speech Emotion Recognition Using Deep Learning Techniques: A Review (2019, background): “The pair of highest score is further combined with a vector known as compositional vector [129]–[131].”
A Unified Framework for Multimodal Domain Adaptation (2018, background): “In addition, many top-performing methods in multimedia applications, such as video recognition [29], always involve multiple modalities.”
End-to-End Multimodal Emotion Recognition Using Deep Neural Networks (2017, background): “[24] proposed to combine modality-specific DNNs to recognize categorical emotions in video.”
Deep Facial Expression Recognition: A Survey (2018, methodology): “Two aggregation approaches have been considered to generate a fixed-length feature vector for each sequence [60], [146]: frame averaging and frame expansion.”
Reliable Crowdsourcing and Deep Locality-Preserving Learning for Unconstrained Facial Expression Recognition (2019, background): “Fortunately, the emerging deep learning techniques have advanced unconstrained expression recognition to a new state-of-theart [31], [45].”
Emotion recognition using wireless signals (2016, background): “Emotion recognition is an emerging field that has attracted much interest from both the industry and the research community [52, 16, 30, 47, 23].”