Combining modality specific deep neural networks for emotion recognition in video

By Samira Ebrahimi Kahou, C. Pal, Xavier Bouthillier, Pierre Froumenty, Caglar Gulcehre, Roland Memisevic, Pascal Vincent, Aaron C. Courville, Yoshua Bengio, Raul Chandias Ferrari and 17 more. 2013

397 citations recorded by the source corpus.

In this paper we present the techniques used for the University of Montréal's team submissions to the 2013 Emotion Recognition in the Wild Challenge. The challenge is to classify the emotions expressed by the primary human subject in short video clips extracted from feature length movies. This involves the analysis of video clips of acted scenes lasting approximately one-two seconds, including the audio track which may contain human voices as well as background music. Our approach combines multiple deep neural networks for different data modalities, including: (1) a deep convolutional neural network for the analysis of facial expressions within video frames; (2) a deep belief net to capture audio information; (3) a deep autoencoder to model the spatio-temporal information produced by the human actions depicted within the entire scene; and (4) a shallow network architecture focused on extracted features of the mouth of the primary human subject in the scene. We discuss each of these techniques, their performance characteristics and different strategies to aggregate their predictions. Our best single model was a convolutional neural network trained to predict emotions from static frames using two large data sets, the Toronto Face Database and our own set of faces images harvested from Google image search, followed by a per frame aggregation strategy that used the challenge training data. This yielded a test set accuracy of 35.58%. Using our best strategy for aggregating our top performing models into a single predictor we were able to produce an accuracy of 41.03% on the challenge test set. These compare favorably to the challenge baseline test set accuracy of 27.56%.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Deep Learning: Methods and Applications (2014, background): “For interested readers, please consult recent papers on the applications of deep learning to optimal control in [219], to reinforcement learning in [256], to malware classification in [66], to compressed sensing in [277], to recognition confidence prediction in [173], to acoustic-articulatory invers”
On the influence of momentum acceleration on online learning (2016, methodology): “O C ] 2 Keywords: online learning, stochastic gradient, momentum acceleration, heavy-ball method, Nesterov method, mean-square-error analysis, convergence rate.”
Training deep networks for facial expression recognition with crowd-sourced label distribution (2016, background): “Lately, due to the great success of DCNN in a wide variety of image classification tasks [16, 28], it has also been applied in emotion recognition [14, 18, 17, 29, 33].”
Developing crossmodal expression recognition based on a deep neural model (2016, background): “The works of Liu et al. (2014) and Kahou et al. (2013) extract more than 100 auditory features each, and use several CNNs to extract facial features.”
Reliable Crowdsourcing and Deep Locality-Preserving Learning for Unconstrained Facial Expression Recognition (2019, background): “tition [11], the winner [32] combined modality specific deep neural network models.”
Reliable Crowdsourcing and Deep Locality-Preserving Learning for Expression Recognition in the Wild (2017, neutral): “In EmotiW 2013 competition [6], the winner [19] combined modality speciﬁc deep neural network models.”
A Low Power, Fully Event-Based Gesture Recognition System (2017, methodology): “Multimodal systems [46, 31, 33, 15] have also been used to improve performance.”
Image based Static Facial Expression Recognition with Multiple Deep Network Learning (2015, background): “To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.”
Multi-scale Deep Learning for Gesture Detection and Localization (2014, background): “[7] won the 2013 Emotion Recognition in the Wild Challenge by building two convolutional architectures on several modalities, such as facial expressions from video frames, audio signal, scene context and features extracted around mouth regions.”
Deep Learning for Emotion Recognition on Small Datasets using Transfer Learning (2015, methodology): “It comes as no surprise that CNNs have worked well for emotion classification, as evidenced by their use in a number of state-of-the-art algorithms for this task, as well as winning related competitions [10], particularly previous years’ EmotiW challenge [11, 14].”
Recurrent Neural Networks for Emotion Recognition in Video (2015, methodology): “The combination weights are determined by random search [4], which was also used for model combination in the winning approach for the 2013 EmotiW challenge [18].”
Deep Multimodal Learning: A Survey on Recent Advances and Trends (2017, background): “There have been several works that employ late-or decision-level fusion for deep multimodal learning [33], [43], [62] in addition to some works listed in Table 3.”
Going deeper in facial expression recognition using deep neural networks (2015, methodology): “In [17] multiple deep neural network architectures are combined to solve the FER problem in video analysis.”
Facial Expression Analysis under Partial Occlusion (2018, background): “Among them, Kahou et al. [2013] integrated four deep neural networks in a unique system for emotion recognition in video by capturing facial expressions, audio information, spatio-temporal patterns of human actions, and features in the mouth region.”