Developing crossmodal expression recognition based on a deep neural model

By Pablo V. A. Barros, Stefan Wermter. 2016

70 citations recorded by the source corpus.

A robot capable of understanding emotion expressions can increase its own capability of solving problems by using emotion expressions as part of its own decision-making, in a similar way to humans. Evidence shows that the perception of human interaction starts with an innate perception mechanism, where the interaction between different entities is perceived and categorized into two very clear directions: positive or negative. While the person is developing during childhood, the perception evolves and is shaped based on the observation of human interaction, creating the capability to learn different categories of expressions. In the context of human–robot interaction, we propose a model that simulates the innate perception of audio–visual emotion expressions with deep neural networks, that learns new expressions by categorizing them into emotional clusters with a self-organizing layer. The proposed model is evaluated with three different corpora: The Surrey Audio–Visual Expressed Emotion (SAVEE) database, the visual Bi-modal Face and Body benchmark (FABO) database, and the multimodal corpus of the Emotion Recognition in the Wild (EmotiW) challenge. We use these corpora to evaluate the performance of the model to recognize emotional expressions, and compare it to state-of-the-art research.

Subfields: Computer Vision and Pattern Recognition, Experimental and Cognitive Psychology, Social Psychology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. German Ignacio Parisi
2. Stefan Wermter
3. Pablo V. A. Barros
4. Nourhan Elfaramawy
5. Shan Li
6. Weihong Deng
7. Sven Magg
8. Luiza Mici

In Experimental and Cognitive Psychology

1. German Ignacio Parisi
2. Stefan Wermter
3. Weihong Deng
4. Shan Li
5. Pablo V. A. Barros
6. Xun Liu
7. Haiyan Wu

In Social Psychology

1. German Ignacio Parisi
2. Pablo V. A. Barros
3. Stefan Wermter
4. Weihong Deng
5. Shan Li

Citations in the corpus

• A Neurorobotic Experiment for Crossmodal Conflict Resolution in Complex Environments * (2018, background): “following a multichannel architecture [18]: one for learning auditory features, a second for body movement and the third for facial expressions.”
• Emotion Recognition from Body Expressions with a Neural Network Architecture (2017, background): “Second, we could investigate the development of a multi-modal emotion recognition scenario, i.e. by taking into account auditory information that complements the use of visual cues [2].”
• Deep Facial Expression Recognition: A Survey (2018, background): “[162], [164], [165]) to capture the spatiotemporal features.”