Multimodal Speech Emotion Recognition Using Audio and Text
By Seunghyun Yoon, Seokhyun Byun, Kyomin Jung. Spoken Language Technology Workshop, 2018
352 citations recorded by the source corpus.
Speech emotion recognition is a challenging task, and extensive reliance has been placed on models that use audio features in building well-performing classifiers. In this paper, we propose a novel deep dual recurrent encoder model that utilizes text data and audio signals simultaneously to obtain a better understanding of speech data. As emotional dialogue is composed of sound and spoken content, our model encodes the information from audio and text sequences using dual recurrent neural networks (RNNs) and then combines the information from these sources to predict the emotion class. This architecture analyzes speech data from the signal level to the language level, and it thus utilizes the information within the data more comprehensively than models that focus on audio features. Extensive experiments are conducted to investigate the efficacy and properties of the proposed model. Our proposed model outperforms previous state-of-the-art methods in assigning data to one of four emotion categories (i.e., angry, happy, sad and neutral) when the model is applied to the IEMOCAP dataset, as reflected by accuracies ranging from 68.8% to 71.8%.
View Paper: https://www.semanticscholar.org/paper/d70ba7cb3872b3df5abadbb230bfaf57b02e7cca
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
- Björn Schuller
- Florian Eyben
- Andreas Triantafyllopoulos
- Maximilian Schmitt
- Taiba Maijd Wani
- Christian Fuegen
- Yang Gao
- Teddy Surya Gunawan
Citations in the corpus
- A Comprehensive Review of Speech Emotion Recognition Systems (2021, methodology): “A multimodal text and audio system using a dual RNN is proposed in [90].”
- Interactive Text-to-Speech via Semi-supervised Style Transfer Learning (2020, methodology): “[22] proposed a multi-modal dual recurrent encoder to simultaneously model the dynamics of both text and audio signals within an utterance to predict emotion classes.”
- Dawn of the Transformer Era in Speech Emotion Recognition: Closing the Valence Gap (2022, background): “Instead, we aim towards a model that only implicitly utilises the linguistic information stream during deployment, and does not require access to ASR and NLP frontends.”