Dawn of the Transformer Era in Speech Emotion Recognition: Closing the Valence Gap
By Johannes Wagner, Andreas Triantafyllopoulos, Hagen Wierstorf, Maximilian Schmitt, Florian Eyben, Björn Schuller. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022
492 citations recorded by the source corpus.
Recent advances in transformer-based architectures have shown promise in several machine learning tasks. In the audio domain, such architectures have been successfully utilised in the field of speech emotion recognition (SER). However, existing works have not evaluated the influence of model size and pre-training data on downstream performance, and have shown limited attention to generalisation, robustness, fairness, and efficiency. The present contribution conducts a thorough analysis of these aspects on several pre-trained variants of wav2vec 2.0 and HuBERT that we fine-tuned on the dimensions arousal, dominance, and valence of MSP-Podcast, while additionally using IEMOCAP and MOSI to test cross-corpus generalisation. To the best of our knowledge, we obtain the top performance for valence prediction without use of explicit linguistic information, with a concordance correlation coefficient (CCC) of. 638 on MSP-Podcast. Our investigations reveal that transformer-based architectures are more robust compared to a CNN-based baseline and fair with respect to gender groups, but not towards individual speakers. Finally, we show that their success on valence is based on implicit linguistic information, which explains why they perform on-par with recent multimodal approaches that explicitly utilise textual information. To make our findings reproducible, we release the best performing model to the community.
Subfields: Artificial Intelligence, Experimental and Cognitive Psychology, Signal Processing
View Paper: https://www.semanticscholar.org/paper/3e61fa97c97815285601b0d1a6f1eb1c3ce0f52a
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Shinji Watanabe
- Björn Schuller
- Alexander I. Rudnicky
- Li-Wei Chen
- Panagiotis Tzirakis
- Ruibo Fu
- John R. Hershey
- Jinyu Li
In Experimental and Cognitive Psychology
- Panagiotis Tzirakis
- Björn Schuller
- Shinji Watanabe
- Florian Eyben
- Maja Pantic
- Anton Batliner
- Stefan Steidl
- Hatice Gunes
In Signal Processing
- Shinji Watanabe
- Björn Schuller
- Panagiotis Tzirakis
- Alexander I. Rudnicky
- Li-Wei Chen
- Andreas Triantafyllopoulos
- Ruibo Fu
- John R. Hershey
Citations in the corpus
- A Vector Quantized Approach for Text to Speech Synthesis on Real-World Spontaneous Speech (2023, methodology): “We use the dimensional emotion classifier released by (Wagner et al. 2022) pretrained on MSP-Podcast (Lotfian and Busso 2019).”
- An Overview of Affective Speech Synthesis and Conversion in the Deep Learning Era (2022, background): “Ultimately, those approaches are limited by the effectiveness of the SER systems, which, although greatly improving in recent years [159], is still far from perfect.”