Emotion Recognition from Speech Using Wav2vec 2.0 Embeddings
By Leonardo Pepino, Pablo Ernesto Riera, Luciana Ferrer. 2021
483 citations recorded by the source corpus.
Emotion recognition datasets are relatively small, making the use of the more sophisticated deep learning approaches challenging. In this work, we propose a transfer learning method for speech emotion recognition where features extracted from pre-trained wav2vec 2.0 models are modeled using simple neural networks. We propose to combine the output of several layers from the pre-trained model using trainable weights which are learned jointly with the downstream model. Further, we compare performance using two different wav2vec 2.0 models, with and without finetuning for speech recognition. We evaluate our proposed approaches on two standard emotion databases IEMOCAP and RAVDESS, showing superior performance compared to results in the literature.
Subfields: Artificial Intelligence, Experimental and Cognitive Psychology, Sentiment & Opinion Analysis, Speech & Audio, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Graham Neubig
- Percy Liang
- Yoshua Bengio
- Quoc V. Le
- Michael Auli
In Experimental and Cognitive Psychology
In Speech & Audio
- Wei-Ning Hsu
- Alexis Conneau
- Alexei Baevski
- Michael Auli
- Xie Chen
- Kyu J. Han
- Yoav Artzi
- Kilian Q. Weinberger
In Signal Processing
- Wei-Ning Hsu
- Alexei Baevski
- Alexis Conneau
- Michael Auli
- Xie Chen
- Kyu J. Han
- Yoav Artzi
- Kilian Q. Weinberger
Citations in the corpus
- Unsupervised Speech Recognition (2021, background): “There has been a lot of recent work in this direction which has shown strong performance in extremely low-labeled data setups across a range of languages (Conneau et al., 2020) and tasks (Fan et al., 2021; Pepino et al., 2021; Wang et al., 2021).”
- emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation (2023, methodology): “We present the first universal speech emotion representation model, whereas most of the previous works directly employ speech pre-training models (Pepino et al., 2021; Li et al., 2022), or fine-tune the pre-training models on their specific emotional data with specific emotional tasks (mostly SER)…”
- Dawn of the Transformer Era in Speech Emotion Recognition: Closing the Valence Gap (2022, unknown): “Throughout the paper, we discuss results obtained with transformer-based models pre-trained on large amounts of unla-belled data.”
- Performance-Efficiency Trade-Offs in Unsupervised Pre-Training for Speech Recognition (2021, background): “…there is significant interest in self-supervised pre-training using unlabeled audio data to learn versatile feature representations, that are subsequently fine-tuned on task-specific annotated audio (Zhang et al., 2020b; Wang et al., 2021b; Xu et al., 2020a; Pepino et al., 2021).”
- Automatic speaker verification spoofing and deepfake detection using wav2vec 2.0 and data augmentation (2022, unknown): “0, have been applied to automatic speech recognition [12, 13], mispronunciation detection [36, 37], speaker recognition [38, 39] and emotion recognition [40].”