Speech Emotion Recognition Using Deep Convolutional Neural Network and Discriminant Temporal Pyramid Matching

By Shiqing Zhang, Shiliang Zhang, Tiejun Huang, Wen Gao. IEEE transactions on multimedia, 2018

385 citations recorded by the source corpus.

Speech emotion recognition is challenging because of the affective gap between the subjective emotions and low-level features. Integrating multilevel feature learning and model training, deep convolutional neural networks (DCNN) has exhibited remarkable success in bridging the semantic gap in visual tasks like image classification, object detection. This paper explores how to utilize a DCNN to bridge the affective gap in speech signals. To this end, we first extract three channels of log Mel-spectrograms (static, delta, and delta delta) similar to the red, green, blue (RGB) image representation as the DCNN input. Then, the AlexNet DCNN model pretrained on the large ImageNet dataset is employed to learn high-level feature representations on each segment divided from an utterance. The learned segment-level features are aggregated by a discriminant temporal pyramid matching (DTPM) strategy. DTPM combines temporal pyramid matching and optimal Lp-norm pooling to form a global utterance-level feature representation, followed by the linear support vector machines for emotion classification. Experimental results on four public datasets, that is, EMO-DB, RML, eNTERFACE05, and BAUM-1s, show the promising performance of our DCNN model and the DTPM strategy. Another interesting finding is that the DCNN model pretrained for image applications performs reasonably good in affective speech feature extraction. Further fine tuning on the target emotional speech datasets substantially promotes recognition performance.

Subfields: Computer Vision and Pattern Recognition, Experimental and Cognitive Psychology, Signal Processing

View Paper: https://www.semanticscholar.org/paper/2cecfe08fb146110aadd93e6d021fb21f06fda23

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Muhammad Sajjad
2. Mustaqeem
3. .. Mustaqeem
4. Haizhou Li
5. Guanglai Gao
6. Antonio Liotta
7. Rui Liu
8. Mingyi Chen

In Experimental and Cognitive Psychology

1. Soonil Kwon
2. Mustaqeem
3. Muhammad Sajjad
4. .. Mustaqeem
5. Haizhou Li
6. Rui Liu
7. Mingyi Chen
8. Han Zhang

In Signal Processing

1. Soonil Kwon
2. Rui Liu
3. Mustaqeem
4. Berrak Sisman
5. Muhammad Sajjad
6. .. Mustaqeem
7. Haizhou Li
8. Guanglai Gao

Citations in the corpus

• A CNN-Assisted Enhanced Audio Signal Processing for Speech Emotion Recognition (2019, methodology): “Recently, researchers have worked with CNN features for SER. Zhang et al. [23] developed a DCNN-based method for SER using the pre-trained CNN AlexNet model to learn deep features and trained a traditional classifier, a SVM, to recognize the emotional state of the speaker.”
• 3-D Convolutional Recurrent Neural Networks With Attention Model for Speech Emotion Recognition (2018, background): “Speech as the most convenient and natural medium for human communication, not only carries the implicit semantic information, but also contains rich affective information [1].”
• A Systematic Review on Affective Computing: Emotion Models, Databases, and Recent Advances (2022, methodology): “The deep ConvNet-based models are also used to extract emotion features from raw speech input for multi-task SER [195].”
• Expressive TTS Training With Frame and Style Reconstruction Loss (2020, methodology): “For SER training, we first split the speech signals into segments of 3 seconds as in [92].”
• Clustering-Based Speech Emotion Recognition by Incorporating Learned Features and Deep BiLSTM (2020, methodology): “Zhang et al. [27] presented a technique for SER by utilizing the pre-trained Alex-Net model for features representation and traditional support vector machine (SVM) for emotions classification.”