Speech emotion recognition using hidden Markov models

By A. Nogueiras, A. Moreno, Antonio Bonafonte, J. Mariño. Interspeech, 2001

830 citations recorded by the source corpus.

This paper introduces a first approach to emotion recognition using RAMSES, the UPC’s speech recognition system. The approach is based on standard speech recognition technology using hidden semi-continuous Markov models. Both the selection of low level features and the design of the recognition system are addressed. Results are given on speaker dependent emotion recognition using the Spanish corpus of INTERFACE Emotional Speech Synthesis Database. The accuracy recognising seven different emotions—the six ones defined in MPEG-4 plus neutral style—exceeds 80% using the best combination of low level features and HMM structure. This result is very similar to that obtained with the same database in subjective evaluation by human judges. Dealing with the speaker’s emotion is one of the latest challenges in speech technologies. Three different aspects can be easily identified: speech recognition in the presence of emotional speech, synthesis of emotional speech, and emotion recognition. In this last case, the objective is to determine the emotional state of the speaker out of the speech samples. Possible applications include from help to psychiatric diagnosis to intelligent toys, and is a subject of recent but rapidly growing interest [1]. This paper describes the TALP researchers first approach to emotion recognition. The work is inserted in the scope of the INTERFACE project [2]. The objective of this European Commission sponsored project is “to define new models and implement advanced tools for audio-video analysis, synthesis and representation in order to provide essential technologies for the implementation of large-scale virtual and augmented environments. The work is oriented to make man-machine interaction as natural as possible, based on everyday human communication by speech, facial expressions and body gestures.” In the field of emotion recognition out of speech, the main goal of the INTERFACE project will be the construction of a real-time multi-lingual speaker independent emotion recogniser. For this purpose, large speech databases with recordings from many speakers and languages are needed. As these resources are not available yet, a reduced problem will be addressed first: emotion recognition in multi-speaker language dependent conditions. Namely, this paper deals with the recognition of emotion for two Spanish speakers using standard hidden Markov models technology.

Subfields: Computer Vision and Pattern Recognition, Experimental and Cognitive Psychology, Signal Processing

View Paper: https://www.semanticscholar.org/paper/52074194ceeec6a0eb20bea44a96f1372a08aa36

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Experimental and Cognitive Psychology

In Signal Processing

Citations in the corpus

Survey on speech emotion recognition: Features, classification schemes, and databases (2011, methodology): “[94] compared a linear-based feature, namely Logfrequency power coefficients (LFPC), and two cepstral-based features, namely LPCC and MFCC.”
Learning Salient Features for Speech Emotion Recognition Using Convolutional Neural Networks (2014, background): “A typical SER system consists of two components: 1) a front-end processing unit that extracts the appropriate features from the speech data, and 2) a classifier that decides the emotion of the speech utterance.”
Hybrid Deep Neural Network--Hidden Markov Model (DNN-HMM) Based Speech Emotion Recognition (2013, background): “2) for frame based dynamic spectral features like Mel Filterbank Cepstrum Coefficient (MFCC), dynamical models such as Hidden Markov Model (HMM) are considered [1], [2].”
Prosody Generation for Speech-to-Speech Translation (2006, background): “some emotions) F0 movements are in some extend language independent ([3, 4]).”
Features and classifiers for emotion recognition from speech: a survey from 2000 to 2011 (2012, background): “Hidden Markov Models (HMMs) are considered to be a common classification/modeling technique for speech emotion recognition (Schuller et al. 2003; Pao et al. 2007a; Ijima et al. 2009; Yun and Yoo 2009; Yu 2008; Fu et al. 2008b; Nogueiras et al. 2001).”
Emotional speech recognition: Resources, features, and methods (2006, methodology): “Finally, the paper is focused on describing the feature extraction methods and the emotion classification techniques, topics that have not been treated in (Scherer, 2003; Pantic and Rothkrantz, 2003).”
Hidden Markov model-based speech emotion recognition (2003, background): “However also first efforts in recognition of instantaneous features exist [3][4].”
Databases, features and classifiers for speech emotion recognition: a review (2018, background): “The utterances of 10 sentences regarding anger, fear, joy, sadness, disgust and surprise emotions in Mandarin and Burnese, were examined by New et al. (2003).”
Sentiment Adaptive End-to-End Dialog Systems (2018, background): “Many studies in emotion recognition (Schuller et al., 2003; Nwe et al., 2003; Bertero et al., 2016) have used only acoustic features.”
Emotion Recognition from Body Expressions with a Neural Network Architecture (2017, background): “In this context, a vast amount of research has been conducted in affective states recognition in the context of HRI scenarios, e.g., using facial expressions [1][13][2], speech detection [17] or a combination of these cues [1][2].”
High-level feature representation using recurrent neural network for speech emotion recognition (2015, methodology): “In the frame approach, low-level features are used directly to generate the distribution of each emotional state by using the Gaussian mixture model (GMM) [1] or the hidden Markov model (HMM) [2].”
A Comprehensive Review of Speech Emotion Recognition Systems (2021, methodology): “Hidden semi-continuous Markov models were utilized to construct a real-time multilingual speaker-independent emotion recognizer [52].”
Primitives-based evaluation and estimation of emotions in speech (2007, neutral): “They all treat the emotion recognition problem as a multiple classiﬁcation task of several emotional categories such as angry, happy, and sad ; or simply, negative and non-negative .”
Speech emotion recognition using convolutional and Recurrent Neural Networks (2016, background): “Traditional speech recognition methods are proposed in many research studies [3-5].”
Speech Emotion Recognition Using Deep Convolutional Neural Network and Discriminant Temporal Pyramid Matching (2018, methodology): “Model (GMM) [27], Hidden Markov Models (HMM) [28], and SVM [29], are widely adopted for speech emotion recognition.”
Emotion recognition from speech: a review (2012, methodology): “From literature, it is observed that, most speech emotion recognition studies are carried out using utterance level static (global) prosodic features (Nwe et al. 2003; Schroder and Cowie 2006; Dellaert et al. 1996; Koolagudi et al. 2009; Ververidis et al. 2004; Iida et al. 2003).”
Automatic speech emotion recognition using modulation spectral features (2011, background): “The spectral features are usually extracted over a short frame duration (e.g. 20–30 ms), with longer temporal information incorporated in the form of local derivatives (e.g. Nwe et al., 2003; Batliner et al., 2006; Vlasenko et al., 2007).”
Speech emotion recognition combining acoustic features and linguistic information in a hybrid support vector machine-belief network architecture (2004, background): “In view of international comparability [4][5] we decided upon this set of seven emotions in our work.”