Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review

By Jianhua Zhang, Z. Yin, Peng Chen, S. Nichele. Information Fusion, 2020

680 citations recorded by the source corpus.

Abstract In recent years, the rapid advances in machine learning (ML) and information fusion has made it possible to endow machines/computers with the ability of emotion understanding, recognition, and analysis. Emotion recognition has attracted increasingly intense interest from researchers from diverse fields. Human emotions can be recognized from facial expressions, speech, behavior (gesture/posture) or physiological signals. However, the first three methods can be ineffective since humans may involuntarily or deliberately conceal their real emotions (so-called social masking). The use of physiological signals can lead to more objective and reliable emotion recognition. Compared with peripheral neurophysiological signals, electroencephalogram (EEG) signals respond to fluctuations of affective states more sensitively and in real time and thus can provide useful features of emotional states. Therefore, various EEG-based emotion recognition techniques have been developed recently. In this paper, the emotion recognition methods based on multi-channel EEG signals as well as multi-modal physiological signals are reviewed. According to the standard pipeline for emotion recognition, we review different feature extraction (e.g., wavelet transform and nonlinear dynamics), feature reduction, and ML classifier design methods (e.g., k-nearest neighbor (KNN), naive Bayesian (NB), support vector machine (SVM) and random forest (RF)). Furthermore, the EEG rhythms that are highly correlated with emotions are analyzed and the correlation between different brain areas and emotions is discussed. Finally, we compare different ML and deep learning algorithms for emotion recognition and suggest several open problems and future research directions in this exciting and fast-growing area of AI.

Subfields: Cognitive Neuroscience, Experimental and Cognitive Psychology, Signal Processing

View Paper: https://www.semanticscholar.org/paper/e5822c78e94e53fb56094fc443bb1c650168d014

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

1. Xindong Wu
2. A. Hammad
3. Essam H. Houssein
4. Weifeng Ge
5. Shirui Pan
6. Abdelmgeid A. Ali

In Experimental and Cognitive Psychology

1. A. Hammad
2. Essam H. Houssein
3. Abdelmgeid A. Ali

In Signal Processing

1. Xindong Wu
2. Antonio Liotta
3. A. Hammad
4. Essam H. Houssein
5. Weifeng Ge
6. Shirui Pan
7. Abdelmgeid A. Ali

Citations in the corpus

• A Systematic Review on Affective Computing: Emotion Models, Databases, and Recent Advances (2022, methodology): “1 EEG-based emotion recognition Compared with other peripheral neuro-physiological signals, EEG signals can directly measure the changes of brain activities, which provide internal features of emotional states [20].”
• Emotion recognition and artificial intelligence: A systematic review (2014-2023) and research recommendations (2023, neutral): “Zhang et al. [32] presented a review of EEG signals and ML techniques for emotion recognition without PRISMA guidelines.”
• Human emotion recognition from EEG-based brain–computer interface using machine learning: a comprehensive review (2022, background): “A dependable, accurate, flexible, and powerful emotion recognition system is required to realize intelligent HCI systems HCI [8].”
• Unifying Large Language Models and Knowledge Graphs: A Roadmap (2023, methodology): “To better unify LLMs and KGs, many advanced techniques need to be incorporated, such as multi-modal learning [261], graph neural network [262], and continuous learning [263].”